Support out-of-source builds

[mathstuf]

It would be nice to be able to have tup build a project without touching the source tree. After using CMake for years, it's something I've grown accustomed to (for one, it allows one source tree to have multiple build trees for different compilers or options). I managed to hack bootstrap.sh to work, but it ended there.

[gittup]

Yes, this would be nice to have. It has been planned for a while now but sadly I have not had time to do it. Out-of-tree builds will likely be a part of build variants, which will be done at some point, but I don't have an official timeline for it. I agree this will be a nice feature to have though!

-Mike

On Thu, Nov 3, 2011 at 12:23 PM, Ben Boeckel reply@reply.github.com wrote:

It would be nice to be able to have tup build a project without touching the source tree. After using CMake for years, it's something I've grown accustomed to (for one, it allows one source tree to have multiple build trees for different compilers or options). I managed to hack bootstrap.sh to work, but it ended there.

---

Reply to this email directly or view it on GitHub: https://github.com/gittup/tup/issues/39

[majewsky]

+1 on this idea. There may be other things which need to be implemented in tup to make it a suitable backend for CMake, but this functionality needs to be available before one can even start thinking about a CMake backend.

[gittup]

On Wed, Apr 11, 2012 at 4:04 PM, Stefan Majewsky reply@reply.github.com wrote:

+1 on this idea. There may be other things which need to be implemented in tup to make it a suitable backend for CMake, but this functionality needs to be available before one can even start thinking about a CMake backend.

---

Reply to this email directly or view it on GitHub: https://github.com/gittup/tup/issues/39#issuecomment-5077714

There is currently a branch called 'out-of-tree' which includes one method of doing builds outside of the src tree. It is still a work in progress though - hopefully I will have something solid in a few weeks!

-Mike

[majewsky]

That's absolutely marvelous! I honestly don't understand why you didn't merge this into master right away. I've just used this branch to compile tup (both in-source and out of source) and checked that the compiled binaries pass the test suite on my Linux system.

[zjg]

I tried the out-of-tree branch with my large project at work (~300 Tupfiles), and it worked fine without any extra fiddling.

The only problem I ran into so far was when I accidentally deleted some directories in the build tree; tup complained with "tup error: Unable to open directory for update work." After manually re-creating the directories, tup was happy again.

[gittup]

On Thu, Apr 19, 2012 at 4:56 PM, Stefan Majewsky reply@reply.github.com wrote:

That's absolutely marvelous! I honestly don't understand why you didn't merge this into master right away. I've just used this branch to compile tup (both in-source and out of source) and checked that the compiled binaries pass the test suite on my Linux system.

It isn't complete yet - more test cases would need to be added to make sure it can handle situations such as where the object directory is a sub-directory of the srcdir, or vice versa. Also it would be nice if it cleaned up old directories (similar to the variant branch). I'm also not sure at this point which of the two branches will be merged to master - if you have a chance to try it out let me know what you think!

[gittup]

On Thu, Apr 19, 2012 at 7:43 PM, Doug Rosvick reply@reply.github.com wrote:

I tried the out-of-tree branch with my large project at work (~300 Tupfiles), and it worked fine without any extra fiddling.

The only problem I ran into so far was when I accidentally deleted some directories in the build tree; tup complained with "tup error: Unable to open directory for update work." After manually re-creating the directories, tup was happy again.

Ahh, it should definitely handle that (by recreating the directories automatically, just like if you remove generated files accidentally). I'll add it to my list of things to fix before merging...

Thanks! -Mike

[gittup]

The variants branch has been merged, which essentially allows building outside of the source tree. It should also handle the case of accidentally removing directories in the build tree by recreating them if necessary.

[eddyp]

Does the already merged code support build variants to any degree? For projects building for several compilers and platforms this would be really necessary.

[ppannuto]

Yes, variants are well-supported / well-tested / well-documented at this point.

This issue should probably just be closed.

[eddyp]

I've looked over the manual, but I find the variants support very limited. I work with a project where there are usually between 500 to 2000 variants of the project because there are some hard requirements to have the system statically configured. These variants are necessary to test the required features in the finished product, so having them all at the same level as the .tup directory is quite unfeasable. Instead we use a directory structure organized in groups and subgroups, is it's a tree of variants.

How hard would it be to support such a monster ? :-)

[FreddieChopin]

Yes, variants are well-supported / well-tested / well-documented at this point.

... on every system but Windows, where they simply don't work at all (;

[eddyp]

This is exactly where I need this to work :-( . I hope I'll have the time to try to make it work somehow on one of the projects that seems more closer to what tup can handle ATM...

[cppmaven]

I also need a tree of output variants as well. And like eddyp, it is quite unfeasable to place these inside the Tup hierarchy.

I can manage organizing the structure, by writing rules myself in Tup or LUA. I only need the generic ability to access files outside the Tup hierarchy. This ability would make Tup more flexible, allowing developers to customize it ways to solve their unique problems, given their unique organizational constraints.

The variants implementation is convenient, but not flexible enough for me to use.

[pradyunsg]

Bump.

[DPD85]

Dump.

[FreddieChopin]

This is perfectly possible now, but with lua parser. It's not the most convenient to use (requires extensive use of <groups>), but works. You can check that out in my RTOS projeckt, but it's only for embedded ARM microcontrollers, but the idea is the same in any case.

https://github.com/DISTORTEC/distortos

[eddyp]

@FreddieChopin Does the feature allow multiple ELFs to be generated without interfering with each other? I mean could I have something like a output/var1, output/var500 and simply not pollute the common source tree at all?

[FreddieChopin]

@eddyp - yes, sure. Why do you think there could be a problem? Generally the only issue is that you usually have to use %<groups> instead of full file names...

[eddyp]

@FreddieChopin because of this:

[..]To create a variant, make a new directory at the top of the tup hierarchy and create a "tup.config" file there.

As I said before, I am interested in a number of variants in the order of hundreds, so having a directory at the top level for each of them is not feasible, while organizing them all in sudirs of a single output dir would be a lot better.

The project I have is comprised of multiple compilable applications within the source tree, with a single core, but some of the applications can have multiple variants, so for me it would make even more sense to be able to place the variants subdirs structure in custom places in the tree.

To get a better idea, here is a simplified listing of the project:

prj/utest/utest_core prj/utest/platform\

prj/common_aux prj/platforms/platform\

/app\ prj/platforms/platform\

/platform_core prj/src/core prj/test/platform\

prj/test/test_core prj/test/test_apps/group\/subgroup\

Each of the places where there is a \<..> item, it means there are/can be multiple such dirs, so depending on the place where we are in the structure, we would compile a different application.

To add insult to injury, there are variants for each test application in each subgroup.

In order to compile a test application for a specific variant of G/SG (let's call it G_SG_VAR) for platform P1 we need to grab things from prj/src/core, prj/platforms/platformP1/core, prj/common_aux and prj/test/test_core.

So in this case it would probably make more sense for me to be able to place all the test outputs in prj/test/platformP/output/* because the test application entry point for this P1 platform is in prj/test/platformP1.

Of course, things are a little more complicated in practice, but that is the whole general idea.

@eddyp - please note that I'm not talking about variants, which I don't use at all. I'm talking about "normal" (that is - "without variants") build (possibly using lua parser), where you can easily place the output files anywhere you want. Such feature (any folder for output) was added to tup a long time ago, but this has nothing to do with variants.

You can emulate some of variants' functionality with lua parser.