gittup
commented
12 years ago On Thu, Oct 4, 2012 at 4:03 PM, ppannuto notifications@github.com wrote:
Tup's bins features are really useful, but bins only last the scope of a single Tupfile. I keep stumbling into cases where I'd like to reference global bins. I would propose a syntax extension such as:
: foreach *.c |> !cc |> {local_objs} {{global_objs}}
Where {local_objs} has local scope, but {{global_objs}} has global scope. This allows for things like this at the root:
Final Link
: {{global_objs}} |> !ld |> my_executable
From what I can tell, current examples suggest things such as using ar to package static libraries, or having a final link step more like:
Final Link
: polygons/.o vectors/.o |> !ld |> my_drawing_program
But I find this inelegant, as you have explicit paths in Tup rules that really don't belong. Furthermore, as the tup examples demonstrate, there are use cases where *.o breaks down.
What use cases are you referring to in the examples? It's been a while since I've looked at them :)
Is there anything like this in the works?
Probably the closest thing to what you want is the groups feature, where you can essentially put multiple outputs from disparate directories into the same group. Eg: : foreach *.c |> !cc |> {local_objs}
(untested, but I think that's how it goes).
However, currently when you use a group as an input, it just means "this
command can use any file from
Is this a terrible idea for some reason not immediately obvious?
No, it's not terrible, but tup currently runs with the idea that command strings are static, until the directory contents or the Tupfile itself changes (at which point we re-parse the Tupfile to get the new static command strings). Tupfiles are also parsed relatively independently, so if we parse a Tupfile like:
:
Then tup is going to try to build a string like "gcc lib/foo.o src/bar.o -o program" and store it in the database. However with a group tag as the input, we may not yet know what objects are going to be in the group at all, since we may not have parsed the lib/Tupfile and src/Tupfile yet. And given the information in this Tupfile, there's no way to know that we need to parse those Tupfiles first. In comparison, if we have:
: lib/.o src/.o |> gcc %f -o %o |> program
Then tup halts parsing this Tupfile, and parses lib/Tupfile so it knows what lib/.o will resolve to (and then similarly for src/Tupfile and src/.o).
If I wanted to start hacking this in, any direction for where to start? I understand tup's model at a high-level, but (I believe) implementing something like this would require another type of symbolic entry in the DAG, is there any kind of primer of tup's internals anywhere?
One method I tried previously for a similar problem is to allow the parser to store things like "gcc %f -o program" in the database, and then expand %f right before the sub-process is executed in the updater. I did not continue with that approach for other reasons (it was being worked in combination with an attempt to allow sub-processes to write to arbitrary files not listed in the Tupfile), but that may work here. The problem with that branch was not related to the delayed %f expansion, but with outputs stomping over existing things in the database. I can try to dig that up if you are interested.
I should note, though, that I think it is manageable to have each directory pull in objects from its immediate subdirectories. This is how the Linux kernel Makefiles work - each directory partially links its objects (using ld -r) to create a super-object named built-in.o. So instead of a top-level Tupfile with:
: src/.o src/foo/.o src/bar/*.o |> ... |>
You have:
src/Tupfile: : foreach .c |> gcc -c ... |> : .o foo/built-in.o bar/built-in.o |> ld -r %f -o %o |> built-in.o
foo/Tupfile: : foreach .c |> gcc -c ... |> : .o |> ld -r %f -o %o |> built-in.o
top-level Tupfile: : src/built-in.o |> ... |> program
This makes each directory easily maintainable, since it is dealing only with its own contents (ie: each c file in the current directory becomes a .o file, and we pull built-in.o from each immediate sub-directory). It also works well with tup's current model, so that if you add a new sub-directory under src/, it only needs to re-parse src/new/Tupfile and src/Tupfile.
-Mike
ppannuto
bradjc
Tup's bins features are really useful, but bins only last the scope of a single Tupfile. I keep stumbling into cases where I'd like to reference global bins. I would propose a syntax extension such as:
: foreach *.c |> !cc |> {local_objs} {{global_objs}}Where
{local_objs}has local scope, but{{global_objs}}has global scope. This allows for things like this at the root:# Final Link : {{global_objs}} |> !ld |> my_executableFrom what I can tell, current examples suggest things such as using ar to package static libraries, or having a final link step more like:
# Final Link : polygons/*.o vectors/*.o |> !ld |> my_drawing_programBut I find this inelegant, as you have explicit paths in Tup rules that really don't belong. Furthermore, as the tup examples demonstrate, there are use cases where *.o breaks down.
Is there anything like this in the works? Is this a terrible idea for some reason not immediately obvious? If I wanted to start hacking this in, any direction for where to start? I understand tup's model at a high-level, but (I believe) implementing something like this would require another type of symbolic entry in the DAG, is there any kind of primer of tup's internals anywhere?