Tup globbing rule not picking up Tup generated files

[VolatileDream]

(This is related to #214, and probably a special case of it.)

There are situations where adding new files is expected to affect the build, the following scenario demonstrates that new files are correctly picked up by relevant build rules, but that subsequent rules fail to register new tup generated files.

$ mkdir tup_bug/ ; cd tup_bug
$ mkdir out-dir

$ cat > Tupfile

# works with new user files, horray!
: foreach *.in |> echo %B > %o ; cat %f >> %o |> out-dir/%B.out

# doesn't pick up new tup generated files from the first command
: out-dir/*.out |> cat out-dir/* > %o |> all_files

$ tup init
.tup repository initialized.

$ echo abc > abc.in
$ echo def > def.in

$ tup
[ tup ] [0.074s] Scanning filesystem...
[ tup ] [0.149s] Reading in new environment variables...
[ tup ] [0.233s] Parsing Tupfiles...
 1) [0.001s] out-dir
 2) [0.003s] .
 [  ] 100%                                                                                                                                                    
[ tup ] [0.247s] No files to delete.                                                                                                                          
[ tup ] [0.247s] Converting normal directories to generated directories...
tup: Converting out-dir to a generated directory.
[ tup ] [0.247s] Generating .gitignore files...
[ tup ] [0.359s] Executing Commands...
 1) [0.005s] echo def > out-dir/def.out ; cat def.in >> out-dir/def.out                                                                                       
 2) [0.013s] echo abc > out-dir/abc.out ; cat abc.in >> out-dir/abc.out                                                                                       
 3) [0.007s] cat out-dir/* > all_files                                                                                                                          
 [   ] 100%
[ tup ] [0.476s] Updated.                                                                                                                                     

$ ls
abc.in  all_files  def.in  out-dir  Tupfile

$ cat all_files # we get expected output
abc
abc
def
def

$ echo ghi > ghi.in

$ tup # this second build doesn't work as expected
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.104s] Reading in new environment variables...
[ tup ] [0.104s] Parsing Tupfiles...
 1) [0.004s] .
 [ ] 100%
[ tup ] [0.120s] No files to delete.                                                                                                                          
[ tup ] [0.120s] Generating .gitignore files...
[ tup ] [0.230s] Executing Commands...
 1) [0.007s] echo ghi > out-dir/ghi.out ; cat ghi.in >> out-dir/ghi.out                                                                                       
 [ ] 100%
[ tup ] [0.314s] Updated.                                                                                                                                     

$ cat all_files # this was never updated for out-dir/ghi.out
abc
abc
def
def

Suggested Workarounds (from #214)

The first fix suggested (by @FreddieChopin) is to put the globbing pattern into the tup input file list, and has been done, but doesn't work.

The second suggested fix (by @Qix-) is to use a shell script to list the dependencies of the second build command. This seems like a lot of work for something that I expected to happen automatically.

Expected behaviour

I expected that the Tupfile in it's current form would catch all of the new tup generated out-dir/*.out files any time an *.in file is added, and update all_files accordingly.

The expectation is that any tup rule that has a globbing pattern will automatically get updated with any new matching tup generated files, because tup knows about them from a previous command.

[gittup]

Is there a reason you can't use %f in the second rule? Eg:

: out-dir/*.out |> cat %f > %o |> all_files

This would cause the command-line to change when a new .out file is created, and so tup will correctly run the new command.

I agree that the current implementation is broken, but to fix it correctly I think we need to introduce dependencies on directories. I'd like to add that at some point, but unfortunately it's not a trivial fix.

[VolatileDream]

Using %f works, I had missed the fact it could be used for multiple files.

After trying %f: using %f causes the executed command to grow proportional to the file list, when this is hundreds of files you can start to get close to limit of what system(3) will accept. In that case, the actual tup rule is more likely to be: : out/* |> find out/ -type f | xargs ... |> file or similar.

tup fails cleanly when I tried to explicitly induce failure with 10k files in the argument list.

I am eventually going to run into the case where the argument list is too long, which makes %f a short term solution. I've also looked into using run ./script.sh but that won't work since the files are not in the root folder.

Another thing worth noting: if I follow the steps above, but after adding ghi.in I also edit a file that has already had its .out rule run, then I run tup, ghi.in is correctly picked up in that run, and all runs following.

I hope this helps.