wildcard globbing

[lcorbet]

Hi remakers,

Just wondering whether it would be possible to add some wildcard globbing / substitution functionality so that we can apply a function to all files in a directory, separately, but only if the input changed.

• If we had N individual IDs to process, and
• we created N input files, all following the name '_input.csv', and
• each input file was a dependency for its respective output file '_output.csv' , and
• we wanted to apply process_input() to each input file separately, with the input file as the argument, to produce _output.csv,

The reason for this is that processing all files may take 2 hours so we might not want to process all files when not all of them have changed (i.e tweaking one or two input files only). The real-world situation has more dependencies and steps so I want a build tool to look after and ensure that all the processing is up to date, but avoid reprocessing all the inputs each time a tweak is made.

Regards, Linton.

[richfitz]

See also #2; this is on the list but mostly blocked by a lack of time and a really good UI for it. Some sort of pattern matching approach is probably best but accompishing that without getting into the cryptic mess that make ends up with is the trick. My current thoughts is that getting all the files into a list (see #8 and the lists branch) could simplify processing and keep the focus on objects rather than files. But at some point this needs dealing with.

You might find the approach we took with baad useful in them meantime. There we use whisker to generate most of the remake file this file generates this very repetitive file which is included in the main remake file.

Any good ideas welcome and thanks for the prod.