Having a separate function handling building input and output filenames was a bit of a pain to deal with and prevented many desirable features, like file hashing, from being easy or possible.
This is a pretty massive change that removes the need for the paths function and allows all producers to just declare their input file patterns without the need for declaring their output files. This also changes the function signature for functions called by producers, instead of being passed in an input file typeddict and an output file typeddict, they now receive an input file typedict and a matchgroup dict. They also now must return a list of file paths that they created.
Overall this drastically decreases the complexity of using the Producer module, but it does increase the complexity under the hood. And mandates that there be at least one cached file that stores some information about previous runs. However this is all automatically configured and should be transparent to the user.
Lastly we have removed the concept of categories. This seemed like a good idea given the experience we had when first writing it, however in practice the category functionality was clunky and never really used. Producers now have names to separate them from each other. Each producer added to a scheduler must have a unique name from all the other producers added to that scheduler.
Having a separate function handling building input and output filenames was a bit of a pain to deal with and prevented many desirable features, like file hashing, from being easy or possible.
This is a pretty massive change that removes the need for the
pathsfunction and allows all producers to just declare their input file patterns without the need for declaring their output files. This also changes the function signature for functions called by producers, instead of being passed in an input file typeddict and an output file typeddict, they now receive an input file typedict and a matchgroup dict. They also now must return a list of file paths that they created.Overall this drastically decreases the complexity of using the Producer module, but it does increase the complexity under the hood. And mandates that there be at least one cached file that stores some information about previous runs. However this is all automatically configured and should be transparent to the user.
Lastly we have removed the concept of categories. This seemed like a good idea given the experience we had when first writing it, however in practice the category functionality was clunky and never really used. Producers now have names to separate them from each other. Each producer added to a scheduler must have a unique name from all the other producers added to that scheduler.