Currently, the contract for deliver() means that users have no indication if there are failed files beyond an error message printed to the console/notebook. There is no programmatic way of knowing this from user code. (The most extreme example of this is if all jobs in an uproot-raw request fail due to a missing branch in the input file, for example, the user will see an empty list for the return value!)
I would propose to add a "strict mode" which would raise an exception for a transform if there are failed files, rather than returning a partial list of the files that do succeed. I would prefer that this become the default; this would technically be a breaking change, and would actually affect the AGC scripts (which currently refer to nonexistent files). But it's a real data integrity problem if we don't address this.
ponyisi
commented
1 week ago
Currently, the contract for
deliver()means that users have no indication if there are failed files beyond an error message printed to the console/notebook. There is no programmatic way of knowing this from user code. (The most extreme example of this is if all jobs in anuproot-rawrequest fail due to a missing branch in the input file, for example, the user will see an empty list for the return value!)I would propose to add a "strict mode" which would raise an exception for a transform if there are failed files, rather than returning a partial list of the files that do succeed. I would prefer that this become the default; this would technically be a breaking change, and would actually affect the AGC scripts (which currently refer to nonexistent files). But it's a real data integrity problem if we don't address this.