It seems that the last remaining issue from #225 is simply that for the very largest of our clusters we end up having unique_id lists so long they max out some length limit for command line invocations. For some bizzare reason this wasn't getting caught in the normal error handling process before (along the lines of, when running with the -k option, some errors don't bubble up the way they're supposed to and end up showing up as Internal Error: no cycle found for node output/laura-mb-2-v3-septmpts/QA255/BF520-l-M9-isub-3/partition/cluster0/min_adcl-dnaml/pruned_ids.txt (<SCons.Node.FS.File object at 0x1822cae0>) in state pending). These long commands also clog up the logging/monitoring flow, and I've been wanting to fix this as it is.
The solution is to write all the ids out to a file, instead of embedding them directly in the process_partis.py command line call, and have the script read in that list from the file.
It seems that the last remaining issue from #225 is simply that for the very largest of our clusters we end up having unique_id lists so long they max out some length limit for command line invocations. For some bizzare reason this wasn't getting caught in the normal error handling process before (along the lines of, when running with the
-koption, some errors don't bubble up the way they're supposed to and end up showing up asInternal Error: no cycle found for node output/laura-mb-2-v3-septmpts/QA255/BF520-l-M9-isub-3/partition/cluster0/min_adcl-dnaml/pruned_ids.txt (<SCons.Node.FS.File object at 0x1822cae0>) in state pending). These long commands also clog up the logging/monitoring flow, and I've been wanting to fix this as it is.The solution is to write all the ids out to a file, instead of embedding them directly in the
process_partis.pycommand line call, and have the script read in that list from the file.