align creates its own output director(ies) using makedirs, but the other commands don't (since their outputs are individual files).
This leads to the problem of brittleness: we run a command for a day or something (outputting to directorythatdoesntexist/out-file.txt), and then at the end -- just as it is about to dump its results to a file -- Python says FileNotFoundError: [Errno 2] No such file or directory.
We can get around this by removing the "basename" from each output filepath, and then calling makedirs on these with exist_ok=True. There will be annoying race conditions, but this should be good enough for most cases.
Update: for what it's worth, the current behavior isn't that bad. At least for the naive calling stuff, we write to these files multiple times, so users should see the error come up pretty quickly. Still would be nice to fix (or at least fail early for) tho.
aligncreates its own output director(ies) using makedirs, but the other commands don't (since their outputs are individual files).This leads to the problem of brittleness: we run a command for a day or something (outputting to
directorythatdoesntexist/out-file.txt), and then at the end -- just as it is about to dump its results to a file -- Python saysFileNotFoundError: [Errno 2] No such file or directory.We can get around this by removing the "basename" from each output filepath, and then calling makedirs on these with
exist_ok=True. There will be annoying race conditions, but this should be good enough for most cases.Update: for what it's worth, the current behavior isn't that bad. At least for the naive calling stuff, we write to these files multiple times, so users should see the error come up pretty quickly. Still would be nice to fix (or at least fail early for) tho.