task: exit on error

[utkarshgupta137]

Currently, rust-parallel doesn't exit (or even warn about) when a task exits unsuccessfully. I've changed this behavior in my fork: https://github.com/utkarshgupta137/rust-parallel/commit/a39253d1f3ea7eb225dfeded7dd432f53e3823f6. I guess to add it to this crate, we'll probably want to add another option such as --keep-going. But it should still exit with an error status. What do you think?

[aaronriekenberg]

Agree to add more logging when a child process fails. Currently there is only 1 debug log of exit status not enabled by default https://github.com/aaronriekenberg/rust-parallel/blob/main/src/command.rs#L64

Also I like the idea to have option to exit rust-parallel with non-0 status when any child process fails. For backwards compatibility we probably want a new option like --exit-on-error that keeps current behavior by default.

Thanks for reporting and the link to your fork @utkarshgupta137 !

[utkarshgupta137]

I feel like this is more of a bug. Almost all the tools have a --keep-going option rather than an --exit-on-error function. I left some jobs running one night & I was very surprised when the jobs continued after the initial failure.

[aaronriekenberg]

Both GNU parallel and xargs will keep going even if a command exits unsuccessfully. Both return non-zero exit status when a job fails. rust-parallel also keeps going but always returns 0 exit status.

For GNU parallel the default for --halt-on-error is never https://www.gnu.org/software/parallel/parallel.html#exit-status. This can be changed to now,fail=1 to fail immediately on first job failure.

For xargs by default it keeps going when commands fail but if a command fails with special status 255 it terminates immediately: https://linux.die.net/man/1/xargs If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error message is issued on stderr when this happens.

@utkarshgupta137 Could you provide an example command line for how you are running your job so that it exits on first error with other tools? I might be missing something. Thanks!

Some small tests of cat for non-existing files - here each cat command exits with status 1:

$ head -3 /usr/share/dict/words  | parallel cat
cat: A: No such file or directory
cat: AA: No such file or directory
cat: AAA: No such file or directory
$ echo $?
3

$ head -3 /usr/share/dict/words  | parallel --halt now,fail=1 cat
cat: A: No such file or directory
parallel: This job failed:
cat A
$ echo $?
1

$ head -3 /usr/share/dict/words  | xargs -P4 -L1 cat
cat: A: No such file or directory
cat: AA: No such file or directory
cat: AAA: No such file or directory
$ echo $?
123

$ head -3 /usr/share/dict/words  | rust-parallel cat
/usr/bin/cat: AA: No such file or directory
/usr/bin/cat: AAA: No such file or directory
/usr/bin/cat: A: No such file or directory
$ echo $?
0

[utkarshgupta137]

I guess you're right. I was comparing rust-parallel to tools like cargo which always have halt on error, but it probably makes more sense to compare it to parallel or xargs in this case. I would definitely appreciate correct exit codes (something which my commit doesn't support).

[utkarshgupta137]

I've created a patch: https://github.com/utkarshgupta137/rust-parallel/commit/cdf1ca860613c17de4c1c8dea0dc76752c651f88. Happy to raise a PR if you don't have a different approach in mind.

[aaronriekenberg]

Yes @utkarshgupta137 I would be happy to review a PR with this patch.

Thanks!

[aaronriekenberg]

Fixed in https://github.com/aaronriekenberg/rust-parallel/pull/14

[utkarshgupta137]

14 just exits with the correct error code. I would still like an option for halting on error.

[aaronriekenberg]

Sure - I still like the idea above to add CLI argument --exit-on-error option to exit when the first command failure happens.

This option would be disabled by default for backwards compatibility, also this is similar to default behavior in xargs and GNU Parallel.

[aaronriekenberg]

@utkarshgupta137 I think this issue can now be closed, let me know what you think.

Thanks again for your PR.

Updates in version 1.15.0 are below:

• Improve error handling and add to manual https://github.com/aaronriekenberg/rust-parallel/wiki/Manual#error-handling
  • Log errors for commands and at end of main log number of failing commands.
  • If at least one command fails rust-parallel exits with status 1
• Add option --exit-on-error to specify rust-parallel exits with status 1 immediately when first command failure is seen.

[utkarshgupta137]

Thanks will test & re-open in case of issues!