bigcrush: a script to run bigcrush in parallel

[jepler]

Since most systems have lots of parallelism available, this is likely to make a bigcrushall run complete in much less time than the "days" threatened by README.md.

No additional bashisms are used.

The printf(1) usage should be POSIX compliant.

While getconf(1) is portable, the getconf(1) usage is not portable, because POSIX does not sysconf does not define _SC_NPROCESSORS_ONLN. However, GNU (Linux), FreeBSD, NetBSD, OpenBSD, and MacOS all seem to.

While POSIX doesn't seem to define xargs' -P or -0 flags, in practice GNU (Linux), FreeBSD, NetBSD, OpenBSD, and MacOS all seem to.

If it's judged to be portable enough, then perhaps this should replace the bigcrushall.sh script, rather than be added alongside it.

References consulted:

Portability of getconf, printf, xargs in general: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/getconf.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html

Widespread availablity of getconf _SC_NPROCESSORS_ONLN: http://man7.org/linux/man-pages/man3/sysconf.3.html http://netbsd.gw.com/cgi-bin/man-cgi?sysconf++NetBSD-current https://man.openbsd.org/sysconf.3 https://www.freebsd.org/cgi/man.cgi?sysconf https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man3/sysconf.3.html

Widespread availablity of xargs -P, -0 as extensions to POSIX: http://man7.org/linux/man-pages/man1/xargs.1.html https://www.freebsd.org/cgi/man.cgi?xargs http://netbsd.gw.com/cgi-bin/man-cgi?xargs++NetBSD-current https://man.openbsd.org/xargs.1 https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/xargs.1.html

[lemire]

Would you add your name to the script? This makes it easier for me to track credit later.

Also, if you have time, adding a few lines to the README.md so that people can know how to call your script and so forth, would be nice (but optional).

[jepler]

I've updated this pull request based on your suggestions, and also made a change so that the list of tests isn't repeated in two places, which would otherwise create a maintenance burden.

[lemire]

Fantastic!

[jepler]

It still took about 4 days, 40000 CPU minutes. eep!

[lemire]

I trust you have the same results I report?

[jepler]

Yes, after eliminating HOST and CPU time reports, there are no other differences in the .log files.

$ git diff -U0 | grep -v "CPU time" | grep '^[-+][^-+]' | sort -u
-HOST = haswell, Linux
+HOST = rat, Linux