Parallelized logs from mega-Makefile hard to read

Due to the nature of multiple processes all writing to the same log, it is now very difficult to parse the order of events or what things failed during a run.

As an example from https://build.geneontology.org/job/geneontology/job/pipeline/job/snapshot/659/consoleText , errors and a stack trace interleaved with a download:

1070600K .......... .......... .......... .......... .......... 15% 65.1M 6m32s
1070650K ...Makefile:59: recipe for target 'target/groups/tair/tair.group' faile
d
make: *** [target/groups/tair/tair.group] Error 1
....... .......... .......... .......... .......... 15% 17.4M 6m32s
1070700K .......... .......... .......... .......... .......... 15% 29.8M 6m32s
1070750K .......... .......... .......... .......... .......... 15% 66.4M 6m32s

1060600K .......... .......... .......... .......... .......... 15% 84.1M 6m33s
1060650K .......... .......... .......... .......... .......
Unzipping /var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/pipeline/target/groups/aspgd/aspgd-src.gaf.gz

Traceback (most recent call last):
  File "mypyenv/bin/validate.py", line 440, in <module>
    cli()
  File "/var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/pipeline/mypyenv/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/pipeline/mypyenv/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/pipeline/mypyenv/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/pipeline/mypyenv/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/pipeline/mypyenv/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "mypyenv/bin/validate.py", line 386, in produce
    unzip(source_zip, source_gaf)
  File "mypyenv/bin/validate.py", line 135, in unzip
    for chunk in chunks:
  File "/var/lib/jenkins/workspace/eontology_pipeline_snapshot-OLCOSBORX7TUJKUSVDZQNZDGXCRFSUQLNESYCP3R63U6FLW5DJ2A/go-site/pipeline/mypyenv/lib/python3.5/site-packages/click/_termui_impl.py", line 257, in next
    return next(self.iter)
  File "mypyenv/bin/validate.py", line 128, in chunk_gen
    chunk = p.read(size=512 * 1024)
  File "/usr/lib/python3.5/gzip.py", line 274, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.5/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib/python3.5/gzip.py", line 461, in read
    if not self._read_gzip_header():
  File "/usr/lib/python3.5/gzip.py", line 409, in _read_gzip_header
    raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'<h')
... 15%  593K 6m33s
1060700K .......... .......... .......... .......... .......... 15% 4.62M 6m33s
1060750K .......... .......... .......... .......... .......... 15% 28.2M 6m33s

As a possible fix, perhaps some logs should be sent to separate files (as well as STDOUT). Or perhaps catch errors and output more informational failure info.

geneontology / go-site

Parallelized logs from mega-Makefile hard to read #991