mulled-build hangs even when error reported

[marcelm]

I was finally able to find out why #3085 would hang when building. Running mulled-build locally and by hand, I got these error messages:

[Dec  7 14:39:30] SOUT Linking packages ...
[Dec  7 14:39:33] SERR /usr/local/bin/.gcc-post-link.sh: line 10: lsb_release: command not found
[Dec  7 14:39:33] SERR /usr/local/bin/.gcc-post-link.sh: line 49: ld: command not found
[gsCouldn't locate crtXXX.o in default library search paths. You may not have it  at all. It is usually packaged in libc6-dev/glibc-devel packages. We will try  to locate crtXXX.o with system installed gcc...
[Dec  7 14:39:33] SOUT There is no /usr/bin/gcc
[Dec  7 14:39:33] SERR *** Can't install the gcc package unless your system has crtXXX.o. ***
[Dec  7 14:39:33] SERR Error: Error: post-link failed for: gcc-4.8.5-7

This finally made me realize that I had incorrectly put libgcc into the build requirements and gcc into the run requirements. Swapping them fixes the error.

Edit: After that error message, the mulled-build process hangs - it does not return to the shell prompt und uses 0% CPU.

However, this bug is about mulled-build hanging at all (in addition to there being no output while it does so, already reported in #3130). It should not do that.

[bgruening]

But this indicates that you are missing some runtime dependencies. Why does this package needs gcc during runtime, and can you solve it by defining gcc as runtime dependency.

The logging is actually a general issue afaik, travis is not showing logs during the run and if it hangs you will not see any logs.

[marcelm]

Sorry, I should have been more terse for the fast readers out there. My last sentence is relevant:

this bug is about mulled-build hanging at all (in addition to there being no output while it does so, already reported in #3130). It should not do that.

The dependencies are not the issue.

[bgruening]

@marcelm

The issue is exactly what you have pasted:

[Dec  7 14:39:30] SOUT Linking packages ...
[Dec  7 14:39:33] SERR /usr/local/bin/.gcc-post-link.sh: line 10: lsb_release: command not found
[Dec  7 14:39:33] SERR /usr/local/bin/.gcc-post-link.sh: line 49: ld: command not found
[gsCouldn't locate crtXXX.o in default library search paths. You may not have it  at all. It is usually packaged in libc6-dev/glibc-devel packages. We will try  to locate crtXXX.o with system installed gcc...
[Dec  7 14:39:33] SOUT There is no /usr/bin/gcc
[Dec  7 14:39:33] SERR *** Can't install the gcc package unless your system has crtXXX.o. ***
[Dec  7 14:39:33] SERR Error: Error: post-link failed for: gcc-4.8.5-7

I'm not sure what is going on with your package here, but this is the error and it seems that this error is somehow not terminating as it should. On a mulled-site we can implement something like kill the container after a certain time period, but time period should it be.

I found this as well: https://github.com/ContinuumIO/docker-images/issues/21

So maybe this is even a conda bug, that does not terminate. I'm not sure how we can convince travis to output on-demand - we probably need to put everything directly into the .tavis.yaml file and not in a separate script.

[marcelm]

This is the process tree in case it helps:

\_ python ./simulate-travis.py --packages=sqt --loglevel=debug
     \_ /bin/bash scripts/travis-run.sh
         \_ /mc3/bin/python /mc3/bin/bioconda-utils build recipes config.yml --docker --loglevel=info --mulled-test --packages=sqt --loglevel=debug
             \_ /mc3/bin/python /mc3/bin/mulled-build build-and-test sqt=0.8.0=py35_0 --name-override tmp --test python -c "import sqt" --extra-channel conda-forge --extra-channel defaults --extra-channel r --extra-channel bioconda --extra-channel file://mc3/conda-bld
                 \_ /bin/sh -c ./involucro -v=3 -f /mc3/lib/python3.5/site-packages/galaxy/tools/deps/mulled/invfile.lua -set CHANNELS='bioconda,file://mc3/conda-bld' -set TEST='python -c "import sqt"' -set TARGETS='sqt=0.8.0=py35_0' -set REPO='quay.io/mulled/tmp' -set BINDS='build/dist:/usr/local/,/mc3/conda-bld:/mc3/conda-bld' build-and-test
                     \_ ./involucro -v=3 -f /mc3/lib/python3.5/site-packages/galaxy/tools/deps/mulled/invfile.lua -set CHANNELS=bioconda,file://mc3/conda-bld -set TEST=python -c "import sqt" -set TARGETS=sqt=0.8.0=py35_0 -set REPO=quay.io/mulled/tmp -set BINDS=build/dist:/usr/local/,/mc3/conda-bld:/mc3/conda-bld build-and-test

It’s been idle for an hour.

I only see that some process that encounters an error does not properly terminate. Maybe this is an upstream bug, but since I don’t know how the bioconda build system works, I’m reporting the problem here.

[bgruening]

I will close this for now. I guess at some point we need to make travis realtime, but until then we need to debug this locally.