search

bioconda / bioconda-recipes

Conda recipes for the bioconda channel.
https://bioconda.github.io
MIT License
1.65k stars 3.28k forks source link

circleci build failed #7492

Open kpepper opened 6 years ago

kpepper commented 6 years ago

I tried running "circleci build" in bioconda-recipes but I get a "cannot find git" error and the build fails. So I updated the bioconda-utils-build-env docker container before running:

docker pull bioconda/bioconda-utils-build-env

but that has made no difference. (have also deleted the container and re-pulled).

Same thing yesterday and today. Any ideas what the issue could be? Is git included in the latest docker container?

Thanks.

Build log:

====>> Spin up Environment Build-agent version 0.0.4666-05dad47 (2018-01-25T18:14:33+0000) Starting container bioconda/bioconda-utils-build-env using image bioconda/bioconda-utils-build-env@sha256:bb7f4dd3dfd337e097b9aa498604940901d7c58e740071cbbf415b59e312d97a

Using build environment variables: BASH_ENV=/tmp/.bash_env-localbuild-1517572626 CI=true CIRCLECI=true CIRCLE_BRANCH=artemis CIRCLE_BUILD_NUM= CIRCLE_JOB=build CIRCLE_NODE_INDEX=0 CIRCLE_NODE_TOTAL=1 CIRCLE_REPOSITORY_URL=https://github.com/kpepper/bioconda-recipes.git CIRCLE_SHA1=4c64726b799453dd6f857b41cc54e47a2ff5c65d CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1517572626 CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code

!/bin/sh

mkdir -p /root/project && cp -r /tmp/_circleci_local_build_repo/. /root/project ====>> Setup ssh

!/bin/bash -eo pipefail

mkdir -p ~/.ssh ssh-keyscan -H github.com >> ~/.ssh/known_hosts

github.com SSH-2.0-libssh_0.7.0

====>> Download common definitions

!/bin/bash -eo pipefail

curl -s https://raw.githubusercontent.com/bioconda/bioconda-common/master/common.sh > .circleci/common.sh ====>> Setup bioconda-utils

!/bin/bash -eo pipefail

.circleci/setup.sh .circleci/setup.sh: line 14: type: bioconda-utils: not found Setting up bioconda-utils... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33.4M 100 33.4M 0 0 27.6M 0 0:00:01 0:00:01 --:--:-- 30.8M PREFIX=/tmp/workspace/miniconda installing: python-3.6.1-2 ... installing: asn1crypto-0.22.0-py36_0 ... installing: cffi-1.10.0-py36_0 ... installing: conda-env-2.6.0-0 ... installing: cryptography-1.8.1-py36_0 ... installing: idna-2.5-py36_0 ... installing: libffi-3.2.1-1 ... installing: openssl-1.0.2l-0 ... installing: packaging-16.8-py36_0 ... installing: pycosat-0.6.2-py36_0 ... installing: pycparser-2.17-py36_0 ... installing: pyopenssl-17.0.0-py36_0 ... installing: pyparsing-2.1.4-py36_0 ... installing: readline-6.2-2 ... installing: requests-2.14.2-py36_0 ... installing: ruamel_yaml-0.11.14-py36_1 ... installing: setuptools-27.2.0-py36_0 ... installing: six-1.10.0-py36_0 ... installing: sqlite-3.13.0-0 ... installing: tk-8.5.18-0 ... installing: xz-5.2.2-1 ... installing: yaml-0.1.6-0 ... installing: zlib-1.2.8-3 ... installing: conda-4.3.21-py36_0 ... installing: pip-9.0.1-py36_1 ... installing: wheel-0.29.0-py36_0 ... Python 3.6.1 :: Continuum Analytics, Inc. creating default environment... installation finished. Warning: 'defaults' already in 'channels' list, moving to the top Warning: 'conda-forge' already in 'channels' list, moving to the top Warning: 'bioconda' already in 'channels' list, moving to the top Fetching package metadata ............. Solving package specifications: .

Package plan for installation in environment /tmp/workspace/miniconda:

The following NEW packages will be INSTALLED:

alabaster:                0.7.10-py36_1                 conda-forge
anaconda-client:          1.6.5-py_0                    conda-forge
argh:                     0.26.2-py36_1                 conda-forge
babel:                    2.5.1-py36_0                  conda-forge
beautifulsoup4:           4.6.0-py36_0                  conda-forge
blas:                     1.1-openblas                  conda-forge
certifi:                  2017.11.5-py36_0              conda-forge
chardet:                  3.0.4-py36_0                  conda-forge
clyent:                   1.2.2-py36_0                  conda-forge
colorlog:                 3.1.0-py36_0                  conda-forge
conda-build:              2.1.18-py36_0                 conda-forge
conda-verify:             2.0.0-py36_0                  conda-forge
decorator:                4.1.2-py36_0                  conda-forge
docutils:                 0.14-py36_0                   conda-forge
filelock:                 2.0.6-py36_0                  conda-forge
galaxy-lib:               17.9.10-py36_0                bioconda   
imagesize:                0.7.1-py36_0                  conda-forge
involucro:                1.1.2-0                       bioconda   
ipython_genutils:         0.2.0-py36_0                  conda-forge
jinja2:                   2.10-py36_0                   conda-forge
jsonschema:               2.6.0-py36_0                  conda-forge
jupyter_core:             4.4.0-py_0                    conda-forge
libgfortran:              3.0.0-1                       defaults   
markupsafe:               1.0-py36_0                    conda-forge
nbformat:                 4.4.0-py36_0                  conda-forge
networkx:                 1.11-py36_0                   conda-forge
numpy:                    1.14.0-py36_blas_openblas_200 conda-forge [blas_openblas]
openblas:                 0.2.20-7                      conda-forge
pandas:                   0.22.0-py36_0                 conda-forge
patchelf:                 0.9-2                         conda-forge
pkginfo:                  1.4.1-py36_0                  conda-forge
pyaml:                    17.12.1-py_0                  conda-forge
pycrypto:                 2.6.1-py36_1                  conda-forge
pydotplus:                2.0.2-py36_0                  conda-forge
pygithub:                 1.34-py36_0                   conda-forge
pygments:                 2.2.0-py36_0                  conda-forge
pyjwt:                    1.5.3-py_0                    conda-forge
pysocks:                  1.6.8-py36_1                  conda-forge
python-dateutil:          2.6.1-py36_0                  conda-forge
pytz:                     2017.3-py_2                   conda-forge
pyyaml:                   3.12-py36_1                   conda-forge
snowballstemmer:          1.2.1-py36_0                  conda-forge
sphinx:                   1.6.6-py36_0                  conda-forge
sphinx_rtd_theme:         0.2.4-py36_0                  conda-forge
sphinxcontrib-websupport: 1.0.1-py36_0                  conda-forge
traitlets:                4.3.2-py36_0                  conda-forge
typing:                   3.5.2.2-py36_0                bioconda   
urllib3:                  1.22-py36_0                   conda-forge

The following packages will be UPDATED:

conda:                    4.3.21-py36_0                 defaults    --> 4.3.33-py36_0 conda-forge
requests:                 2.14.2-py36_0                 defaults    --> 2.18.4-py36_1 conda-forge
six:                      1.10.0-py36_0                 defaults    --> 1.11.0-py36_1 conda-forge

involucro-1.1. 100% || Time: 0:00:01 2.06 MB/ss libgfortran-3. 100% || Time: 0:00:00 13.11 MB/ss patchelf-0.9-2 100% || Time: 0:00:00 3.90 MB/ss openblas-0.2.2 100% || Time: 0:00:02 6.32 MB/ss alabaster-0.7. 100% || Time: 0:00:00 89.49 kB/ss argh-0.26.2-py 100% || Time: 0:00:00 208.97 kB/ss beautifulsoup4 100% || Time: 0:00:00 543.42 kB/ss blas-1.1-openb 100% || Time: 0:00:00 760.10 kB/ss certifi-2017.1 100% || Time: 0:00:00 1.19 MB/ss chardet-3.0.4- 100% || Time: 0:00:00 1.11 MB/ss colorlog-3.1.0 100% || Time: 0:00:00 121.11 kB/ss decorator-4.1. 100% || Time: 0:00:00 192.75 kB/ss docutils-0.14- 100% || Time: 0:00:00 943.85 kB/ss filelock-2.0.6 100% || Time: 0:00:00 3.55 MB/ss imagesize-0.7. 100% || Time: 0:00:00 2.40 MB/ss ipython_genuti 100% || Time: 0:00:00 12.91 MB/ss jsonschema-2.6 100% || Time: 0:00:00 22.37 MB/ss markupsafe-1.0 100% || Time: 0:00:00 16.01 MB/ss pkginfo-1.4.1- 100% || Time: 0:00:00 13.53 MB/ss pycrypto-2.6.1 100% || Time: 0:00:00 1.15 MB/ss pysocks-1.6.8- 100% || Time: 0:00:00 7.36 MB/ss pytz-2017.3-py 100% || Time: 0:00:00 1.33 MB/ss pyyaml-3.12-py 100% || Time: 0:00:00 1.20 MB/ss six-1.11.0-py3 100% || Time: 0:00:00 10.18 MB/ss snowballstemme 100% || Time: 0:00:00 1.01 MB/ss sphinx_rtd_the 100% || Time: 0:00:01 991.79 kB/ss sphinxcontrib- 100% || Time: 0:00:00 18.79 MB/ss typing-3.5.2.2 100% || Time: 0:00:00 15.09 MB/ss babel-2.5.1-py 100% || Time: 0:00:04 1.02 MB/ss clyent-1.2.2-p 100% || Time: 0:00:00 11.71 MB/ss conda-verify-2 100% || Time: 0:00:00 7.53 MB/ss galaxy-lib-17. 100% || Time: 0:00:00 1.72 MB/ss jinja2-2.10-py 100% || Time: 0:00:00 1.14 MB/ss networkx-1.11- 100% || Time: 0:00:00 1.83 MB/ss numpy-1.14.0-p 100% || Time: 0:00:07 1.33 MB/ss pyaml-17.12.1- 100% || Time: 0:00:00 107.80 kB/ss pydotplus-2.0. 100% || Time: 0:00:00 249.50 kB/ss pygments-2.2.0 100% || Time: 0:00:01 1.19 MB/ss python-dateuti 100% || Time: 0:00:00 415.93 kB/ss traitlets-4.3. 100% || Time: 0:00:00 824.08 kB/ss jupyter_core-4 100% || Time: 0:00:00 8.48 MB/ss pandas-0.22.0- 100% || Time: 0:00:14 1.84 MB/ss nbformat-4.4.0 100% || Time: 0:00:00 868.02 kB/ss pyjwt-1.5.3-py 100% || Time: 0:00:00 92.95 kB/ss pygithub-1.34- 100% || Time: 0:00:01 1.17 MB/ss urllib3-1.22-p 100% || Time: 0:00:00 384.65 kB/ss requests-2.18. 100% || Time: 0:00:00 382.28 kB/ss anaconda-clien 100% || Time: 0:00:00 735.46 kB/ss conda-4.3.33-p 100% || Time: 0:00:00 1.06 MB/ss sphinx-1.6.6-p 100% || Time: 0:00:01 1.17 MB/ss conda-build-2. 100% || Time: 0:00:00 685.95 kB/ss Collecting git+https://github.com/bioconda/bioconda-utils.git@v0.12.0 Cloning https://github.com/bioconda/bioconda-utils.git (to v0.12.0) to /tmp/pip-mtd470q7-build Error [Errno 2] No such file or directory: 'git' while executing command git clone -q https://github.com/bioconda/bioconda-utils.git /tmp/pip-mtd470q7-build Cannot find command 'git' Error: Exited with code 1 Step failed Task failed

johanneskoester commented 6 years ago

Should be fixed with PR #7509. Feel free to reopen if it still does not work.

karl616 commented 6 years ago

I still have this problem. I could solve it by inserting a: conda install git in .circleci/setup.sh

Could the fact that I have to run circleci with sudo have an impact on this? I only have user-specific installation of conda... no installation for root... and I can see that the setup-script is referring to the /root folder. I'm still not versed enough with containers to pick out when it is /root on the host or in the container.

johanneskoester commented 6 years ago

Since 2 days, bioconda-utils depends on git, such that it should now be installed automatically. Can you do a docker pull bioconda/bioconda-utils-build-env before circleci build? Does that help, without the manual change to install git?

karl616 commented 6 years ago

I tried to do the pull before commenting. Then it didn't work without the manual fix. Today it does. I cannot explain why, but I'm happy with the result. Thanks!

mbhall88 commented 5 years ago

I am seeing this issue also.
I ran docker pull bioconda/bioconda-utils-build-env today.
Then circleci build and receive the following. 

Docker image digest: sha256:a6decb2448f4371c64efde9dad90cc4efe0a72216ba7dc288fc89341d0b0c9d2
====>> Spin up Environment
Build-agent version 0.1.1809-71691ee9 (2019-02-08T15:06:57+0000)
Starting container bioconda/bioconda-utils-build-env
  using image bioconda/bioconda-utils-build-env@sha256:deafc3d76b2239cd1802d4012cbc6673850ed7e3900da5185d12aa51f5831734

Using build environment variables:
  BASH_ENV=/tmp/.bash_env-localbuild-1549639793
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=mykrobe
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://github.com/mbhall88/bioconda-recipes.git
  CIRCLE_SHA1=4ed98aea65fa1375c71fef84134b22e0ca5156e1
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1549639793
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /root/project
/bin/bash: git: command not found
tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
Error: Exited with code 2
Step failed
Error: runner failed
{"Runner":true,"level":"error","msg":"runner failed","task-id":"localbuild-1549639793","time":"2019-02-08T15:29:56Z"}
Task failed
Error: task failed

I am wondering whether this has something to do with commit https://github.com/bioconda/bioconda-utils/commit/d446d42c876ca44dfb246183de4319671d8a25d6#diff-3254677a7917c6c01f55212f86c57fbf changing the base image from conda-forge?

holtjma commented 5 years ago

I'm having the exact same issue, @mbhall88. I'm new, so is there some work around and should we reopen this issue?

Takadonet commented 5 years ago

Also having same issue with the latest docker of bioconda/bioconda-utils-build-env where git is not found.

edgano commented 5 years ago

+1 with the same issue. I'm using this container:using image bioconda/bioconda-utils-build-env@sha256:027b5c49311e9f08ac52ff3e688ea17b63a8b9bb3495a4b947a65e135a5520b2

  #!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /root/project
/bin/bash: git: command not found
seb-mueller commented 5 years ago

I've got the same error:

$ docker pull bioconda/bioconda-utils-build-env                                                                                                                                                       
Using default tag: latest
latest: Pulling from bioconda/bioconda-utils-build-env
Digest: sha256:907795f9f5a5635987b1ba0b93351d6ecba5e4eab70bddf4fe5b75fb99ff61da
Status: Image is up to date for bioconda/bioconda-utils-build-env:latest

Followed by:

circleci build              
...                                                                                                                                                                       
/bin/bash: git: command not found

Note sure how to best proceed from here? Or maybe this still not fixed?

cpwardell commented 5 years ago

I'm seeing the same error:

`$ circleci build Docker image digest: sha256:60a98b5fc00a4302c46be2217d6e4e4eb8491d466fa62fed14362a85100bcd68 ====>> Spin up Environment Build-agent version 1.0.15283-35354d0e (2019-09-09T12:09:35+0000) Docker Engine Version: 18.09.3 Kernel Version: Linux 6730d5672d20 3.10.0-514.21.1.el7.x86_64 #1 SMP Thu May 25 17:04:51 UTC 2017 x86_64 Linux Starting container bioconda/bioconda-utils-build-env using image bioconda/bioconda-utils-build-env@sha256:524360f41b2e90f9127f8313ff189b0c7fed7c5717f2e15429353a89f337f985

Using build environment variables BASH_ENV=/tmp/.bash_env-localbuild-1568386778 CI=true CIRCLECI=true CIRCLE_BRANCH=fings CIRCLE_BUILD_NUM= CIRCLE_JOB=build CIRCLE_NODE_INDEX=0 CIRCLE_NODE_TOTAL=1 CIRCLE_REPOSITORY_URL=https://github.com/cpwardell/bioconda-recipes.git CIRCLE_SHA1=9a73ad6de226f2e19c9561883e3aab223903df34 CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1568386778 CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code

!/bin/bash -eo pipefail

mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /root/project && cp -a /tmp/_circleci_local_build_repo/.git /root/project {"Runner":true,"level":"error","msg":"Error copying logs: read /dev/ptmx: input/output error","task-id":"localbuild-1568386778","time":"2019-09-13T14:59:41Z"} /bin/bash: git: command not found Error: Exited with code 127 Step failed Error: runner failed (exited with 101) Task failed Error: task failed `

bgruening commented 5 years ago

You need to have git installed.

seb-mueller commented 5 years ago

@bgruening , what do you mean by "you"? The local host system or the docker container? As for my part git is installed (/usr/bin/git) To have it installed in the container, it would be helpful to be a bit more specific on how to do this or how to point to the local install from within the container. As far as I know this should have been included in the docker image.

nathanweeks commented 5 years ago

I started to look into this within the last couple weeks, as I see the same problem on macOS; IIRC git is installed in the container image, but for some reason it is not in the PATH during execution.

nathanweeks commented 5 years ago

git is in /opt/conda/bin/ ; the following patch, which modifyies .circleci/setup.yml so that the conda environment is activated before invoking git allows it to be available in the PATH during circleci build, appears to fix circleci build on macOS:

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 229ab2dc0..895592b41 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -78,14 +78,14 @@ jobs:
     docker:
       - image: bioconda/bioconda-utils-build-env
     steps:
-      - checkout
+      - run: echo ". /opt/conda/etc/profile.d/conda.sh" >> $BASH_ENV
+      - run: echo "conda activate" >> $BASH_ENV
+      - run: mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /root/project && cp -a /tmp/_circleci_local_build_repo/.git /root/project
       - run:
           name: Setup ssh
           command: |
             mkdir -p ~/.ssh
             ssh-keyscan -H github.com >> ~/.ssh/known_hosts
-      - run: echo ". /opt/conda/etc/profile.d/conda.sh" >> $BASH_ENV
-      - run: echo "conda activate" >> $BASH_ENV
       - *common
       - *setup
       - run:

The above patch also replaces the circleci checkout command to work around this issue: https://discuss.circleci.com/t/local-checkout-fails-using-cli-tar-terminates-with-signal-13/28632 (an alternative workaround is presented here here: https://github.com/CircleCI-Public/circleci-cli/issues/281). I'm not familiar-enough with CircleCI to know that workaround is future-proof, though.