search

bioconda / bioconda-utils

Utilities for building and managing bioconda recipes
MIT License
97 stars 133 forks source link

git not found in local circleci build #505

Open skurscheid opened 5 years ago

skurscheid commented 5 years ago

I am seeing the same issues described in #13932 ...

Local testing with circleci (installed via snap) fails. For testing purposed I have setup a clean Ubuntu 16.04 VM, installed git and docker (via snap), forked bioconda-recipes and created a branch in my fork

Error message from circleci build:

skurscheid@biocondaDev:~/bioconda-recipes$ circleci build
Docker image digest: sha256:58c0be919018ee5ef1a34509a25fcf0b2c57ea1322ab2167632456d316c7b937
====>> Spin up Environment
Build-agent version 1.0.10572-3ce00c85 (2019-04-15T22:09:28+0000)
Docker Engine Version: 18.06.1-ce
Kernel Version: Linux 27d0087d5c23 4.15.0-1041-azure #45-Ubuntu SMP Fri Mar 15 14:41:00 UTC 2019 x86_64 Linux
Starting container bioconda/bioconda-utils-build-env
  using image bioconda/bioconda-utils-build-env@sha256:359113e96c2c6b85583c04c0b4f0b084019677ba2a1c9d48bc15e467ba2f9964

Using build environment variables
  BASH_ENV=/tmp/.bash_env-localbuild-1555382943
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=genrich
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=git@github.com:skurscheid/bioconda-recipes.git
  CIRCLE_SHA1=46c2ea8a9634dface074a4fbfccaf9dec3b7b334
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1555382943
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /root/project && cp -a /tmp/_circleci_local_build_repo/.git /root/project
/bin/bash: git: command not found
tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
Error: Exited with code 2
Step failed
Error: runner failed (exited with 101)
Task failed
Error: task failed

But git is present in docker image:

skurscheid@biocondaDev:~/bioconda-recipes$ docker run -it bioconda/bioconda-utils-build-env git --version
git version 2.14.2

However the binary is not where the circleci script expects it to be:

skurscheid@biocondaDev:~/bioconda-recipes$ docker run -it bioconda/bioconda-utils-build-env which git
/opt/conda/bin/git
dpryan79 commented 5 years ago

Can you just use bioconda-utils directly? I find it to be less error-prone than the circleci client. The step throwing the error doesn't even need to be run locally.

skurscheid commented 5 years ago

@dpryan79 thanks for getting back. I have tried the local test as described in the "contributing" instructions:

23:50:36 BIOCONDA INFO Generating DAG
Loading Recipes: 100%|███████████████████████████████████████████| 1/1 [00:00<00:00, 80.41it/s]
23:50:36 BIOCONDA INFO Building and testing 1 recipes in total
23:50:36 BIOCONDA INFO Recipes to build:
genrich
23:50:36 BIOCONDA INFO Building and testing subdag 1 of 1 (1 recipes)
23:50:36 BIOCONDA INFO Determining expected packages
Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
Adding in variants from /tmp/miniconda/miniconda/conda_build_config.yaml
Adding in variants from /tmp/miniconda/miniconda/lib/python3.6/site-packages/bioconda_utils/bioconda_utils-conda_build_config.yaml
Downloading: 100%|████████████████████████████████████████████| 9/9 [00:08<00:00,  1.42s/files]
Attempting to finalize metadata for genrich
Collecting package metadata: ...working... done
Solving environment: ...working... doneB/s]
Collecting package metadata: ...working... done
Solving environment: ...working... done
Collecting package metadata: ...working... done
Solving environment: ...working... done
23:51:20 BIOCONDA INFO BUILD START recipes/genrich
23:52:01 BIOCONDA ERROR COMMAND FAILED: docker run -t --net host --rm -v /tmp/tmpmawbalsc/build_script.bash:/opt/build_script.bash -v /tmp/miniconda/miniconda/conda-bld:/opt/host-conda-bld -v /home/skurscheid/bioconda-recipes/recipes/genrich:/opt/recipe -e CONDA_SHLVL=2 -e CONDA_PROMPT_MODIFIER=(base)  -e CONDA_EXE=/tmp/miniconda/miniconda/bin/conda -e CONDA_PREFIX_1=/home/skurscheid/miniconda3 -e LANG=en_US.UTF-8 -e CONDA_PYTHON_EXE=/tmp/miniconda/miniconda/bin/python -e CONDA_DEFAULT_ENV=base -e CONDA_ROOT=/tmp/miniconda/miniconda -e CONDA_ALLOW_SOFTLINKS=false -e HOST_USER_ID=1000 tmp-bioconda-builder /bin/bash /opt/build_script.bash
23:52:01 BIOCONDA ERROR STDOUT+STDERR:

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.6/site-packages/conda/cli/main.py", line 138, in main
        return activator_main()
      File "/opt/conda/lib/python3.6/site-packages/conda/activate.py", line 945, in main
        print(activator.execute(), end='')
      File "/opt/conda/lib/python3.6/site-packages/conda/activate.py", line 177, in execute
        return getattr(self, self.command)()
      File "/opt/conda/lib/python3.6/site-packages/conda/activate.py", line 154, in activate
        builder_result = self.build_activate(self.env_name_or_prefix)
      File "/opt/conda/lib/python3.6/site-packages/conda/activate.py", line 279, in build_activate
        return self._build_activate_stack(env_name_or_prefix, False)
      File "/opt/conda/lib/python3.6/site-packages/conda/activate.py", line 337, in _build_activate_stack
        deactivate_scripts = self._get_deactivate_scripts(old_conda_prefix)
      File "/opt/conda/lib/python3.6/site-packages/conda/activate.py", line 605, in _get_deactivate_scripts
        prefix, 'etc', 'conda', 'deactivate.d', '*' + self.script_extension
      File "/opt/conda/lib/python3.6/posixpath.py", line 80, in join
        a = os.fspath(a)
    TypeError: expected str, bytes or os.PathLike object, not NoneType

`$ /opt/conda/bin/conda shell.posix activate base`

  environment variables:
                 CIO_TEST=<not set>
    CONDA_ALLOW_SOFTLINKS=false
        CONDA_DEFAULT_ENV=base
                CONDA_EXE=/opt/conda/bin/conda
           CONDA_PREFIX_1=/home/skurscheid/miniconda3
    CONDA_PROMPT_MODIFIER=(base)
         CONDA_PYTHON_EXE=/opt/conda/bin/python
               CONDA_ROOT=/tmp/miniconda/miniconda
              CONDA_SHLVL=2
                     PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : None
            shell level : 2
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.6.12
    conda-build version : 3.17.8
         python version : 3.6.7.final.0
       base environment : /opt/conda  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.6.12 requests/2.20.1 CPython/3.6.7 Linux/4.15.0-1041-azure centos/6.10 glibc/2.12
                UID:GID : 0:0
             netrc file : None
           offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?

Timeout reached. No report sent.

It does not seem to be linked to the actual recipe, but rather my local installation. As I mentioned in the OP, this is on a clean VM, with miniconda3 and git installed by me... From my perspective it seems to be linked to docker, but docker itself runs:

(base) skurscheid@biocondaDev:~/bioconda-recipes$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/
dpryan79 commented 5 years ago

You'll need to unset any CONDA_* environment variables. I'll make a PR on bioconda-utils to hopefully resolve that.

dpryan79 commented 5 years ago

I've moved this issue over to bioconda-utils so it's a bit easier to track.

nick-youngblut commented 5 years ago

I'm running into the same issue on macOS 10.14.6, and

You'll need to unset any CONDA_* environment variables

...doesn't help.

Full output:

$ circleci build
Docker image digest: sha256:60a98b5fc00a4302c46be2217d6e4e4eb8491d466fa62fed14362a85100bcd68
====>> Spin up Environment
Build-agent version 1.0.15283-35354d0e (2019-09-09T12:09:35+0000)
Docker Engine Version: 18.09.2
Kernel Version: Linux 56fa995ee690 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 Linux
Starting container bioconda/bioconda-utils-build-env
  using image bioconda/bioconda-utils-build-env@sha256:907795f9f5a5635987b1ba0b93351d6ecba5e4eab70bddf4fe5b75fb99ff61da

Using build environment variables
  BASH_ENV=/tmp/.bash_env-localbuild-1568183011
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=antismash
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=git@github.com:nick-youngblut/bioconda-recipes.git
  CIRCLE_SHA1=250723bc614500d47737575f74650272e7d03784
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1568183011
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /root/project && cp -a /tmp/_circleci_local_build_repo/.git /root/project
{"Runner":true,"level":"error","msg":"Error copying logs: read /dev/ptmx: input/output error","task-id":"localbuild-1568183011","time":"2019-09-11T06:23:35Z"}
/bin/bash: git: command not found
Error: Exited with code 127
Step failed
Error: runner failed (exited with 101)
Task failed
Error: task failed
$ docker run -it bioconda/bioconda-utils-build-env git --version
git version 2.14.2
blinard-BIOINFO commented 5 years ago

Hi, Same issue here, on ubuntu 18.04.2 LTS. Using a snap installation of docker circleci. The docker image contains the git installation but the script searches for a different path.

$ docker run -it bioconda/bioconda-utils-build-env which git
/opt/conda/bin/git
$ circleci build
Docker image digest: sha256:ff905e9755646ca6c10dc7ffed95a3325a9aa180274f3efe53f3fefd12c4bff3
====>> Spin up Environment
Build-agent version 1.0.15767-4b78b3bd (2019-09-18T18:07:37+0000)
Docker Engine Version: 18.06.1-ce
Kernel Version: Linux dc7ba6abdc2c 4.15.0-62-generic #69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019 x86_64 Linux
Starting container bioconda/bioconda-utils-build-env
  using image bioconda/bioconda-utils-build-env@sha256:524360f41b2e90f9127f8313ff189b0c7fed7c5717f2e15429353a89f337f985

Using build environment variables
  BASH_ENV=/tmp/.bash_env-localbuild-1568899729
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=rappas
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://github.com/blinard-BIOINFO/bioconda-recipes.git
  CIRCLE_SHA1=7222b435a1397daad27e2bfc0c574c79afe34268
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1568899729
  CIRCLE_WORKING_DIRECTORY=~/project

The redacted variables listed above will be masked in run step output

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /root/project && cp -a /tmp/_circleci_local_build_repo/.git /root/project
/bin/bash: git: command not found
Error: Exited with code 127
Step failed
Error: runner failed (exited with 101)
Task failed
Error: task failed
eseiler commented 5 years ago

I have the same problem, for now you can just add (as a workaround)

- run: ln -s /opt/conda/bin/git /bin/git

to your bioconda-recipes/.circleci/config.yaml between lines 80 and 81:

https://github.com/bioconda/bioconda-recipes/blob/3402c292d3a222d7b0582603b6fd1e0f9d9a639e/.circleci/config.yml#L80-L81