search

CircleCI-Public / circleci-cli

Use CircleCI from the command line
https://circleci-public.github.io/circleci-cli/
MIT License
409 stars 232 forks source link

local execution checkout failing #330

Open robeferre-leroy opened 5 years ago

robeferre-leroy commented 5 years ago

Do you want to request a feature or report a bug? BUG

What is the current behavior?

robeferre@mylinux:~/Workspace/golang/src/github.com/robeferre-leroy/golang-sample$ sudo circleci local execute
Docker image digest: sha256:60a98b5fc00a4302c46be2217d6e4e4eb8491d466fa62fed14362a85100bcd68
====>> Spin up Environment
Build-agent version 1.0.15283-35354d0e (2019-09-09T12:09:35+0000)
Docker Engine Version: 18.09.7
Kernel Version: Linux 6772389d44a6 5.0.0-27-generic #28~18.04.1-Ubuntu SMP Thu Aug 22 03:00:32 UTC 2019 x86_64 Linux
Starting container circleci/buildpack-deps:latest-dind
  using image circleci/buildpack-deps@sha256:f9dea38e6583c663bbe2d950b8c9abdb771f9dd587fdb18460da2f4253c3a70a

Using build environment variables
  BASH_ENV=/tmp/.bash_env-localbuild-1568756269
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=feature/circleci
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://github.com/robeferre-leroy/golang-sample.git
  CIRCLE_SHA1=0660ce60a8c0e3a4de9e571566ce6646a81a426a
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1568756269
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /home/circleci/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /home/circleci/project && cp -a /tmp/_circleci_local_build_repo/.git /home/circleci/project
{"Runner":true,"level":"error","msg":"Error copying logs: read /dev/ptmx: input/output error","task-id":"localbuild-1568756269","time":"2019-09-17T21:37:54Z"}
/bin/bash: line 0: cd: /tmp/_circleci_local_build_repo: No such file or directory
Error: Exited with code 1
Step failed
Error: runner failed (exited with 101)
Task failed

Can you provide an example?

robeferre@mylinux:~/Workspace/golang/src/github.com/robeferre-leroy/golang-sample$ sudo circleci local execute
Docker image digest: sha256:60a98b5fc00a4302c46be2217d6e4e4eb8491d466fa62fed14362a85100bcd68
====>> Spin up Environment
Build-agent version 1.0.15283-35354d0e (2019-09-09T12:09:35+0000)
Docker Engine Version: 18.09.7
Kernel Version: Linux 6772389d44a6 5.0.0-27-generic #28~18.04.1-Ubuntu SMP Thu Aug 22 03:00:32 UTC 2019 x86_64 Linux
Starting container circleci/buildpack-deps:latest-dind
  using image circleci/buildpack-deps@sha256:f9dea38e6583c663bbe2d950b8c9abdb771f9dd587fdb18460da2f4253c3a70a

Using build environment variables
  BASH_ENV=/tmp/.bash_env-localbuild-1568756269
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=feature/circleci
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://github.com/robeferre-leroy/golang-sample.git
  CIRCLE_SHA1=0660ce60a8c0e3a4de9e571566ce6646a81a426a
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1568756269
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /home/circleci/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /home/circleci/project && cp -a /tmp/_circleci_local_build_repo/.git /home/circleci/project
{"Runner":true,"level":"error","msg":"Error copying logs: read /dev/ptmx: input/output error","task-id":"localbuild-1568756269","time":"2019-09-17T21:37:54Z"}
/bin/bash: line 0: cd: /tmp/_circleci_local_build_repo: No such file or directory
Error: Exited with code 1
Step failed
Error: runner failed (exited with 101)
Task failed

What is the expected behavior? testing builds locally

Which version of the CLI and OS are you using? Did this work in previous versions? circleci version 0.1.5879+416032d

NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)"

Please provide the output of circleci version and circleci diagnostic.

CircleCI CLI Diagnostics

Debugger mode: false Config found: /home/robeferre/.circleci/cli.yml API host: https://circleci.com API endpoint: graphql-unstable OK, got a token. Trying an introspection query on API... Ok. Hello, Roberto.

If you have any questions, feel free to ping us at @CircleCI-Public/x-team.

DerrickRice commented 5 years ago

Same. It seems that even though --skip-checkout is the default (and even if it's explicitly mentioned), the "Checkout code" step tries to change to a directory that does not exist:

/bin/bash: line 0: cd: /tmp/_circleci_local_build_repo: Permission denied

Seems like a pretty obvious failure. Curious that this issue has not had any comments in over a month.

alexkli commented 4 years ago

I have the same issue, using 0.1.5879+416032d.

Looking at docker inspect $(docker ps -q) while circleci build is running, I can see that mounts the local working directory under /tmp/_circleci_local_build_repo inside the build container. 

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/Users/me/my/code",
                "Destination": "/tmp/_circleci_local_build_repo",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },

But when I do an ls /tmp/_circleci_local_build_repo (or with sudo) in my job, it says this path does not exist. So it seems the volume mounting silently fails for some reason.

Investigating further, I compared it to a manual docker run -it -v /Users/me/my/code:/tmp/_circleci_local_build_repo circleci/buildpack-deps:latest-dind /bin/bash where the mount works. The only difference I saw is when running mount, the broken one also has "none on tmpfs /tmp", which might interfere with the /tmp/... volume mount. No idea where it comes from though.

alexkli commented 4 years ago

Found a solution: manually mount the local directory using -v on circleci build. A bit of symlinking before the checkout command and it works!

config.yml snippet:

jobs:
  build:
    machine: true
    steps:
      # fix for local builds https://github.com/CircleCI-Public/circleci-cli/issues/330
      - run:
          name: Local build handling
          command: |
            if [[ ${CIRCLE_SHELL_ENV} =~ "localbuild" ]]; then
              if [ -d /workdir ]; then
                ln -s /workdir /tmp/_circleci_local_build_repo
              else
                echo "Run this local build using: circleci build -v \$(pwd):/workdir"
                exit 1
              fi
            fi
      - checkout

Then run local build using:

circleci build -v $(pwd):/workdir
notpushkin commented 4 years ago

@alexkli Thank you so much for this solution!

marcomorain commented 4 years ago

Logged as CIRCLE-25051.

Slach commented 4 years ago

@alexkli thanks for workaround but it's not works for me Ubuntu 18.04

 circleci local execute -c .circleci/config.yml -v .:/workdir --job build_branch
Error:   error starting container circleci/buildpack-deps:latest-dind: Error response from daemon: create [.: "[." includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

I don't know why circleci use "[." instead of "."

manual docker run works fine absolute path $(pwd) also not works

Unexpected environment preparation error: Error response from daemon: create [/home/test/circleci_local: "[/home/test/circleci_local" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

maybe you can suggest workaround?

alexkli commented 4 years ago

@Slach

  1. it must be an absolute path
  2. looks like you run into this major regression in the latest cli version which adds [] around arguments like -v or -e: #394 and #391 - you must downgrade the cli
marcomorain commented 4 years ago

I've been looking into this issue. The issue seems to be related to the image that used for the build.

Failing images:

Images that work:

When we execute a build, we mount the pwd into the circleci/picard image, using --volume $pwd:$pwd. We think launch the docker container for the build with --volume $pwd:/tmp/_circleci_local_build_repo. On builds when this fails, this second mount at /tmp/_circleci_local_build_repo does not exist inside the innermost container.

BenWibking commented 4 years ago

Neither circleci/python:3.7.2 or cimg/base:2020.05 work for me. 0.1.7971+423e103 (release) and Docker version 19.03.11, build 42e35e61f3 on Ubuntu 18.04.4 LTS.

fasterthanlime commented 3 years ago

Can confirm that the simplest job fails to run locally with cimg/rust:1.49.0 (but cimg/base:2020.05 does work)

image

fasterthanlime commented 3 years ago

Quick update: I'm able to always reproduce this when using cimg/rust, but whenever using another base image (from the Docker Hub, or the private registry), I don't encounter this issue and I'm able to run jobs locally.

KaiyuWangCompass commented 3 years ago

Found a solution: manually mount the local directory using -v on circleci build. A bit of symlinking before the checkout command and it works!

config.yml snippet:

jobs:
  build:
    machine: true
    steps:
      # fix for local builds https://github.com/CircleCI-Public/circleci-cli/issues/330
      - run:
          name: Local build handling
          command: |
            if [[ ${CIRCLE_SHELL_ENV} =~ "localbuild" ]]; then
              if [ -d /workdir ]; then
                ln -s /workdir /tmp/_circleci_local_build_repo
              else
                echo "Run this local build using: circleci build -v \$(pwd):/workdir"
                exit 1
              fi
            fi
      - checkout

Then run local build using:

circleci build -v $(pwd):/workdir

This isn't working for me. CircleCI is not even executing this code !

kuzdogan commented 3 years ago

My local execution also fails because CircleCI checkout can't copy files with special characters. The special characters are needed in the directory name for the tests for special characters. Couldn't find a workaround. 

====>> Checkout code
Making checkout directory "/home/circleci/source-verify"
Copying files from "/tmp/_circleci_local_build_repo" to "/home/circleci/source-verify"
Error: tar: "services/validation/test/files/single-add-trailing-\\n/1_Storage.sol": Cannot stat: No such file or directory
....
....
Renkas commented 1 year ago 
====>> Checkout code
Making checkout directory "/home/circleci/project"
Copying files from "/tmp/_circleci_local_build_repo" to "/home/circleci/project"
Error: cp: cannot create regular file '/home/circleci/project/.git/objects/dd/f6fdf3c814d536cad1bd77b2bc86235d4da9c2': Permission denied
cp: cannot create regular file '/home/circleci/project/.git/objects/dd/56867087b589d8fa97ee544cda65a648745787': Permission denied
cp: cannot create regular file '/home/circleci/project/.git/objects/ce/acf067890b31197ce075de50b3e11bd9a29964': Permission denied
...

So what now? How can I get this to work?

Running on MacOS