Image built with Kaniko is missing directory created as a side-effect of a `RUN` call.

papiro commented 3 years ago

Actual behavior Image built with Kaniko is missing node_modules. All other files and directories copied over exist as expected.

Expected behavior Image should be built with the directory node_modules existing in /workspace

To Reproduce Steps to reproduce the behavior: Build the following Dockerfile using Kaniko:

FROM node:lts-alpine
WORKDIR /workspace

RUN apk add python3 git bash && \
    python3 -m pip install pre-commit

COPY package.json .
RUN yarn install

COPY .

Additional Information When built using docker build, expected behavior is observed.

.dockerignore file contains the line node_modules/

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [X ]
Please check if this error is seen when you use `--cache` flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

joeypedicini92 commented 3 years ago

Just to add I also tried without .dockerignore, doing npm ci which clears node_modules first and re-installs, and node_modules is still missing

aaron-prindle commented 1 year ago

Can you also provide the package_json and/or additional source files required to repro this? Thanks

aaron-prindle commented 1 year ago

In attempting to reproduce this using the given Dockerfile as is, I was seeing symlink related issues using Kaniko v1.17.0.

aprindle@aprindle-ssd ~/kaniko  [main]./run_in_docker.sh /workspace/Dockerfile /usr/local/google/home/aprindle/kaniko-ctx-dir gcr.io/aprindle-test-cluster/kaniko-test:latest false |& tee ~/out.txt
INFO[0000] Retrieving image manifest node:lts-alpine    
INFO[0000] Retrieving image node:lts-alpine from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest node:lts-alpine    
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'node:lts-alpine' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd RUN apk add python3 py3-pip git bash &&     python3 -m pip install pre-commit requires it. 
INFO[0004] WORKDIR /workspace                           
INFO[0004] Cmd: workdir                                 
INFO[0004] Changed working directory to /workspace      
INFO[0004] No files changed in this command, skipping snapshotting. 
INFO[0004] RUN apk add python3 py3-pip git bash &&     python3 -m pip install pre-commit 
INFO[0004] Initializing snapshotter ...                 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0005] Cmd: /bin/sh                                 
INFO[0005] Args: [-c apk add python3 py3-pip git bash &&     python3 -m pip install pre-commit] 
INFO[0005] Running: [/bin/sh -c apk add python3 py3-pip git bash &&     python3 -m pip install pre-commit] 
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/32) Installing ncurses-terminfo-base (6.4_p20230506-r0)
(2/32) Installing libncursesw (6.4_p20230506-r0)
(3/32) Installing readline (8.2.1-r1)
(4/32) Installing bash (5.2.15-r5)
Executing bash-5.2.15-r5.post-install
(5/32) Installing ca-certificates (20230506-r0)
(6/32) Installing brotli-libs (1.0.9-r14)
(7/32) Installing libunistring (1.1-r1)
(8/32) Installing libidn2 (2.3.4-r1)
(9/32) Installing nghttp2-libs (1.57.0-r0)
(10/32) Installing libcurl (8.4.0-r0)
(11/32) Installing libexpat (2.5.0-r1)
(12/32) Installing pcre2 (10.42-r1)
(13/32) Installing git (2.40.1-r0)
(14/32) Installing libbz2 (1.0.8-r5)
(15/32) Installing libffi (3.4.4-r2)
(16/32) Installing gdbm (1.23-r1)
(17/32) Installing xz-libs (5.4.3-r0)
(18/32) Installing mpdecimal (2.5.1-r2)
(19/32) Installing libpanelw (6.4_p20230506-r0)
(20/32) Installing sqlite-libs (3.41.2-r2)
(21/32) Installing python3 (3.11.6-r0)
(22/32) Installing python3-pycache-pyc0 (3.11.6-r0)
(23/32) Installing pyc (0.1-r0)
(24/32) Installing py3-setuptools-pyc (67.7.2-r0)
(25/32) Installing py3-pip-pyc (23.1.2-r0)
(26/32) Installing py3-parsing (3.0.9-r2)
(27/32) Installing py3-parsing-pyc (3.0.9-r2)
(28/32) Installing py3-packaging-pyc (23.1-r1)
(29/32) Installing python3-pyc (3.11.6-r0)
(30/32) Installing py3-packaging (23.1-r1)
(31/32) Installing py3-setuptools (67.7.2-r0)
(32/32) Installing py3-pip (23.1.2-r0)
Executing busybox-1.36.1-r2.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 87 MiB in 49 packages
Collecting pre-commit
  Downloading pre_commit-3.5.0-py2.py3-none-any.whl (203 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 203.7/203.7 kB 5.7 MB/s eta 0:00:00
Collecting cfgv>=2.0.0 (from pre-commit)
  Downloading cfgv-3.4.0-py2.py3-none-any.whl (7.2 kB)
Collecting identify>=1.0.0 (from pre-commit)
  Downloading identify-2.5.31-py2.py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.9/98.9 kB 15.5 MB/s eta 0:00:00
Collecting nodeenv>=0.11.1 (from pre-commit)
  Downloading nodeenv-1.8.0-py2.py3-none-any.whl (22 kB)
Collecting pyyaml>=5.1 (from pre-commit)
  Downloading PyYAML-6.0.1-cp311-cp311-musllinux_1_1_x86_64.whl (748 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 748.5/748.5 kB 32.0 MB/s eta 0:00:00
Collecting virtualenv>=20.10.0 (from pre-commit)
  Downloading virtualenv-20.24.6-py3-none-any.whl (3.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 82.2 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/lib/python3.11/site-packages (from nodeenv>=0.11.1->pre-commit) (67.7.2)
Collecting distlib<1,>=0.3.7 (from virtualenv>=20.10.0->pre-commit)
  Downloading distlib-0.3.7-py2.py3-none-any.whl (468 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.9/468.9 kB 38.9 MB/s eta 0:00:00
Collecting filelock<4,>=3.12.2 (from virtualenv>=20.10.0->pre-commit)
  Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
Collecting platformdirs<4,>=3.9.1 (from virtualenv>=20.10.0->pre-commit)
  Downloading platformdirs-3.11.0-py3-none-any.whl (17 kB)
Installing collected packages: distlib, pyyaml, platformdirs, nodeenv, identify, filelock, cfgv, virtualenv, pre-commit
Successfully installed cfgv-3.4.0 distlib-0.3.7 filelock-3.13.1 identify-2.5.31 nodeenv-1.8.0 platformdirs-3.11.0 pre-commit-3.5.0 pyyaml-6.0.1 virtualenv-20.24.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
INFO[0010] Taking snapshot of full filesystem...        
INFO[0013] COPY package.json .                          
INFO[0013] Taking snapshot of files...                  
INFO[0013] RUN yarn install                             
INFO[0013] Cmd: /bin/sh                                 
INFO[0013] Args: [-c yarn install]                      
INFO[0013] Running: [/bin/sh -c yarn install]           
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 4.97s.
INFO[0019] Taking snapshot of full filesystem...        
INFO[0023] COPY . .                                     
error building image: error building stage: failed to execute command: copying dir: symlink  /workspace/node_modules/.bin/acorn: no such file or directory

Modifiying the Dockerfile to use yarn install --no-bin-links I was able to get success:

FROM node:lts-alpine
WORKDIR /workspace

RUN apk add python3 py3-pip git bash && \
    python3 -m pip install pre-commit

COPY package.json .
RUN yarn install --no-bin-links

COPY . .

Successful run:

aprindle@aprindle-ssd ~/kaniko  [main]./run_in_docker.sh /workspace/Dockerfile /usr/local/google/home/aprindle/kaniko-ctx-dir gcr.io/aprindle-test-cluster/kaniko-test:latest false |& tee ~/out.txt
INFO[0000] Retrieving image manifest node:lts-alpine    
INFO[0000] Retrieving image node:lts-alpine from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest node:lts-alpine    
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'node:lts-alpine' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd RUN apk add python3 py3-pip git bash &&     python3 -m pip install pre-commit requires it. 
INFO[0004] WORKDIR /workspace                           
INFO[0004] Cmd: workdir                                 
INFO[0004] Changed working directory to /workspace      
INFO[0004] No files changed in this command, skipping snapshotting. 
INFO[0004] RUN apk add python3 py3-pip git bash &&     python3 -m pip install pre-commit 
INFO[0004] Initializing snapshotter ...                 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0005] Cmd: /bin/sh                                 
INFO[0005] Args: [-c apk add python3 py3-pip git bash &&     python3 -m pip install pre-commit] 
INFO[0005] Running: [/bin/sh -c apk add python3 py3-pip git bash &&     python3 -m pip install pre-commit] 
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/32) Installing ncurses-terminfo-base (6.4_p20230506-r0)
(2/32) Installing libncursesw (6.4_p20230506-r0)
(3/32) Installing readline (8.2.1-r1)
(4/32) Installing bash (5.2.15-r5)
Executing bash-5.2.15-r5.post-install
(5/32) Installing ca-certificates (20230506-r0)
(6/32) Installing brotli-libs (1.0.9-r14)
(7/32) Installing libunistring (1.1-r1)
(8/32) Installing libidn2 (2.3.4-r1)
(9/32) Installing nghttp2-libs (1.57.0-r0)
(10/32) Installing libcurl (8.4.0-r0)
(11/32) Installing libexpat (2.5.0-r1)
(12/32) Installing pcre2 (10.42-r1)
(13/32) Installing git (2.40.1-r0)
(14/32) Installing libbz2 (1.0.8-r5)
(15/32) Installing libffi (3.4.4-r2)
(16/32) Installing gdbm (1.23-r1)
(17/32) Installing xz-libs (5.4.3-r0)
(18/32) Installing mpdecimal (2.5.1-r2)
(19/32) Installing libpanelw (6.4_p20230506-r0)
(20/32) Installing sqlite-libs (3.41.2-r2)
(21/32) Installing python3 (3.11.6-r0)
(22/32) Installing python3-pycache-pyc0 (3.11.6-r0)
(23/32) Installing pyc (0.1-r0)
(24/32) Installing py3-setuptools-pyc (67.7.2-r0)
(25/32) Installing py3-pip-pyc (23.1.2-r0)
(26/32) Installing py3-parsing (3.0.9-r2)
(27/32) Installing py3-parsing-pyc (3.0.9-r2)
(28/32) Installing py3-packaging-pyc (23.1-r1)
(29/32) Installing python3-pyc (3.11.6-r0)
(30/32) Installing py3-packaging (23.1-r1)
(31/32) Installing py3-setuptools (67.7.2-r0)
(32/32) Installing py3-pip (23.1.2-r0)
Executing busybox-1.36.1-r2.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 87 MiB in 49 packages
Collecting pre-commit
  Downloading pre_commit-3.5.0-py2.py3-none-any.whl (203 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 203.7/203.7 kB 5.4 MB/s eta 0:00:00
Collecting cfgv>=2.0.0 (from pre-commit)
  Downloading cfgv-3.4.0-py2.py3-none-any.whl (7.2 kB)
Collecting identify>=1.0.0 (from pre-commit)
  Downloading identify-2.5.31-py2.py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.9/98.9 kB 8.8 MB/s eta 0:00:00
Collecting nodeenv>=0.11.1 (from pre-commit)
  Downloading nodeenv-1.8.0-py2.py3-none-any.whl (22 kB)
Collecting pyyaml>=5.1 (from pre-commit)
  Downloading PyYAML-6.0.1-cp311-cp311-musllinux_1_1_x86_64.whl (748 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 748.5/748.5 kB 12.4 MB/s eta 0:00:00
Collecting virtualenv>=20.10.0 (from pre-commit)
  Downloading virtualenv-20.24.6-py3-none-any.whl (3.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 24.1 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/lib/python3.11/site-packages (from nodeenv>=0.11.1->pre-commit) (67.7.2)
Collecting distlib<1,>=0.3.7 (from virtualenv>=20.10.0->pre-commit)
  Downloading distlib-0.3.7-py2.py3-none-any.whl (468 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.9/468.9 kB 27.4 MB/s eta 0:00:00
Collecting filelock<4,>=3.12.2 (from virtualenv>=20.10.0->pre-commit)
  Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
Collecting platformdirs<4,>=3.9.1 (from virtualenv>=20.10.0->pre-commit)
  Downloading platformdirs-3.11.0-py3-none-any.whl (17 kB)
Installing collected packages: distlib, pyyaml, platformdirs, nodeenv, identify, filelock, cfgv, virtualenv, pre-commit
Successfully installed cfgv-3.4.0 distlib-0.3.7 filelock-3.13.1 identify-2.5.31 nodeenv-1.8.0 platformdirs-3.11.0 pre-commit-3.5.0 pyyaml-6.0.1 virtualenv-20.24.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
INFO[0010] Taking snapshot of full filesystem...        
INFO[0013] COPY package.json .                          
INFO[0013] Taking snapshot of files...                  
INFO[0013] RUN yarn install --no-bin-links              
INFO[0013] Cmd: /bin/sh                                 
INFO[0013] Args: [-c yarn install --no-bin-links]       
INFO[0013] Running: [/bin/sh -c yarn install --no-bin-links] 
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 4.80s.
INFO[0018] Taking snapshot of full filesystem...        
INFO[0023] COPY . .                                     
INFO[0024] Taking snapshot of files...                  
INFO[0025] Pushing image to gcr.io/aprindle-test-cluster/kaniko-test:latest 
INFO[0030] Pushed gcr.io/aprindle-test-cluster/kaniko-test@sha256:10c57ca1b3760b9908c85119f21ead0b35cc3667c67f3fe784cb942b3362d0db

node_modules is present in /workspace

aprindle@aprindle-ssd ~/kaniko  [main]docker run -it  gcr.io/aprindle-test-cluster/kaniko-test@sha256:10c57ca1b3760b9908c85119f21ead0b35cc3667c67f3fe784cb942b3362d0db bash
Unable to find image 'gcr.io/aprindle-test-cluster/kaniko-test@sha256:10c57ca1b3760b9908c85119f21ead0b35cc3667c67f3fe784cb942b3362d0db' locally
gcr.io/aprindle-test-cluster/kaniko-test@sha256:10c57ca1b3760b9908c85119f21ead0b35cc3667c67f3fe784cb942b3362d0db: Pulling from aprindle-test-cluster/kaniko-test
96526aa774ef: Already exists 
277539faec26: Already exists 
2e04e6e2ab16: Already exists 
71ddc29abe1f: Already exists 
5fef82943757: Pull complete 
28ceb26290bf: Pull complete 
23fd1a09706d: Pull complete 
6056a97cf3cb: Pull complete 
Digest: sha256:10c57ca1b3760b9908c85119f21ead0b35cc3667c67f3fe784cb942b3362d0db
Status: Downloaded newer image for gcr.io/aprindle-test-cluster/kaniko-test@sha256:10c57ca1b3760b9908c85119f21ead0b35cc3667c67f3fe784cb942b3362d0db
6c3b7b9adc0e:/workspace# ls
Dockerfile    LICENSE       README.md     __tests__     index.js      node_modules  package.json  yarn.lock
6c3b7b9adc0e:/workspace#

Can you confirm if this is still an issue and if you able to attempt using the --no-bin-links yarn flag as a potential workaround? Perhaps the best action item here is to document current symlink issues, workarounds (for common use-cases, runtimes, etc.) and work towards symlink issue root-cause and fix.

GoogleContainerTools / kaniko

Image built with Kaniko is missing directory created as a side-effect of a `RUN` call. #1628