Kaniko is unable to clean filesystem

[payxn]

Actual behavior Kaniko is unable to clean filesystem at the end of the build

INFO[2023-09-07T11:54:02Z] Taking snapshot of full filesystem...        
INFO[2023-09-07T11:55:10Z] Pushing layer xxx to cache now 
INFO[2023-09-07T11:55:10Z] Pushing image to xxx
INFO[2023-09-07T11:55:51Z] Pushed xxx
INFO[2023-09-07T11:55:51Z] Saving file usr/src/app/dist for later use   
INFO[2023-09-07T11:55:51Z] Deleting filesystem...                       
error building image: deleting file system after stage 2: unlinkat //usr/src/app: directory not empty

To Reproduce I'm running Kaniko latest version through a gitlab runner with the following args

    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${DOCKERFILE_PATH}"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
      --build-arg NPM_INDEX_PASSWORD=${NPM_INDEX_PASSWORD}
      --build-arg YARN_WORKDIR=${YARN_WORKDIR} 
      --log-timestamp=true
      --snapshot-mode=redo
      --cache=true

Additionnal information

This was supposed to be a POC to test Kaniko, the build works well with Docker.

Dockerfile :

### NODE SETUP ###
FROM node:14-alpine as node_setup

### SETUP ###
FROM node_setup as setup

ARG NPM_INDEX_PASSWORD
ENV NPM_INDEX_PASSWORD ${NPM_INDEX_PASSWORD}
ARG YARN_WORKDIR
ENV YARN_WORKDIR ${YARN_WORKDIR}

RUN mkdir -p ${YARN_WORKDIR}
WORKDIR ${YARN_WORKDIR}

COPY package.json yarn.lock .npmrc decorate-angular-cli.js ./
RUN yarn --ignore-scripts --frozen-lockfile &&\
  yarn postinstall

### BUILD ###
FROM setup as build

COPY angular.json nx.json tsconfig.base.json typings.ts webpack.partial.js ./
COPY shared-assets shared-assets
COPY libs libs
COPY apps apps

ENV NODE_OPTIONS "--max_old_space_size=8192"

RUN yarn build --prod
RUN yarn build

### RUN ###
FROM node_setup as production

ARG NPM_INDEX_PASSWORD
ENV NPM_INDEX_PASSWORD ${NPM_INDEX_PASSWORD}
ARG YARN_WORKDIR
ENV YARN_WORKDIR ${YARN_WORKDIR}

WORKDIR /opt/dx-project

COPY package.json yarn.lock .npmrc ./
COPY --from=build ${YARN_WORKDIR}/dist ./

RUN yarn install --production --ignore-scripts && yarn cache clean

COPY version.txt ./

EXPOSE 3050

CMD ["node", "main.js"]

[aaron-prindle]

Hi @payxn, thanks for flagging this. For a multi-stage build Kaniko works by pulling down each base image and unpacking the image into the root FS and storing files necessary for future builds in preserved root directories. Kaniko then cleans the root dir and copies from these preserved files/dirs as needed in future stages. This issue is likely related to attempting to cleanup the intermediate root fs (from a prev stage of the multi-stage build) but encountering some issue most likely with symlinks (either kaniko's handling of symlinks generally or in this special cleanup case in which a symlink from a cleanup directory maps to a directory that kaniko has marked to not cleanup). This is likely a dupe of #1697

[supersmile2009]

We're running into the same problem. And in our case it fails frequently but inconsistently. We have a number of projects where this kind of multi-stage build works just fine. And one project where it randomly fails. If you simply keep retrying, no changes to the code, scripts or environment, it randomly succeeds like 1 out of 6-7 times.