tar throws "Directory renamed before its status could be extracted" error

[siennathesane]

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

{
   "name": "",
   "build": "",
   "support": "",
   "version": 0,
   "description": "",
   "authorization_endpoint": "https://login.dev.cfdev.sh",
   "token_endpoint": "https://uaa.dev.cfdev.sh",
   "min_cli_version": null,
   "min_recommended_cli_version": null,
   "app_ssh_endpoint": "ssh.dev.cfdev.sh:2222",
   "app_ssh_host_key_fingerprint": "96:4d:89:2d:39:18:bc:16:e1:d3:d8:44:f8:16:af:85",
   "app_ssh_oauth_client": "ssh-proxy",
   "doppler_logging_endpoint": "wss://doppler.dev.cfdev.sh:443",
   "api_version": "2.133.0",
   "osbapi_version": "2.14",
   "routing_endpoint": "https://api.dev.cfdev.sh/routing"
}

> cf version
cf.exe version 6.43.0+815ea2f3d.2019-02-20

What version of the buildpack you are using?

v1.6.49

If you were attempting to accomplish a task, what was it you were attempting to do?

# clone and cd
git clone https://github.com/codercom/code-server && cd code-server

# local install
npm install

# build the manifest
cat manifest.yml << EOF
---
applications:
- name: vscode
  memory: 1G
  disk_quota: 2G
  instances: 1
  command: yarn && yarn task build:server:binary && ./packages/server/cli-linux-x64 --host 0.0.0.0 --port $PORT --allow-http --no-auth
  buildpacks:
    - https://github.com/cloudfoundry/nodejs-buildpack#v1.6.49
  stack: cflinuxfs3
  env:
    NODE_ENV: production
    OPTIMIZE_MEMORY: true
EOF

# push
cf push -f manifest.yml

What did you expect to happen?

The application was able to push successful.

What was the actual behavior?

Pushing from manifest to org apps / space code-server as admin...
Using manifest file manifest.yml
Getting app info...
Updating app with these attributes...
  name:                vscode
  path:                /p/javascript/code-server
  buildpacks:
+   https://github.com/cloudfoundry/nodejs-buildpack#v1.6.49
-   nodejs_buildpack
  command:             yarn && yarn task build:server:binary && ./packages/server/cli-linux-x64 --host 0.0.0.0 --port $PORT --allow-http --no-auth
  disk quota:          2G
  health check type:   port
  instances:           1
  memory:              1G
  stack:               cflinuxfs3
  env:
    NODE_ENV
    OPTIMIZE_MEMORY
  routes:
    vscode.dev.cfdev.sh

Updating app vscode...
Mapping routes...
Comparing local files to remote cache...
Packaging files to upload...
Uploading files...
 56.67 MiB / 56.67 MiB [==========================================================================================================] 100.00% 3s
Waiting for API to complete processing files...

Staging app and tracing logs...
   Cell 0285258e-caad-4381-a7be-74268c9e1b87 creating container for instance 5134b103-3e66-4b2c-b165-55fc48443343
   Cell 0285258e-caad-4381-a7be-74268c9e1b87 successfully created container for instance 5134b103-3e66-4b2c-b165-55fc48443343
   Downloading build artifacts cache...
   Downloading app package...
   Downloaded build artifacts cache (43.2M)
   Copying app package into the container failed: stream-in: nstar: error streaming in: exit status 2. Output: tar: packages/server/node_modules/css-loader/node_modules/.bin: Directory renamed before its status could be extracted
   tar: packages/server/node_modules/css-loader/node_modules: Directory renamed before its status could be extracted
   tar: packages/server/node_modules/css-loader: Directory renamed before its status could be extracted
   tar: packages/server/node_modules/.bin: Directory renamed before its status could be extracted
   tar: packages/protocol/node_modules/trash/node_modules/.bin: Directory renamed before its status could be extracted
   tar: packages/protocol/node_modules/trash/node_modules: Directory renamed before its status could be extracted
   tar: packages/protocol/node_modules/trash: Directory renamed before its status could be extracted
   tar: packages/protocol/node_modules/send/node_modules/.bin: Directory renamed before its status could be extracted
   Cell 0285258e-caad-4381-a7be-74268c9e1b87 stopping instance 5134b103-3e66-4b2c-b165-55fc48443343
   Cell 0285258e-caad-4381-a7be-74268c9e1b87 destroying container for instance 5134b103-3e66-4b2c-b165-55fc48443343
Error staging application: Staging error: staging failed
FAILED

Can you provide a sample app?

Code Server

This was apparently a known issue in nodejs/docker-node/379

Please confirm where necessary:

• [X] I have included a log output
• [X] My log includes an error message
• [X] I have included steps for reproduction

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/165574363

The labels on this github issue will be updated when the story is started.

[dwillist]

Thanks for opening an issue. We will prioritize this ASAP.

[emalm]

Hmm, since that error occurs when the Diego cell rep copies the app source files into the staging-task container, I wouldn't expect it to be related to the buildpack at all. That error looks similar to one I noticed and reported to Garden in 2017 (see https://www.pivotaltracker.com/story/show/150961639) but I thought we got the fix for it (something about how overlay handles inodes) backported to the 4.4 kernel. I'd be surprised for it to appear again on what from the /v2/info data seems to be an up-to-date cfdev installation.

/cc @cloudfoundry/cf-garden for advice

[Callisto13]

Hey @mxplusb Garden team here. Could you provide us with the following information:

• The IAAS your Cloud Foundry is deployed on (AWS/GCP/etc).
• The version of Garden-Runc-Release deployed in your Cloud Foundry.
• The Stemcell your CF is using.

[siennathesane]

@Callisto13 this was deployed with cfdev v0.0.15. @sclevine would know the specifics of CPI, garden, and stemcells that version deploys with.

[Callisto13]

@mxplusb

Aaah I missed that it was cfdev! This changes the game a little: cfdev uses a linuxkit kernel not the ubuntu polished one we put in our stemcells. I did a quick cf dev start and can see that the kernel is 4.14.70-linuxkit.

If @emalm's suspicions are correct (it certainly looks like he might be) and this is the same thing, then there is the possibility that the fix for the overlay inode problem may not have made it in there. Or it did make it in there and this is something completely different but similar 😄.

Either way, this is almost certainly not a buildpack problem! Please could you open an issue in garden-runc? Thanks!

[chhhavi]

Looks like this isn't a buildpacks problem, we are closing this on our end.

[siennathesane]

Sounds good, thanks!