nstar error streaming in (possible regression in cfdev linuxkit kernel)

siennathesane commented 5 years ago

Copied from cloudfoundry/nodejs-buildpack#188 as they recommended re-submitting here. thoughts from @emalm:

Hmm, since that error occurs when the Diego cell rep copies the app source files into the staging-task container, I wouldn't expect it to be related to the buildpack at all. That error looks similar to one I noticed and reported to Garden in 2017 (see https://www.pivotaltracker.com/story/show/150961639) but I thought we got the fix for it (something about how overlay handles inodes) backported to the 4.4 kernel. I'd be surprised for it to appear again on what from the /v2/info data seems to be an up-to-date cfdev installation.

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running `cf curl /v2/info && cf version`?

{
   "name": "",
   "build": "",
   "support": "",
   "version": 0,
   "description": "",
   "authorization_endpoint": "https://login.dev.cfdev.sh",
   "token_endpoint": "https://uaa.dev.cfdev.sh",
   "min_cli_version": null,
   "min_recommended_cli_version": null,
   "app_ssh_endpoint": "ssh.dev.cfdev.sh:2222",
   "app_ssh_host_key_fingerprint": "96:4d:89:2d:39:18:bc:16:e1:d3:d8:44:f8:16:af:85",
   "app_ssh_oauth_client": "ssh-proxy",
   "doppler_logging_endpoint": "wss://doppler.dev.cfdev.sh:443",
   "api_version": "2.133.0",
   "osbapi_version": "2.14",
   "routing_endpoint": "https://api.dev.cfdev.sh/routing"
}

> cf version
cf.exe version 6.43.0+815ea2f3d.2019-02-20

What version of the buildpack you are using?

v1.6.49

If you were attempting to accomplish a task, what was it you were attempting to do?

# clone and cd
git clone https://github.com/codercom/code-server && cd code-server

# local install
npm install

# build the manifest
cat manifest.yml << EOF
---
applications:
- name: vscode
  memory: 1G
  disk_quota: 2G
  instances: 1
  command: yarn && yarn task build:server:binary && ./packages/server/cli-linux-x64 --host 0.0.0.0 --port $PORT --allow-http --no-auth
  buildpacks:
    - https://github.com/cloudfoundry/nodejs-buildpack#v1.6.49
  stack: cflinuxfs3
  env:
    NODE_ENV: production
    OPTIMIZE_MEMORY: true
EOF

# push
cf push -f manifest.yml

What did you expect to happen?

The application was able to push successful.

What was the actual behavior?

Pushing from manifest to org apps / space code-server as admin...
Using manifest file manifest.yml
Getting app info...
Updating app with these attributes...
  name:                vscode
  path:                /p/javascript/code-server
  buildpacks:
+   https://github.com/cloudfoundry/nodejs-buildpack#v1.6.49
-   nodejs_buildpack
  command:             yarn && yarn task build:server:binary && ./packages/server/cli-linux-x64 --host 0.0.0.0 --port $PORT --allow-http --no-auth
  disk quota:          2G
  health check type:   port
  instances:           1
  memory:              1G
  stack:               cflinuxfs3
  env:
    NODE_ENV
    OPTIMIZE_MEMORY
  routes:
    vscode.dev.cfdev.sh

Updating app vscode...
Mapping routes...
Comparing local files to remote cache...
Packaging files to upload...
Uploading files...
 56.67 MiB / 56.67 MiB [==========================================================================================================] 100.00% 3s
Waiting for API to complete processing files...

Staging app and tracing logs...
   Cell 0285258e-caad-4381-a7be-74268c9e1b87 creating container for instance 5134b103-3e66-4b2c-b165-55fc48443343
   Cell 0285258e-caad-4381-a7be-74268c9e1b87 successfully created container for instance 5134b103-3e66-4b2c-b165-55fc48443343
   Downloading build artifacts cache...
   Downloading app package...
   Downloaded build artifacts cache (43.2M)
   Copying app package into the container failed: stream-in: nstar: error streaming in: exit status 2. Output: tar: packages/server/node_modules/css-loader/node_modules/.bin: Directory renamed before its status could be extracted
   tar: packages/server/node_modules/css-loader/node_modules: Directory renamed before its status could be extracted
   tar: packages/server/node_modules/css-loader: Directory renamed before its status could be extracted
   tar: packages/server/node_modules/.bin: Directory renamed before its status could be extracted
   tar: packages/protocol/node_modules/trash/node_modules/.bin: Directory renamed before its status could be extracted
   tar: packages/protocol/node_modules/trash/node_modules: Directory renamed before its status could be extracted
   tar: packages/protocol/node_modules/trash: Directory renamed before its status could be extracted
   tar: packages/protocol/node_modules/send/node_modules/.bin: Directory renamed before its status could be extracted
   Cell 0285258e-caad-4381-a7be-74268c9e1b87 stopping instance 5134b103-3e66-4b2c-b165-55fc48443343
   Cell 0285258e-caad-4381-a7be-74268c9e1b87 destroying container for instance 5134b103-3e66-4b2c-b165-55fc48443343
Error staging application: Staging error: staging failed
FAILED

Can you provide a sample app?

Code Server

This was apparently a known issue in nodejs/docker-node/379

Please confirm where necessary:

[X] I have included a log output
[X] My log includes an error message
[X] I have included steps for reproduction

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/165949516

The labels on this github issue will be updated when the story is started.

Callisto13 commented 5 years ago

Relevant Garden information:

Kernel: 4.14.70-linuxkit GRR Version: 1.19.1 IAAS: virtualbox cfdev

danail-branekov commented 5 years ago

Hi @mxplusb

We tried reproducing the issue on cfdev 0.0.15 (the one that you use) but we could not. Therefore could you please provide as many details as possible so that we can reproduce it on our side. For example:

When you reproduce the issue could you please ssh on the diego cell and execute the /var/vcap/packages/docntpanic script and share the report? You can do that via executing the following commands:
- Setup bosh:
  - Install bosh cli v2 (https://bosh.io/docs/cli-v2/)
  - Setup bosh environment variables:
    - Unix: eval "$(cf dev bosh env)"
    - Windows: cf dev bosh env | Invoke-Expression
- ssh on the diego cell: bosh -d cf ssh diego-cell
- su to root: sudo su -
- execute the dontpanic script via running /var/vcap/packages/dontpanic/bin/dontpanic
- the script produces an archive containing verbose debug information such as kernel logs, garden logs, cgroups, etc. You can download the archive on your local machine via running bosh -d cf scp diego-cell:/var/vcap/data/tmp/os-report-<random uid>_<date>.tgz . on your machine. The archive would be downloaded in the current directory.
Do you reproduce the issue when running on Windows only, or you can do that on e.g. Mac or Linux. What is the Windows version that you use?
How often is the issue reproduced? Does it happen every time? Does it happen on other machines?
Can you reproduce the issue via pushing other applications?

danail-branekov commented 5 years ago

Hi @mxplusb

We had no luck in reproducing the issue on our side though we tried hard. The only difference seems to be the OS we run the experiment on - we deploy cfdev on MacOS, and it seems that you do that on Windows. Unfortunately we do not have access to real Windows machines and cannot give your scenario a shot.

I will close this issue as "not reproducible", please feel free to reopen if/when it occurs again, please also make sure that you share the information requested above.

siennathesane commented 5 years ago

Sounds good, thanks!

cloudfoundry / garden-runc-release