search

deis / slugrunner

Buildpack application runner for Deis Workflow.
https://deis.com
MIT License
13 stars 18 forks source link

tar fails extracting slug with resource limits set #61

Open drock opened 7 years ago

drock commented 7 years ago

I am seeing a strange problem when trying to deploy my app with resource limits set. With no resource limits set, everything works fine. However, when I set the resource limits of the app below a certain value, it fails to launch

deis limits:set web=512M

After doing so kubectl will show the new pod in a CrashLoopBackoff and the logs of the pod show

tar: ./.heroku/node/bin: Directory renamed before its status could be extracted
tar: ./client/node_modules/regjsparser/node_modules/.bin: Directory renamed before its status could be extracted
tar: ./client/node_modules/regjsparser/node_modules: Directory renamed before its status could be extracted
tar: ./client/node_modules/regjsparser: Directory renamed before its status could be extracted
tar: ./client/node_modules/webpack/node_modules/.bin: Directory renamed before its status could be extracted
tar: ./client/node_modules/webpack/node_modules: Directory renamed before its status could be extracted
tar: ./client/node_modules/webpack: Directory renamed before its status could be extracted
tar: ./client/node_modules/acorn-jsx/node_modules/.bin: Directory renamed before its status could be extracted
tar: ./client/node_modules/acorn-jsx/node_modules: Directory renamed before its status could be extracted
tar: ./client/node_modules/acorn-jsx: Directory renamed before its status could be extracted
tar: ./client/node_modules/eslint/node_modules/.bin: Directory renamed before its status could be extracted
tar: ./client/node_modules/eslint/node_modules: Directory renamed before its status could be extracted
tar: ./client/node_modules/eslint: Directory renamed before its status could be extracted
tar: ./client/node_modules/gauge/node_modules/.bin: Directory renamed before its status could be extracted
tar: ./client/node_modules/gauge/node_modules: Directory renamed before its status could be extracted
tar: ./client/node_modules/gauge: Directory renamed before its status could be extracted
tar: ./client/node_modules/.bin: Directory renamed before its status could be extracted
tar: ./client/node_modules: Directory renamed before its status could be extracted
tar: ./client: Directory renamed before its status could be extracted
tar: ./bin: Directory renamed before its status could be extracted
tar: Exiting with failure status due to previous errors

This is all strange as the error message seem to be unrelated to memory limits. The compressed slug size is ~150M and the uncompressed app folder tree is:

deis run 'du -sh'
Running 'du -sh'...
747M    .

Bumping the memory limit up to only 600M solves the problem though.

Related, but probably separate is that if I drop the memory limit even lower to 400M then the failure happens when trying to merely download the slug with only this in the logs

/bin/get_object: line 13:     9 Killed                  objstorage --storage-type="$BUILDER_STORAGE" download "$SLUG_URL" "$GET_PATH"

My app only needs ~300M to run so I was hoping to not set the kubernetes limits so high.

bacongobbler commented 7 years ago

I wonder if https://github.com/deis/controller/pull/1179 might resolve this by proxy as it will be using a separate init container to pull down the slug.

drock commented 7 years ago

That sounds plausible since only the memory limit seems to trigger this issue and that would download the slug and extract it before those limits are enforced?

mboersma commented 7 years ago

This seems similar to docker/hub-feedback#727.

Cryptophobia commented 6 years ago

This issue was moved to teamhephy/slugrunner#4