nodejs(all): appsody run creates root owned node_modules on dev host

[sam-github]

Describe the bug A clear and concise description of what the bug is.

appsody run causes a docker build, which causes a RUN npm install as root, which creates a node_modules on the dev host with root ownership. That root owned directory makes it difficult for the user to do things like install their app's dependencies.

To Reproduce Steps to reproduce the behavior:

cloud-node/ex-ext2 % rm -rf node_modules                                               
..... appsody run.... ctrl-C (or not)
cloud-node/ex-ext2 % ls -l                                                             
...
drwxr-xr-x  2 root root  4096 Nov 22 12:53 node_modules/                               
-rwxrwxr-x  1 sam  sam    506 Nov 22 12:49 package.json*
...
cloud-node/ex-ext2 % npm install --save lodash           
npm WARN checkPermissions Missing write access to /home/sam/w/cloud-node/ex-ext2/node_modules
npm ERR! code EACCES                                                                   
npm ERR! syscall access
npm ERR! path /home/sam/w/cloud-node/ex-ext2/node_modules                              
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, access '/home/sam/w/cloud-node/ex-ext2/node_modules'

Expected behavior A clear and concise description of what you expected to happen.

To be able add a new dependency to my project.

Actual behaviour What is the actual behaviour.

It failed to do so.

Environment Details (please complete the following information):

• OS: [e.g. iOS] Ubuntu 19.10
• Version [e.g. 22] Version of what?

If applicable please specify:

• CLI version: 0.5.0
• Stack you are using (including the version): nodejs-express current

I considered a USER node is needed before the RUN npm install in the user-app/ directory, but even that would create a node_modules with a uid != to mine. Perhaps the volume can be mounted in a way that uid mapping occurs? I'm not sure, does something similar happen in other stacks?

[sam-github]

Maybe I should report this as a seperate bug, but is related. The npm install runs as root in the container, and since npm runs package-defined scripts, it has a protective mechanism to avoid running them as root, it drops its privileges to "nobody". This goes quite poorly when the package install has to write artifacts to disk, like every C++ addon does.

Reproduction:

appsody init nodejs-express
npm install modern-syslog
appsody run
cloud-node/ex-ext2 % appsody run
Running development environment...
Using local cache for image dev.local/nodejs-express:0.3.0                             
Running command: docker run --rm -p 3000:3000 -p 9229:9229 --name ex-ext2-dev -v /home/sam/w/cloud-node/ex-ext2/:/project/user-app -v ex-ext2-deps:/project/user-app/node_modu
les -v appsody-controller-0.3.1:/.appsody -t --entrypoint /.appsody/appsody-controller dev.local/nodejs-express:0.3.0 --mode=run
[Container] Running APPSODY_PREP command: npm install --prefix user-app && npm audit fix --prefix user-app
[Container] [..................] | rollbackFailedOptional: verb npm-session 42605430b3 

[Container] > modern-syslog@1.2.0 install /project/user-app/node_modules/modern-syslog 
[Container] > node-gyp rebuild
[Container]                                
[Container] gyp WARN EACCES user "undefined" does not have permission to access the dev dir "/root/.node-gyp/10.16.3"
[Container] gyp WARN EACCES attempting to reinstall using temporary dev dir "/project/user-app/node_modules/modern-syslog/.node-gyp"
[Container] gyp WARN install got an error, rolling back install                        
[Container] gyp WARN install got an error, rolling back install                        
[Container] gyp ERR! configure error 
[Container] gyp ERR! stack Error: EACCES: permission denied, mkdir '/project/user-app/node_modules/modern-syslog/.node-gyp'
[Container] gyp ERR! System Linux 5.3.0-23-generic                                     
[Container] gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
[Container] gyp ERR! cwd /project/user-app/node_modules/modern-syslog                  
[Container] gyp ERR! node -v v10.16.3
[Container] gyp ERR! node-gyp -v v3.8.0
[Container] gyp ERR! not ok 
[Container] npm ERR! code ELIFECYCLE

Some possible fixes for this:

• do npm install --unsafe-perm: this means when the install runs as root, it stays root. This might be OK since the install is occurring in a container anyhow, maybe exploitation of the container's root user is not a concern?
• run npm install as a non-root user, this might fix the other issue, where running as root is causing root-owned files to show up in the developer's account.

[sam-github]

Reproduced by @skoh7645 , see https://github.com/appsody/stacks/issues/530#issuecomment-559735706

Apparently this is Linux specific, or possibly, "anything but minikube" specific.

[nastacio]

This issue seems possible across different stacks. If so, it sounds like something that the appsody CLI itself should check-and-fail or workaround.

cc @neeraj-laad

[neeraj-laad]

@sam-github I'm curious to understand which stack are you using and why are node_modules created on the host?

[sam-github]

@neeraj-laad name of the stack is in the issue title: nodejs-express Are you not able to repro?

I'm not sure why it is created on the host, I assumed it was intentional, and it was only the permissions that were a bug.

The two lines of the Dockerfile-stack look like they might be responsible:

ENV APPSODY_DEPS=/project/user-app/node_modules                                
ENV APPSODY_PREP="npm install --prefix user-app && npm audit fix --prefix user-app"

[sam-github]

Spent 20 min poking around, not sure, but the actual dependencies installed by npm install --prefix user-app do not show up on the host fs, so I suspect node_modules was somehow created before hand, somehow, before the volume mapping was setup. It should probably NOT be created on the host, its unnecessary. This issue isn't specific to Node.js, you must have seen #550

[sam-github]

@neeraj-laad

It's because of the way the named volume is mounted.

appsody run does:

% appsody run                                                                                                                                      
Running development environment...                                                     
Pulling docker image docker.io/appsody/nodejs:0.3                                   
Running command: docker pull docker.io/appsody/nodejs:0.3                           
0.3: Pulling from appsody/nodejs                                                       
Digest: sha256:32bb268006b1607a534d76ca042018d46237273706b4a8c224db1afd7a725bfb                                                                                               
Status: Image is up to date for appsody/nodejs:0.3                                     
docker.io/appsody/nodejs:0.3                                                           
Running command: docker run --rm -p 3000:3000 -p 9229:9229 --name ex-bug-dev -v /home/sam/w/cloud-node/ex-bug/:/project/user-app -v ex-bug-deps:/project/user-app/node_modules
 -v appsody-controller-0.3.3:/.appsody -t --entrypoint /.appsody/appsody-controller docker.io/appsody/nodejs:0.3 --mode=run

Using a chopped down version of the above, I reproduced using an ubuntu image that runs echo and exits, so its not anything that happens in the container while its running, see below:

// There is no node_modules
cloud-node/ex-bug % docker run --rm  -v /home/sam/w/cloud-node/ex-bug/:/project/user-app -v ex-bug-deps:/project/user-app/node_modules  ubuntu echo hello
hello
// the -v caused node_modules to be created, probably by dockerd, and apparently as root...
cloud-node/ex-bug % ls -ld node_modules
drwxr-xr-x 2 root root 4096 Dec 12 14:30 node_modules/
cloud-node/ex-bug % rm -rf node_modules 
// If there is an existing node_modules, it isn't created, it just remains
cloud-node/ex-bug % mkdir node_modules
cloud-node/ex-bug % docker run --rm  -v /home/sam/w/cloud-node/ex-bug/:/project/user-app -v ex-bug-deps:/project/user-app/node_modules  ubuntu echo hello
hello
// Note the existing one didn't have its permissions changed
cloud-node/ex-bug % ls -ld node_modules                                                                                                                  
drwxrwxr-x 2 sam sam 4096 Dec 12 14:30 node_modules/

The first mapping -v $PWD:/project/user-app is necessary to cause the issue.

My theory is that the named volume ex-bug-dep that appsody created is mounted into the container as mount ... /project/user-app/node_modules, and mount usually mounts a FS over an existing directory, so when that directory doesn't exist, docker creates it, expecting it be created only in the container, as root, but because of the earlier mapping it ends up being created on the host FS.

The fix might be for appsody run to create the APPSODY_DEPS directory on the host before doing the docker run. It might also be possible to change how the deps volume is mounted into the container.

[neeraj-laad]

The DEPS volume was done after the USER_APP mount intentionally and from what I can recall the node_modules folder was not showing up on the host system - which is the behaviour we were after.

Something seems to have changed along the way with the CLI or with docker itself. Still investigating further.

[sam-github]

https://github.com/appsody/stacks/issues/593, APPSODY_USER_RUN_AS_LOCAL might be a fix for this.

[Kamran64]

@sam-github - Hope you don't mind I changed the title slightly. We'll use this issue to track the general problem of node stacks creating root owned files on the host machine (rather than just express). I'm not sure if any stacks other than the node ones also share this problem but I will reupdate the issue if that's the case so we can cover all stacks.

[sam-github]

Hi, so this is a blocker for:

• https://github.com/appsody/stacks/issues/622
• https://github.com/appsody/stacks/pull/713
• https://github.com/appsody/stacks/issues/593
• https://github.com/appsody/stacks/issues/530

(at least).

The problem is caused by the unfortunate interaction between a few docker behaviours, as used by appsody.

• When docker -v is mounting from a volume into a container, and the target mount directory does not exist, the targer is created (as uid 0), and the volume is mounted to the target, so that the volume / ("root") is uid 0 owned. This happens even if the contents of the volume are not root-owned.

This is a deal-breaker for non-root containers. npm install (or any language specific attempt to install deps) will fail if it tries to create files in a uid 0 APPSODY_DEPS (if its not running as root).

• When appsody mounts local host FS to /project/user-app, and then mounts the deps volume to /project/user-app/DEP, that target volume gets created on the host FS, as uid 0.

This is arguably only "cosmetic". It prevents the user on the host from installing any deps for their app, at least until they delete the root-owned DEPS folder.

These have an odd interaction, though.

docker -v VOL:/some/location doesn't always mount the root of VOL as uid 0... what it actually does is use the owner uid/git of /some/location, but only if it exists.

In appsody's case, it doesn't exist, so gets created as root, and thus, the volume is mounted as root, but it doesn't have to work that way.

I suggest 2 fixes:

1. If APPSODY_DEP is a sub-path of APPSODY_MOUNT, then it should be created on the host OS before run.
2. Before the controller is run, a chmod -R USER:GROUP should be run by appsody, to ensure that the deps volume has the correct permissions.

Demo:

appsody/ex (master *% u=) % docker volume rm DEP; rm -rf dep; mkdir dep; docker run --rm -it -v $PWD:/project -v DEP:/project/dep node:12 chown -R node:node /project/dep; docker run --user node --rm -it -v $PWD:/project -v DEP:/project/dep node:12 sh -c "ls -l /project && touch /project/dep/whatever"; ls -l  dep
DEP
total 4
drwxr-xr-x 2 node node 4096 Mar 20 23:40 dep
total 8
drwxrwxr-x 2 sam sam 4096 Mar 20 16:40 ./
drwxrwxr-x 3 sam sam 4096 Mar 20 16:40 ../

success, dep created in container, host dep is user-owned

appsody/ex (master *% u=) % docker volume rm DEP; rm -rf dep; docker run --rm -it -v $PWD:/project -v DEP:/project/dep node:12 chown -R node:node /project/dep; docker run --user node --rm -it -v $PWD:/project -v DEP:/project/dep node:12 sh -c "ls -l /project && touch /project/dep/whatever"; ls -l  dep 
DEP
total 4
drwxr-xr-x 2 node node 4096 Mar 20 23:41 dep
total 8
drwxr-xr-x 2 root root 4096 Mar 20 16:41 ./
drwxrwxr-x 3 sam  sam  4096 Mar 20 16:41 ../

Sortof sucess... dep was created in container, but host folder created as root.

appsody/ex (master *% u=) % docker volume rm DEP; rm -rf dep; docker run --user node --rm -it -v $PWD:/project -v DEP:/project/dep node:12 sh -c "ls -l /project && touch /project/dep/whatever"; ls -l  dep 
DEP
total 4
drwxr-xr-x 2 root root 4096 Mar 20 23:41 dep
touch: cannot touch '/project/dep/whatever': Permission denied
total 8
drwxr-xr-x 2 root root 4096 Mar 20 16:41 ./
drwxrwxr-x 3 sam  sam  4096 Mar 20 16:41 ../

Failure, the current case, dep can't be created in-container, and host folder is created root-owned.

Or maybe there is another way around this, but without this fixed, I can't see a way to create a nodejs stack that runs as non-root in dev/test mode.

So basically, the stack can either be certified, but not runnable in local dev by appsody, or not certifiable, but runnable in local dev.

I'm happy to be proved wrong! :-) If there is a way to use APPSODY_DEP and have its root be owned by the container user, would love to know.

Things I tried that did not work: using any combination of mount options with --mount to specify the desired permissions. See https://github.com/moby/moby/issues/21259#issuecomment-238024179 , docker has taken the stance that its our problem for not setting up the permissions of the mount target correctly.

[sam-github]

@neeraj-laad Simple non-nodejs reproduction: https://github.com/appsody/stacks/pull/739