frappe / frappe_docker

Docker images for production and development setups of the Frappe framework and ERPNext
MIT License
1.49k stars 1.38k forks source link

Permission denied for nginx container after enable SELinux in docker daemon #567

Closed Tus1688 closed 2 years ago

Tus1688 commented 3 years ago

Description of the issue

Hi everyone! Thank you for making ERP next deployment become easier and less complicated. I tried to harden docker daemon using SELinux. Luckily, this happen to my testing server, I create /etc/docker/daemon.json then fill it with

{
        "selinux-enabled": true
}

FYI, I have enabled selinux before updating to latest version of ERP next, and it worked as it supposed to be. Then.... everything has change after rebuilding container to latest version using docker-compose pull and docker-compose up -d, I got bad gateway when trying to access our sites.

OS: Rocky Linux 8.4 Docker: Docker version 20.10.10, build b485636 Compose: docker-compose version 1.29.2, build 5becea4c

Steps to reproduce the issue

  1. Create /etc/docker/daemon.json
  2. Fill it with
    {
        "selinux-enabled": true
    }
  3. systemctl restart docker
  4. restorecon -R -v /var/lib/docker
  5. restorecon -R -v /usr/bin
  6. cd /frappe-docker
  7. docker-compose pull
  8. docker-compose up -d
  9. Accessing out sites
  10. I got Bad Gateway in top left corner
  11. docker ps
  12. Everything is running normally except frappe/erpnext-nginx:version-13 *logs below

Observed result

Get Bad Gateway

Expected result

Can access our site normally

Stacktrace / full error message if available

rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/.package.json.UaHnNY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/.inquirer.js.tXEiEY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/objects/.choice.js.Akuf6X" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/objects/.choices.js.L0oHlX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/objects/.separator.js.usVRjY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.base.js.TjOl5V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.checkbox.js.oXxEtX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.confirm.js.mAAO0V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.editor.js.LkNTFZ" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.expand.js.ORZdPX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.input.js.gdysHY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.list.js.4gAO2V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.number.js.cbaqFW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.password.js.EG0AoZ" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.rawlist.js.GYgdWY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/ui/.baseUI.js.gKkcbX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/ui/.bottom-bar.js.5hUE6W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/ui/.prompt.js.AkrNSY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.events.js.zceOyZ" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.incrementListIndex.js.WDbbgY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.paginator.js.fjdUMV" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.readline.js.1PBltX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.screen-manager.js.xYMupW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.utils.js.cFH5lW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.index.d.ts.2v4PlY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.index.js.JNl6HX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.license.RVvVJW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.package.json.5jOkyY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.readme.md.L9bF3X" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.index.d.ts.o7WnAV" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.index.js.yjSaRX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.license.KERgVW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.package.json.rxk7iX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.readme.md.8jroHY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.index.d.ts.4D1biV" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.index.js.4GDc2V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.license.Fnc18V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.package.json.Hp6tKW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.readme.md.frsc0Y" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/.index.d.ts.wP2VXY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/.license.BW89hW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/.package.json.QaS65W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/.readme.md.yJ8e7W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/source/.index.js.0yUX9X" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/source/.templates.js.tEj4zY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/source/.util.js.8yDwKW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/color-convert/.CHANGELOG.md.RME2iY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/color-convert/.LICENSE.mxCv9W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/color-convert/.README.md.RoZO3W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/color-convert/.conversions.js.1WXBzX" failed: Permission denied (13)

output of docker info

Server:
 Containers: 13
  Running: 12
  Paused: 0
  Stopped: 1
 Images: 38
 Server Version: 20.10.10
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  selinux
 Kernel Version: 4.18.0-305.19.1.el8_4.x86_64
 Operating System: Rocky Linux 8.4 (Green Obsidian)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.775GiB
 Name: 
 ID: 
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I haven't been able to find workaround, do you have any clue? Thanks in advance 🙂

revant commented 3 years ago

except for nginx container all containers run as non root

this was a pending issue https://github.com/frappe/frappe_docker/issues/493

Tus1688 commented 3 years ago

Hi @revant, thank you for replying. I wonder if podman work with compose installation? Because, it would be awesome if all containers run as non root

revant commented 3 years ago

as mentioned in the issue, probably we'll have to build nginx image from scratch with uid:gid 1000:1000.

anyone up for that refactor?

revant commented 3 years ago

nginx unprivileged image does not run as 1000:1000 user. worker image runs as 1000:1000

Another idea is to build worker images that can be made to use configurable uid gid? It'll need to handle migration of current 1000:1000 setups with docker and kubernetes.

Any ideas what else can be done?

revant commented 2 years ago

Check if you can build the image based on PR and try it out.

continue discussion on PR

revant commented 2 years ago

can you add 1 more container to you docker-compose that fixes the vol permission. I'm trying to get it running with podman and facing the issue. I'll update here.

...
  fix-vol-permissions:
    image: frappe/frappe-worker:${ERPNEXT_VERSION}
    user: root
    command: chown -R 1000:1000 /sites /assets /logs
    volumes:
      - sites-vol:/sites
      - assets-vol:/assets
      - logs-vol:/logs
...

Note: I've not yet found a way to fix it.

Tus1688 commented 2 years ago

Sure, let me try

Update: after adding 1 more container, I still got same error. To be honest, I am not really sure where the error come from

revant commented 2 years ago

it comes from erpnext-nginx container.

https://github.com/frappe/frappe_docker/blob/abe6d670c4400f188972756da95208ab8ef6ea38/build/frappe-nginx/docker-entrypoint.sh#L7

can you try adding depends_on to erpnext-nginx service:

  erpnext-nginx:
    ...
    depends_on:
      - fix-vol-permissions
...

make sure you pull the images again after #572 is merged

Tus1688 commented 2 years ago

It seems permission issue has gone away, but new issue arise. When I open my site, I got 404 page not found

logs of erpnext-nginx:

Generating default template
Waiting for frappe-python to be available on erpnext-python port 8000
Frappe-python available on erpnext-python port 8000
Waiting for frappe-socketio to be available on frappe-socketio port 9000
Frappe-socketio available on frappe-socketio port 9000
2021/11/20 03:15:39 [notice] 1#1: using the "epoll" event method
2021/11/20 03:15:39 [notice] 1#1: nginx/1.21.4
2021/11/20 03:15:39 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2021/11/20 03:15:39 [notice] 1#1: OS: Linux 4.18.0-305.19.1.el8_4.x86_64
2021/11/20 03:15:39 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/11/20 03:15:39 [notice] 1#1: start worker processes
2021/11/20 03:15:39 [notice] 1#1: start worker process 36
2021/11/20 03:15:39 [notice] 1#1: start worker process 37

Edit: have you get it running on podman?

revant commented 2 years ago

have you get it running on podman?

not exactly. everything started but nginx kept failing. I'm using rootless podman, I think containers with ports on < 1000 may cause issues. I tried to publish it on 8080.

there is no permission error after fix-vol-permissions. the erpnext-nginx container keeps restarting even in 8080

for the 404 not found check this: https://discuss.erpnext.com/t/production-installation-using-frappe-docker/82677/2?u=revant_one

I'll close this issue. Re-open or open another if needed.