Using Nginx Unit with a ReadOnly filesystem.

[langchain-infra]

Hi all,

Trying to use Nginx unit on a read only filesystem with kubernetes and this doesn't seem to work? When I run my container with no tmp dirs mounted, I run into the following error(which is expected:

backend-cf448fcd7-z9z5c backend 2024/02/20 20:41:33 [alert] 9#9 Unable to create certificates storage directory: mkdir(/var/lib/unit/certs/) failed (30: Read-only file system)
backend-cf448fcd7-z9z5c backend 2024/02/20 20:41:33 [alert] 9#9 Unable to create scripts storage directory: mkdir(/var/lib/unit/scripts/) failed (30: Read-only file system)
backend-cf448fcd7-z9z5c backend 2024/02/20 20:41:33 [alert] 9#9 bind(6, unix:/var/run/control.unit.sock.tmp) failed (30: Read-only file system)

When I try to mount tmp dirs for the container to use doing something like this:

    volumes:
      - name: certs
        emptyDir: {}
      - name: scripts
        emptyDir: {}
      - name: control
        emptyDir: {}
    volumeMounts:
      - name: certs
        mountPath: /var/lib/unit/certs
      - name: scripts
        mountPath: /var/lib/unit/scripts
      - name: control
        mountPath: /var/run

I then run into an error where the entrypoint doesn't actually perform initialization due to the /var/lib/unit having values inside.

/usr/local/bin/docker-entrypoint.sh: /var/lib/unit/ is not empty, skipping initial configuration...
2024/02/20 20:44:49 [info] 1#1 unit 1.31.1 started
2024/02/20 20:44:49 [info] 9#9 discovery started
2024/02/20 20:44:49 [notice] 9#9 module: python 3.11.8 "/usr/lib/unit/modules/python3.unit.so"
2024/02/20 20:44:49 [info] 1#1 controller started
2024/02/20 20:44:49 [notice] 1#1 process 9 exited with code 0
2024/02/20 20:44:49 [info] 11#11 router started
2024/02/20 20:44:49 [info] 11#11 OpenSSL 1.1.1w  11 Sep 2023, 1010117f

Is the only way around this overriding the entrypoint script? I would just remove the check on the directory being full but worried that this may have some unforeseen consequences? Any guidance would be appreciated! Separately, is there a way to add a config without having to curl the control endpoint? We could potentially use that option instead as well!

[ac000]

Yeah, Unit will want to create a few file-system objects.

The obvious thing I can think of is to start unit and point the various things it wants to create to someplace where it can, even if it's just a tmpfs filesystem.

$ /opt/unit/sbin/unitd --help

unit options:

  --version            print unit version and configure options

  --no-daemon          run unit in non-daemon mode

  --control ADDRESS    set address of control API socket
                       default: "unix:/opt/unit/control.unit.sock"

  --control-mode MODE  set mode of the control API socket
                       default: 0600

  --control-user USER    set the owner of the control API socket

  --control-group GROUP  set the group of the control API socket

  --pid FILE           set pid filename
                       default: "/opt/unit/unit.pid"

  --log FILE           set log filename
                       default: "/opt/unit/unit.log"

  --modulesdir DIR     set modules directory name
                       default: "/opt/unit/modules"

  --statedir DIR       set state directory name
                       default: "/opt/unit/state"

  --tmpdir DIR         set tmp directory name
                       default: "/var/tmp"

  --modules DIR        [deprecated] synonym for --modulesdir
  --state DIR          [deprecated] synonym for --statedir
  --tmp DIR            [deprecated] synonym for --tmpdir

  --user USER          set non-privileged processes to run as specified user
                       default: "nobody"

  --group GROUP        set non-privileged processes to run as specified group
                       default: user's primary group

[tippexs]

Hi @langchain-infra

I have added an multistage build approach to this discussion thread:

https://github.com/nginx/unit/discussions/1114#discussioncomment-8385447

Usually I would try to go down this road. Build it and deploy it with r/o so nothing can be changed! With this there would be no need to mount any tmpfs as everything will already be there on container startup.

This sounds like a cool demo for kubeCon - so I am more than happy to assist.

Are you able to configure unit at build time of the container? Are you using any CI/CD integration for that?

[langchain-infra]

Hi @langchain-infra

I have added an multistage build approach to this discussion thread:

#1114 (reply in thread)

Usually I would try to go down this road. Build it and deploy it with r/o so nothing can be changed! With this there would be no need to mount any tmpfs as everything will already be there on container startup.

This sounds like a cool demo for kubeCon - so I am more than happy to assist.

Are you able to configure unit at build time of the container? Are you using any CI/CD integration for that?

@tippexs thanks for the help! So normally that would be the approach i would take as well, but we unfortunately need to be able to configure port in our helm chart. This means that I do need to have some mechanism of mounting a new config that unit would read on startup. However I can't see a format to mount directly to the state directory hence my reliance on the endpoint. This is actually exactly how we are able to use nginx as a proxy to serve our frontend files(we are now trying to use unit for our asgi servers)

[bunny-therapist]

I am running unit with asgi app in kubernetes deployments with read-only filesystem except for mounted /tmp. I did it by rewriting the unit entrypoint script to pass in tmpdir, statedir, etc both times unitd is launched, pointing them to files/directories in /tmp. I think the official entrypoint script does not pass any arguments to unitd the first time it is started, so this was required.

[langchain-infra]

@bunny-therapist nice that is very helpful and i'll likely do that as well!