Misleading application logging information

[lcrilly]

When explaining the stderr and stdout options, https://unit.nginx.org/configuration/#applications mentions that

When running in --no-daemon mode, application output is always redirected to Unit’s log file.

This is misleading because only stderr is redirected. stdout still defaults to /dev/null and cannot be redirected when using --no-daemon mode.

Fixing that is a separate issue (https://github.com/nginx/unit/issues/1178) but for now the docs should be clearer.

[ac000]

Hopefully this will clear things up...

When running with --no-daemon, stdout goes to the console (tty unit is running on) and stderr goes to the unit log. E.g

$ ps -ef | grep unit
andrew       804     775  0 13:23 pts/1    00:00:00 unit: main v1.33.0 [/opt/unit/sbin/unitd --no-daemon]
...
andrew       809     808  0 13:23 pts/1    00:00:00 unit: "test" application
$ ls -l /proc/809/fd/{1,2}
lrwx------ 1 andrew andrew 64 May 23 13:23 /proc/809/fd/1 -> /dev/pts/1
l-wx------ 1 andrew andrew 64 May 23 13:23 /proc/809/fd/2 -> /opt/unit/unit.log

The stdout and stderr config options have no effect when running with --no-daemon.

When running as a daemon (I.e without --no-daemon) then we have

$ ps -ef | grep unit
andrew       900       1  0 13:33 ?        00:00:00 unit: main v1.33.0 [/opt/unit/sbin/unitd]
...
andrew       909     908  0 13:33 ?        00:00:00 unit: "test" application
$ ls -l /proc/909/fd/{1,2}
lrwx------ 1 andrew andrew 64 May 23 13:33 /proc/909/fd/1 -> /dev/null
l-wx------ 1 andrew andrew 64 May 23 13:33 /proc/909/fd/2 -> /opt/unit/unit.log

When setting the stdout & stderr config settings, e.g

            "stdout": "/tmp/unit-stdout.log",                                   
            "stderr": "/tmp/unit-stderr.log",

When running with--no-daemon we get

$ ps -ef | grep unit
andrew      1102     775  0 13:42 pts/1    00:00:00 unit: main v1.33.0 [/opt/unit/sbin/unitd --no-daemon]
...
andrew      1107    1106  0 13:42 pts/1    00:00:00 unit: "test" application
$ ls -l /proc/1107/fd/{1,2}
lrwx------ 1 andrew andrew 64 May 23 13:46 /proc/1107/fd/1 -> /dev/pts/1
l-wx------ 1 andrew andrew 64 May 23 13:46 /proc/1107/fd/2 -> /opt/unit/unit.log

No change...

Running as a daemon

$ ps -ef | grep unit
andrew      1220       1  0 13:47 ?        00:00:00 unit: main v1.33.0 [/opt/unit/sbin/unitd]
...
andrew      1229    1228  0 13:47 ?        00:00:00 unit: "test" application
$ ls -l /proc/1229/fd/{1,2}
l-wx------ 1 andrew andrew 64 May 23 13:48 /proc/1229/fd/1 -> /tmp/unit-stdout.log
l-wx------ 1 andrew andrew 64 May 23 13:48 /proc/1229/fd/2 -> /tmp/unit-stderr.log

The key to fixing https://github.com/nginx/unit/issues/1178 is to revert the change that started running unit with --no-daemon.

[MagnusOxlund]

What happens to stdout when Unit is run as a systemd service (with --no-daemon)?

[MagnusOxlund]

To answer my own question, it looks like stdout is sent to the systemd journal, which is different from Unit's own log file where stderr is sent.

How come stderr and stdout are separated?

Would it be possible to direct both to the same place?

Directing the output from within the service file doesn't seem to have any effect (but maybe that's the same issue as nginx/unit#1178)?

[Service]
StandardError=journal
StandardOutput=journal

[ac000]

With --no-daemon, stdout goes to the console, stderr goes to the unit log. When running under systemd, because there is then no direct console systemd will get stdout

Would it be possible to direct both to the same place?

That should work when not using --no-daemon and using the stdout & stderr config options.

[ac000]

On Fri, 24 May 2024 10:08:00 -0700 MagnusOxlund @.***> wrote:

That's not what I'm seeing.

You won't by default and with the current systemd unit file.

[Service] Type=simple

That needs to be

Type=forking

Environment="UNITD_OPTIONS=--log /var/log/unit/unit.log --pid /var/run/unit/unit.pid" ExecStart=/usr/sbin/unitd $UNITD_OPTIONS --no-daemon

You need to remove '--no-daemon'...

@.** ~]$ cat /var/www/my_project/app/config.json { "listeners": { ":80": { "pass": "applications/test" } },

"applications": {
    "test": {
        "type": "python 3.6",
        "working_directory": "/var/www/my_project/app",
        "path": "/var/www/my_project/app",
        "home": "/var/www/my_project/venv",
        "module": "app",
        "callable": "app",
        "user": "opc",
        "group": "opc",
        "protocol": "wsgi"
    }
}

}

You don't seem to have stdout or stderr set...

[MagnusOxlund]

Thank you for detailing how to configure stdout and stderr when running Unit without --no-daemon. My apologies for the deleted comment you've quoted above. I removed the comment when I realized I had misread your previous response.

Personally, I'm trying to direct both stdout and stderr to Unit's systemd journal to streamline log management. It's been a challenge to get stderr to appear there. Setting StandardError and StandardOutput in the service file doesn't seem to affect the output, but if it's due to misconfiguring on my end, I'll be happy to open a discussion on the main repo rather than discuss it here.

I'm also curious if there is a technical advantage to separating the two streams by default in --no-daemon mode?

For context, on the Nginx Slack community, I brought up Apache's mod_wsgi's previous default behavior of ignoring WSGI apps' stdout to encourage compliance with PEP-3333 among WSGI middleware. From your clarification, however, it sounds like Unit doesn't actually suppress stdout and that the behavior is not specific to WSGI apps?

Thank you.

[ac000]

Personally, I'm trying to direct both stdout and stderr to Unit's systemd journal to streamline log management.

Have you tried this configuration...

In /etc/systemd/system/unit.service

[Service]
Type=forking
Environment="UNITD_OPTIONS=--log /var/log/unit/unit.log --pid /var/run/unit/unit.pid"
ExecStart=/usr/sbin/unitd $UNITD_OPTIONS
...
StandardOutput=journal
StandardError=journal

And in your unit application section

"stdout": "/dev/stdout",
"stderr": "/dev/stderr"

From your clarification, however, it sounds like Unit doesn't actually suppress stdout and that the behavior is not specific to WSGI apps?

IIRC stdout is suppressed, at least for some language modules... as I noted in the commit message

    NOTE:

    You may or may not see stuff printed to stdout as stdout was
    traditionally used by CGI applications to communicate with the
    webserver.

[MagnusOxlund]

Have you tried this configuration...

I have. Unfortunately stderr only shows up in /var/log/unit/unit.log.

I can get stdout to appear in the journal if I use --no-daemon but not without it.

[ac000]

Without --no-daemon with a unit application config of

"stdout": "/dev/stdout",
"stderr": "/dev/stderr"

Where does stdout/stderr go?