nginx file uploads now stored in /tmp/nginx_client_body, which gets removed by systemd-tmpfiles after 10 days

[pdg137]

Describe the bug

In 544c526393f15c946a77ed0f6f11f5d3b90e50b8 @Izorkin changed the default of client_body_temp_path from /var/cache/nginx/client_body to /tmp/nginx_client_body. This folder is automatically created, but since it's in tmp, it gets removed by systemd-tmpfiles if unused for 10 days, which can surprisingly break servers.

After it's broken by systemd-tmpfiles, any file uploads give an instant 500 error and you'll see something like this in the system log:

2023/10/06 13:58:42 [crit] 993#993: *37276 open() "/tmp/nginx_client_body/0000000018" failed (2: No such file or directory), client: 127.0.0.1, server: localhost, request: "POST / HTTP/1.1", host: "localhost"

See my comment below for step-by-step details on how to reproduce.

A workaround is to set this in the conf file:

client_body_temp_path /var/cache/nginx/client_body;

Notify maintainers

@fpletz @ajs124 @RaitoBezarius

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.39, NixOS, 23.05 (Stoat), 23.05pre-git`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - nixpkgs: `/run/current-system/nixpkgs`

[Izorkin]

@pdg137 needed full nginx configuration.

[pdg137]

Okay, @Izorkin, here is one way to reproduce. Turns out it's related to systemd-tmpfiles, so I'm not sure whose fault it really is.

1. Set up basic configuration with a proxy:

   services.nginx = {
    enable = true;
    virtualHosts."localhost" = { locations."/".proxyPass = "http://localhost:12345"; };
   };

   (You don't need to actually have anything running on port 12345, this is just to convince nginx to accept the upload.)

2. Test a 10k upload:

   dd if=/dev/zero of=/tmp/upload_test bs=10k count=1
   curl -v -X POST --output /dev/null -F 'data=@/tmp/upload_test' http://localhost/

   You should see a HTTP/1.1 502 Bad Gateway response.

3. Fake an old timestamp on the nginx_client_body folder:

   cd /tmp/*nginx*/tmp/
   NOW=$(date)
   date -s '2023-09-01'
   rm -r nginx_client_body
   mkdir nginx_client_body
   chown nginx:nginx nginx_client_body
   date -s "$NOW"

4. Run systemd-tmpfiles --clean and the nginx_client_body folder will disappear.
5. Try the dd command above and get a HTTP/1.1 500 Internal Server Error response.

[pdg137]

I edited the original post with details now that I understand it better.

[Izorkin]

@pdg137 check this PR - https://github.com/NixOS/nixpkgs/pull/257828

[pdg137]

That should fix it, thanks