PHP timeout when sending multiple pictures bigger than 3-4 MB

[laurent-indermuehle]

---

Expected Behavior

Upload of multiple pictures with a drag-and-drop in the web interface should process all photo.

Current Behavior

Only a random number, in a random order of photo can be uploaded.

The log is filled with those lines:

2019-10-14 18:57:45 - notice - Lychee\Modules\Photo::add (242)  - Skipped adjustment of photo (IMG_3903)
2019-10-14 18:58:03 - error - Lychee\Modules\Photo::add (113)   - The uploaded file was only partially uploaded
2019-10-14 18:58:03 - error - Lychee\Modules\Photo::add (113)   - The uploaded file was only partially uploaded
2019-10-14 18:58:03 - error - Lychee\Modules\Photo::add (113)   - The uploaded file was only partially uploaded

Environment

OS: CentOS 7 on dedicated kimsufi (Atom, 4GB, 2TB) CPU architecture: x86_64 How docker service was installed: docker-compose (not Swarm mode)

Command used to create docker container

version: "3.5"

networks:
  pubnet:
    name: pubnet
  privnet:
    name: privnet
    internal: true 

 lycheefamily:
    image: linuxserver/lychee:amd64-v3.2.16-ls32
    networks:
      - pubnet
      - privnet
    volumes:
      - lycheefamilyconfvol:/config:rw
      - lycheefamilypicturesvol:/pictures:rw
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=pubnet"
      - "traefik.http.routers.lycheefamily_http.rule=HostRegexp(`galerie.example.com`)"
      - "traefik.http.routers.lycheefamily_http.entrypoints=http"
      - "traefik.http.middlewares.https_redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.lycheefamily_http.middlewares=https_redirect"
      - "traefik.http.routers.lycheefamily.rule=HostRegexp(`galerie.example.com`)"
      - "traefik.http.routers.lycheefamily.entrypoints=https"
      - "traefik.http.routers.lycheefamily.tls=true"
      - "traefik.http.routers.lycheefamily.tls.certresolver=ovh"
      - "traefik.http.routers.lycheefamily.tls.domains[inbo].main=galerie.example.com"
    environment:
      TZ: "Europe/Zurich"
      PUID: 1000
      PGID: 1000
    restart: unless-stopped

volumes:
  mariadbfamilyvol:
  lycheefamilyconfvol:
  lycheefamilypicturesvol:

Docker logs

Docker logs

all green

Nginx logs

2019/10/14 20:58:03 [error] 645#645: *446 upstream timed out (110: Operation timed out) while sending request to upstream,
 client: 172.18.0.3, server: _, request: "POST /php/index.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "gale
rie.example.com", referrer: "https://galerie.example.com/"

PHP logs

[14-Oct-2019 18:56:05] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

[laurent-indermuehle]

I think the issue is that PHP hasn't been configured as it should be by the file /config/lychee/user.ini If I look on "post_max_size" and "upload_max_filesize" in user.ini I have : 400M, 200M But if I logged in the container the php --info says 8M, 2M

I've taken the settings recommended by Lychee and added them in /config/php/php-local.ini and restarted the container (docker-compose restart lycheefamily).

This time, the PHP settings seems good :

root@6a4294c3bf16:/# php --info | grep -E "(post_max|upload_max)"
post_max_size => 100M => 100M
upload_max_filesize => 20M => 20M

[aptalca]

2 ways to fix it

1) Under lychee's config folder, under /lychee, there is a file called user.ini, you can put your php settings in there. That's a lychee feature 2) Alternatively our nginx baseimage that this image uses, allows for editing php variables via two external files: /config/php/php-local.ini, any value in here will override what's in php.ini, and there is /config/php/www2.conf and that will override www.conf that php uses

[laurent-indermuehle]

By the way, thanks to the linuxserver team for the great images and support :)

Thanks @aptalca for your reply.

Regarding your point 1. : I think I've demonstrated that the file /config/lychee/user.ini does nothing. I've looked in the file docker-lychee/blob/master/root/etc/cont-init.d/40-config line 17 :

cp /config/lychee/user.ini /usr/share/webapps/lychee/.user.ini

Neither nginx or php are reading in /usr/share/webapps/lychee right ? And also, this is executed after line 46 of the Dockerfile :

composer install -d /usr/share/webapps/lychee && \
 echo "**** cleanup ****" && \
 rm -rf \
    /tmp/*

And above all : Even with my modifications of PHP, which means that it should accept bigger pictures now, I still have many pictures failing to upload with the exact sames errors :(

[aptalca]

.user.ini is handled by lychee itself, we merely make it available under /config for user customization and copy it into the container during start.

I just checked my local install and my user.ini file under /config/lychee contains the following:

max_execution_time = 200
post_max_size = 2000M
upload_max_size = 2000M
upload_max_filesize = 2000M
max_file_uploads = 2000

My php-local.ini only sets the timezone.

With the above, I am able to upload large videos ~300MB

[laurent-indermuehle]

@aptalca Please, can you do 2 quick tests?

1) What is the output of php --info | grep -E "(post_max|upload_max)" inside your container?

2) Can you upload more than 20 pictures of 4MB+ each in the same drag-n-drop and tell me if all of them have been uploaded successfully.

In fact, I also can upload one big file. Problems arise when I try to upload a full album at once.

[aptalca]

root@f0a51d03bc71:/# php --info | grep -E "(post_max|upload_max)"
post_max_size => 8M => 8M
upload_max_filesize => 2M => 2M

As expected since my php ini is stock

But that proves the user.ini is working because prior to that I could not upload anything over 2M

I'm not able to test batch uploads at the moment. Maybe later when I have time

[laurent-indermuehle]

Additional thoughts : This problem could be caused by Traefik 2 (it use more CPU than all php-fpm combined when I upload photos) or Lychee itself.

I'll need to install VMs to test different combination. I'll get back here when I got new elements.

[laurent-indermuehle]

I may have reached a solution. Set post_max_size = 0 in /config/lychee/user.ini It solved the problem above. I could upload 20 pictures, 15MB each in a row successfully.

Additionally, I've changed 2000M to 2G. But I don't know if it has any incidence.

---

As a side note, I installed a fresh Lychee on a VM (CentOS 8, Apache, PHP7, MariaDB) and had a similar but different issue as above (_error - Lychee\Modules\Photo::add (106) - The uploaded file exceeds the upload_maxfilesize directive in php.ini). Again, the upload was only possible after I set post_max_size = 0 and upload_max_filesize = 2G. But upload_max_filesize = 2G alone didn't work.

[laurent-indermuehle]

And... problem is back :( The failing uploads are more frequent when I'm at home (far from a city on ADSL). At work on a university line, it works. It's definitely a problem with a timeout.

[laurent-indermuehle]

Testing again with Lychee on Apache with php 7.3.8. No problem. I'm beginning to search in nginx and php-fpm7 config.

[laurent-indermuehle]

Problem solved, it was Nginx !

In the file /config/nginx/site-confs/defaults into location ~ \.php$ { some timeouts needed to be raised:

           fastcgi_read_timeout 1800;
            fastcgi_send_timeout 1800;
            proxy_read_timeout 1800;
            proxy_send_timeout 1800;

Is it possible to add those values in the default file?

The PHP problems I had earlier were due to some comments I added with # instead of ;

Here's the complete configuration :

/config/php/php-local.ini:

; Edit this file to override php.ini directives and restart the container

date.timezone = Europe/Zurich

max_execution_time = 0
max_input_time = 0
max_file_uploads = 20
post_max_size = 5G
upload_max_size = 5G
upload_max_filesize = 500M
memory_limit = 512M

/config/php/www2.conf

; Edit this file to override www.conf and php-fpm.conf directives and restart the container

; Pool name
[www]
pm.max_children = 9
pm.start_servers = 3
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.max_requests = 200

/config/nginx/site-confs/default

server {
        listen 80 default_server;
        root /var/www/localhost/lychee;
        index index.html index.htm index.php;

        server_name _;
        client_max_body_size 0;

        location / {
                try_files $uri $uri/ /index.html /index.php?$args =404;
                }

         location ~* \.(?:ico|css|js|gif|jpe?g|png)$ {
                expires max;
                add_header Pragma public;
                add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }

        location ~ \.php$ {
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                # With php7-cgi alone:
                fastcgi_pass 127.0.0.1:9000;
                # With php7-fpm:
                #fastcgi_pass unix:/var/run/php7-fpm.sock;
                fastcgi_index index.php;
                include /etc/nginx/fastcgi_params;
                fastcgi_read_timeout 1800;
                fastcgi_send_timeout 1800;
                proxy_read_timeout 1800;
                proxy_send_timeout 1800;

        }
}

[aptalca]

thanks for troubleshooting