Command preview:pre-generate not work on Docker NC 23 and 24

[cesarjhony]

Hello everyone,

I'm new in nextcloud's world, I have been testing the nextcloud as a online gallery where I have a external sync folder collecting photos in a folder and nextcloud should show them as gallery (external storage).

At every new file I make a command: ./occ preview:generate-all --path=/username/files/SyncFolder All work as expected, but it is very slow.

In the previewgenerator docs says "Add a (system) cron job for ./occ preview:pre-generate ", but when it is executed it don't return any error, any response, and quick finish. I tried theses variations logged as www-data:

• Added a cron job */5 * * * * php -f /var/www/html/occ preview:pre-generate
• Executed ./occ preview:pre-generate
• Executed php -f /var/www/html/occ preview:pre-generate
• Executed /usr/local/bin/php -f /var/www/html/occ preview:pre-generate
• Executed before ./occ files:scan --all (I tried with and without this scan)

My procedure is this:

• docker compose up
• docker exec -u www-data -it app sh
  • ./occ preview:generate-all --path=/username/files/SyncFolder
• add new jpeg photos
• docker exec -u www-data -it app sh
  • ./occ files:scan --all (new files are discovered)
  • ./occ preview:pre-generate (nothing happen)
• Check in web app if it was generated (but was not)

My system is:

• Docker 4.11.1 on Windows 10
  • 512GB HDD with 4GB DDR3
  • Hyper-V mode
• Nextcloud 24 fpm alpine
  • I tried Nextcloud 23 fpm alpine
  • I tried Nextcloud 23 fpm alpine php7.4
  • I tried Nextcloud 24 apache debian
  • I tried Nextcloud 24 fpm debian
• External Storage using local in /var/syncfolder docker volume
  • I tried in /syncfolder docker volume
  • I tried SMB too and I get the same behavior
  • I tried SFTP too and I get the same behavior
• PreviewGenerator 5.0.0
  • I tried PreviewGenerator 3.4.1

The docker-compose.yml is:

version: '3'

services:
  db:
    image: mariadb:10.9
    container_name: db
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - database_nextcloud:/var/lib/mysql
    env_file:
      - db.env
    mem_limit: 200m
    #cpu_shares: 210
    #cpus: 2

  redis:
    image: redis:alpine
    container_name: redis
    restart: always
    mem_limit: 200m
    #cpu_shares: 210
    #cpus: 2

  app:
    image: nextcloud:fpm-alpine
    container_name: app
    restart: always
    volumes:
      - html_nextcloud:/var/www/html
      - ./data/nextcloud_data/config:/var/www/html/config
      - "D:/Sincronismov2:/synchost"  # SyncThing
      #- ./data/fpm/config:/usr/local/etc/php-fpm.d
      - ./data/fpm/config/php-fpm.d/www.conf:/usr/local/etc/php-fpm.d/www.conf
      - ./data/nextcloud_data/apps:/var/www/html/custom_apps
    environment:
      - MYSQL_HOST=db
      - REDIS_HOST=redis
      - PHP_MEMORY_LIMIT=400M 
    env_file:
      - db.env
    depends_on:
      - db
      - redis
    #  - imaginary
    command:
      - sh
      - -c
      - |
        apk add ffmpeg
        apk add ffmpeg
        /entrypoint.sh php-fpm
    #ports:
    #  - 9001:80  #bypass para acessar xxxxxxxx.v6.army/status >> não funciona
    mem_limit: 450m
    #cpu_shares: 200
    #cpus: 2

  cron:
    image: nextcloud:fpm-alpine
    container_name: cron
    restart: always
    volumes_from:
      - app
    #volumes:
    #  - html_nextcloud:/var/www/html
      #- ./data/nextcloud_data/config:/var/www/html/config:ro
      #- ./data/fpm/config/:/usr/local/etc/php-fpm.d/:ro
    #  - ./data/fpm/config/php-fpm.d/www.conf:/usr/local/etc/php-fpm.d/www.conf
    entrypoint: /cron.sh
    depends_on:
      - db
      - redis
    mem_limit: 200m

  web:
    build: ./proxy/web
    container_name: web
    restart: always
    volumes:
      - html_nextcloud23:/var/www/html:ro
    environment:
      - VIRTUAL_HOST=xxxxxxx.v6.army
      - LETSENCRYPT_HOST=xxxxxxx.v6.army
      - LETSENCRYPT_EMAIL=xxxxxxx@gmail.com
    depends_on:
      - app
    networks:
      - proxy-tier
      - default
    #ports:
    #  - 8080:80  #bypass
    mem_limit: 200m
    #cpu_shares: 210
    #cpus: 2

  proxy:
    build: ./proxy/reverse
    container_name: proxy
    restart: always
    ports:
      - 80:80
      - 443:443
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
    volumes:
      - ./ssl/certs:/etc/nginx/certs:ro
      - ./ssl/vhost.d:/etc/nginx/vhost.d
      - html_ssl:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - proxy-tier
    mem_limit: 200m

  letsencrypt-companion:
    image: nginxproxy/acme-companion
    container_name: letsencrypt-companion
    restart: always
    volumes:
      - ./ssl/certs:/etc/nginx/certs
      - ./ssl/acme:/etc/acme.sh
      - ./ssl/vhost.d:/etc/nginx/vhost.d
      - html_ssl:/usr/share/nginx/html
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy-tier
    depends_on:
      - proxy
    mem_limit: 200m

volumes:
  database_nextcloud:
    external: true
  html_nextcloud:
    external: true
  html_ssl:
    external: true

networks:
  proxy-tier:

[cesarjhony]

Hello,

Can Anybody comment the preview:pre-generate normal behavior, some docker experience with Linux or windows using external storage?

[Thomas-Ganter]

I run nextcloud on my NAS.
From what I understand the Docker Container simply does not include a CRON service.

What I do is run cron on the NAS itself and execute

*/10 * * * *  docker exec -itu www-data nextcloud ./occ preview:pre-generate >>/tmp/pregenerate.out 2>&1

and that works like a charm.

[cesarjhony]

@Thomas-Ganter, has your NC a external storage? The great problem for me it is detect new photos on external storage.

[Thomas-Ganter]

Aaah — now I see. Sorry, i was not careful enough understanding your problem.

I fear your scenario will not work.
I read your description of external sync folder as folder that stuff gets put in from outside nextcloud not by means of API or GUI.

If my reading is correct then you are out of luck.
The pregeneration utilizes file modifications inside the nextcloud database to figure out what is new, not a filesystem scan.

[cesarjhony]

Thank you Thomas.

[Thomas-Ganter]

@cesarjhony — have you tried issuing a ./occ scan:files command first?

Like described in the Documentation?

The Preview Generator adds a listener that acts upon file write events in nextcloud. I am not too savvy when exactly these events are fired, but it might be worth a shot.

What should work in any case is to not copy the files into your directory directly but rather upload them through the interface. Here is a link to a blog post how to do this. You could then even automate this by using a staging directory:

cd your_staging_directory
find . \
  -newer .lastscan \
  ! -name .lastscan \
  -exec curl_into_your_nextcloud {} \; \
  -exec here_you_could_run_a_diff_between_staging_and_final_directory {} \; \
  -rm
touch .lastscan

There are certainly edge cases but this should work and might be your best shot.