search

truenas / charts

TrueNAS SCALE Apps Catalogs & Charts
BSD 3-Clause "New" or "Revised" License
306 stars 293 forks source link

Paperless-ngx App stuck at deploying #1881

Closed weiglbua closed 11 months ago

weiglbua commented 11 months ago

The paperless-ngx App is stuck in the deploying phase and the webinterface cannot be reached.

Name: paperless-ngx App Version: 2.1.1 Chart Version: 1.2.5

Kubernetes Event: 13/12/2023 17:03:14 Startup probe failed: Get "http://172.16.0.49:30070/": dial tcp 172.16.0.49:30070: connect: connection refused

This issue is reproducable on different systems according to different users encountering the same error on the Truenas forum https://www.truenas.com/community/threads/paperless-deploying-forever.114751/

Steps to reproduce:

  1. Install the paperless-ngx App in the TrueNas Scale UI by clicking the "Install" button
  2. Change the mandatory settings for username password and email
  3. Apply the settings and start the deployment of the App
  4. All images are pulled correctly according to logs but the "deploying" process never finishes and does not progress after the kubernetes event stated above
  5. The webinterface of paperless cannot be reached
stavros-k commented 11 months ago

Hello, You can check the logs (TrueNAS Bluefin): by clicking the 3 dots in the app card (TrueNAS Cobia): By clicking the log icon in under the app details

But given the limit info I have right now, maybe this comment below matches your issue?

https://github.com/truenas/charts/issues/1826#issuecomment-1850131098

weiglbua commented 11 months ago

After looking at the issue u mentioned I can confirm that I indeed run into the same problem.

After changing the user und group id both to 1000 as the deployment process got a little further but got stuck at a different point.

Here are the Logfiles:

2023-12-13 22:12:22.117501+01:00Paperless-ngx docker container starting... 2023-12-13 22:12:22.144328+01:00Creating directory /tmp/paperless 2023-12-13 22:12:22.149086+01:00Adjusting permissions of paperless files. This may take a while. 2023-12-13 22:12:22.193454+01:00Waiting for PostgreSQL to start... 2023-12-13 22:12:22.475155+01:00Waiting for Redis... 2023-12-13 22:12:22.480555+01:00Connected to Redis broker. 2023-12-13 22:12:22.539651+01:00Apply database migrations... 2023-12-13 22:12:23.840624+01:00Operations to perform: 2023-12-13 22:12:23.840747+01:00Apply all migrations: admin, auth, authtoken, contenttypes, django_celery_results, documents, guardian, paperless_mail, sessions 2023-12-13 22:12:23.840795+01:00Running migrations: 2023-12-13 22:12:23.840831+01:00No migrations to apply. 2023-12-13 22:12:24.079859+01:00Running Django checks 2023-12-13 22:12:25.551452+01:00SystemCheckError: System check identified some issues: 2023-12-13 22:12:25.551620+01:002023-12-13T22:12:25.551620707+01:00 2023-12-13 22:12:25.551660+01:00ERRORS: 2023-12-13 22:12:25.551697+01:00?: PAPERLESS_TRASH_DIR is not writeable 2023-12-13 22:12:25.551731+01:00HINT: Set the permissions of 2023-12-13 22:12:25.551765+01:00drwxr-xr-x root root /usr/src/paperless/trash 2023-12-13 22:12:25.551798+01:00to be writeable by the user running the Paperless services 2023-12-13 22:12:25.551851+01:002023-12-13T22:12:25.551851068+01:00 2023-12-13 22:12:25.551885+01:00System check identified 1 issue (0 silenced).

I understand that it is a permission problem with the trash-folder of paperless but shouldnt the system take care of the dataset structure/permission if it is set to iXVolume during the installation?

stavros-k commented 11 months ago

I understand that it is a permission problem with the trash-folder of paperless but shouldnt the system take care of the dataset structure/permission if it is set to iXVolume during the installation?

Yes, there is some work going on to fix that. This is the part 1 https://github.com/truenas/charts/pull/1882 and soon will be the rest of the apps

machinsk commented 8 months ago

This appears to still be an issue, this thread has continued with the same issue. I get the problem on the newest version.

thickmelon commented 7 months ago

I have the same issue

from the context, I think @stavros-k is saying I should enable ACL for all directories even ix-applications ones?

stavros-k commented 7 months ago

I have the same issue

from the context, I think @stavros-k is saying I should enable ACL for all directories even ix-applications ones?

No, not for ix-applications.

You only need to have correct permissions for the directories the app will use. eg: postgres data, postgres backup, paperless data directories etc.

You dont need ACLs, you can use unix permissions too. In both cases you need to give access to the proper user and group *id's eg postgres needs 999:999 paperless needs the uid/gid you entered on the installation wizard

thickmelon commented 7 months ago

Yeah actually funny thing @stavros-k, I did the proper user id and group id as you recommended in the other thread (set them as 1000), but I still got the error [Startup probe failed: Get "http://172.16.0.49:30070/": dial tcp 172.16.0.49:30070: connect: connection refused].

What was weirder was that it started running regardless after I waited for about 30 mins or so. I also remember my other apps doing this when I install them. May I ask what this error even is? It seems to be unimportant to whether the app will run or not..

stavros-k commented 7 months ago

Yeah actually funny thing @stavros-k, I did the proper user id and group id as you recommended in the other thread (set them as 1000), but I still got the error [Startup probe failed: Get "http://172.16.0.49:30070/": dial tcp 172.16.0.49:30070: connect: connection refused].

You set 1000 on both the installation wizard [Run As fileds] and on directory as permissions? Also note that paperless on each startup goes through ALL the files and checks if permissions are correct and if not tries to correct them This can be io heavy, so it can take a while if you have enough files https://github.com/paperless-ngx/paperless-ngx/blob/c4a9697e02358744884c4a247359583504078bde/docker/docker-entrypoint.sh#L100-L105

What was weirder was that it started running regardless after I waited for about 30 mins or so. I also remember my other apps doing this when I install them. May I ask what this error even is? It seems to be unimportant to whether the app will run or not..

The error is that the application did not respond to the probes, and kubernetes logging that event. Startup probes are setup to withstand many many failed responses (180). The probbing starts 30seconds after the paperless container starts and each probe after that occurs on a 5second interval So it will check if the app started for 15 minutes before it kills it. and the container restarts.

(here are the increased failure thresholds, the default interval is not showed here) https://github.com/truenas/charts/blob/e8dc7338bd219980cc5f90ba4dba5cc6dab4f102/library/ix-dev/community/paperless-ngx/templates/_paperless.tpl#L54-L56

thickmelon commented 7 months ago

Ah, so it just means that A probe failed, and kubernetes logged that ONLY. No wonder it ended up working haha. Thank you!