[Fixed in next version/ Release Candidate] File permission issues on NAS systems

TheDMV2 commented 1 month ago

File ingest issues from a fresh docker install from dockerhub.

1) the processor isn't recursively going through folders (top level only). It worked on previous version install, but had the repeated book glitch. After deleting everything and doing a complete reinstall, now it isn't processing subfolders.

"Ingest folder currently no longer looks recursively through folders, only the files in the main directory due to an oversight following a recent bugfix" - I thought this was a fixed bug?

2) no covers are shown.

jpgs are definitely there, but they are not showing up:

3) download filenames are just a number, the number being the order in which they were processed / added to my library.

/download/1/epub/1.epub

Clicking on it gives a 404 error.

jmarmstrong1207 commented 1 month ago

I just discovered this too and I'm investigating a fix. It works when copying the folder, but not when moving the folder to cwa-book-ingest.

jmarmstrong1207 commented 1 month ago

The covers issue seems to be just an error with the file as I I have no issue with the covers.

Not sure what you mean about the download filenames. If you mean when downloading epubs from the UI, it downloads with the correct title. Also probably due to an error in the file..?

TheDMV2 commented 1 month ago

Covers: 1) it worked fine on the previous version I had (1.1.0) 2) the azw3 file, on it's own, was imported, and the three files you can see in my OP were created by the import process. The cover.jpg is fine; almost exactly the same as the non-imported version. Just... isn't showing up, as you can see.

Downloads:

1) the |/ EPUB (657.23kB) link goes to {mydomainname}/download/1/epub/1.epub - which gives a 404.

Also, trying to view/read the book online doesn't work at all for the same reason.

(Both of these worked fine in the previous version)

jmarmstrong1207 commented 1 month ago

Covers:

1. it worked fine on the previous version I had (1.1.0)

2. the azw3 file, on it's own, was imported, and the three files you can see in my OP were created by the import process. The cover.jpg is fine; almost exactly the same as the non-imported version. Just... isn't showing up, as you can see.

Downloads:

1. the |/ EPUB (657.23kB) link goes to {mydomainname}/download/1/epub/1.epub - which gives a 404.

Also, trying to view/read the book online doesn't work at all for the same reason.

(Both of these worked fine in the previous version)

Oh, @crocodilestick might know as he is the sole person that updated from v1.1.0 to v1.2.0

jmarmstrong1207 commented 1 month ago

I have made PR #31 that should fix the first problem. I have tested it and should be ready for deployment, but waiting for further testing by crocodilestick.

I'm pretty sure now there aren't any further problems with the auto importing stability now. I have personally tested both cp and mv on folders of books

crocodilestick commented 1 month ago

Covers: 1) it worked fine on the previous version I had (1.1.0) 2) the azw3 file, on it's own, was imported, and the three files you can see in my OP were created by the import process. The cover.jpg is fine; almost exactly the same as the non-imported version. Just... isn't showing up, as you can see.

Downloads:

1) the |/ EPUB (657.23kB) link goes to {mydomainname}/download/1/epub/1.epub - which gives a 404.

Also, trying to view/read the book online doesn't work at all for the same reason.

(Both of these worked fine in the previous version)

Hi @TheDMV2 , is this the only book with these cover problems for you or are all of them having this issue?

TheDMV2 commented 1 month ago

All books. I only bothered putting in two books though, because it wasn't recursively processing folders, so would have taken hours to manually copy every file.

crocodilestick commented 1 month ago

Hey the @TheDMV2 , with the release of Version 1.2.2 your issues should be resolved! To update, please update your container by repulling the latest image and let us know if it helps!

TheDMV2 commented 1 month ago

Updated 1.2.2: no change. Still doesn't scan subfolders. Doesn't show covers. New books are just a number.

To bundle in more errors:

Uploading files fails partially successfully. That is, it uploads, converts, and then there's no file in the filesystem, but the book is showing up (without a cover) in the web UI.
Deleting books from the UI doesn't delete them from the filesystem.

Deleting book 2 from database only, book path in database not valid: A. A. Milne/Winnie-the-Pooh (2)

I'm certain this problem is related to that all imported files are just referenced by a number (Book ID), not the actual filenames.

Here's the logs when trying to upload & delete:

[2024-07-31 04:20:07,978] ERROR {cps.helper:582} Rename title from /tmp/calibre_web/a6cc3053ae26fa79162ea765b91ccece to /cwa/calibre-main/Calibre Library/Ben Aaronovitch/Broken Homes (PC Peter Grant) (3) failed with error: [Errno 13] Permission denied: '/cwa'
[2024-07-31 04:20:07,979] ERROR {cps:1414} Exception on /upload [POST]
Traceback (most recent call last):
  File "/lsiopy/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/lsiopy/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/lsiopy/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/lsiopy/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/app/calibre-web/cps/usermanagement.py", line 34, in decorated_view
    return func(*args, **kwargs)
  File "/app/calibre-web/cps/editbooks.py", line 59, in inner
    return f(*args, **kwargs)
  File "/app/calibre-web/cps/editbooks.py", line 271, in upload
    error = helper.update_dir_structure(book_id,
  File "/app/calibre-web/cps/helper.py", line 736, in update_dir_structure
    return update_dir_structure_file(book_id,
  File "/app/calibre-web/cps/helper.py", line 498, in update_dir_structure_file
    rename_all_files_on_change(local_book, new_path, new_path, all_new_name)
  File "/app/calibre-web/cps/helper.py", line 395, in rename_all_files_on_change
    os.makedirs(new_path)
  File "/usr/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 1 more time]
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/cwa'

[2024-07-31 04:27:04,602] ERROR {cps.helper:983} File not found: /cwa/calibre-main/Calibre Library/Ben Aaronovitch/Broken Homes (PC Peter Grant)/Broken Homes (PC Peter Grant).epub
[2024-07-31 04:30:06,892] ERROR {cps.helper:385} Deleting book 3 from database only, book path in database not valid: Ben Aaronovitch/Broken Homes (PC Peter Grant)
[2024-07-31 04:30:09,978] ERROR {cps.helper:385} Deleting book 2 from database only, book path in database not valid: A. A. Milne/Winnie-the-Pooh (2)

R/W permissions were always given:

No idea why it's saying permission denied.

TheDMV2 commented 1 month ago

To add re: not scanning subfolders: it DOES, but it only scans one folder deep.

If you copy over a normal Calibre library, it won't work, since that has books in subfolders by author.

So, the ingester will scan "A. A. Milne", but not "Winnie-the-Pooh (1)". If I copy the azw3 file to the "A.A. Milne" parent folder, it will process the file normally, creating the author and book subfolders, cover.jpg and metadata.opf files too.

jmarmstrong1207 commented 1 month ago

I have tested with subfolders within subfolders and it should work. This seems like a permission issue with your end.

Try doing chown mod recursively on the entire folder of books, and chmod to your user and group to see if it works again. /cwa permission denied indicates its either or those. If I doesn't work, try to see if the auto import works on a brand new, empty database

If it doesn't work still, could you provide the docker logs with docker logs calibre-web-automated? This is where the auto import errors go.

jmarmstrong1207 commented 1 month ago

It looks like the error log was from importing with the upload button on the webpage. We haven't modified this part of the program, so if you have issues with this, it's very likely your database and/or permissions setup is set up incorrectly.

TheDMV2 commented 1 month ago

try to see if the auto import works on a brand new, empty database

I did; no difference.

Given that everything worked in v1.10, (except for the multi-import bug), I wish I could use dockerhub to pull in old versions, but it doesn't look like versioning is set up correctly there - only "latest" shows up, with no version number, and no previous versions.

In any case, here's the log from importing the book:

2024/07/31 05:35:50 | stdout | [new-book-detector]: /etc/calibre-web-automator/cwa-import/Winnie-the-Pooh - A. A. Milne.epub successfully moved/converted, the Ingest Folder has been emptied and is ready
-- | -- | --
2024/07/31 05:35:40 | stdout | [new-book-detector]: Removing /etc/calibre-web-automator/cwa-import/Winnie-the-Pooh - A. A. Milne.epub from import folder...
2024/07/31 05:35:40 | stdout | [new-book-detector] Added /etc/calibre-web-automator/cwa-import/Winnie-the-Pooh - A. A. Milne.epub to Calibre database
2024/07/31 05:35:40 | stdout | Added book ids: 3
2024/07/31 05:35:39 | stdout | Upgrading database to version 26...
2024/07/31 05:35:39 | stdout | [books-to-process]: New files successfully moved/converted, the Ingest Folder has been emptied and is ready to go again.
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout | [new-book-processor]: Processing of new files completed in 0.00 seconds.
2024/07/31 05:35:39 | stdout | Cannot convert /cwa-book-ingest/A. A. Milne/@eaDir/Winnie-the-Pooh - A. A. Milne.azw3@SynoResource
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout | [new-book-processor]: Converting file from to epub format...
2024/07/31 05:35:39 | stdout | [new-book-processor]: No epub files found in the current directory. Starting conversion process...
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout | [books-to-process]: New files detected - /cwa-book-ingest/A. A. Milne/@eaDir/Winnie-the-Pooh - A. A. Milne.azw3@SynoResource
2024/07/31 05:35:39 | stdout | [books-to-process]: New files successfully moved/converted, the Ingest Folder has been emptied and is ready to go again.
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout | [new-book-processor]: Processing of new files completed in 2.50 seconds.
2024/07/31 05:35:39 | stdout | [new-book-processor]: All new epub files have now been moved to the calibre-web import folder.
2024/07/31 05:35:39 | stdout | [new-book-processor]: conversion to .epub format completed succsessfully in 2.50 seconds.
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout | [new-book-processor]: END_CON: Conversion of Winnie-the-Pooh - A. A. Milne.azw3 complete in 2.50 seconds.
2024/07/31 05:35:39 | stdout |  
2024/07/31 05:35:39 | stdout | Output saved to   /etc/calibre-web-automator/cwa-import/Winnie-the-Pooh - A. A. Milne.epub
2024/07/31 05:35:39 | stdout | EPUB output written to /etc/calibre-web-automator/cwa-import/Winnie-the-Pooh - A. A. Milne.epub
2024/07/31 05:35:39 | stdout | [0m[new-book-detector]: New file detected: /etc/calibre-web-automator/cwa-import/Winnie-the-Pooh - A. A. Milne.epub
2024/07/31 05:35:39 | stdout | [33mThe cover image has an id != "cover". Renaming to work around bug in Nook Color
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0016.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0015.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0012.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0002.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0001.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0000.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0014.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0013.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0011.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0008.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0009.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0010.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0005.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0004.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0003.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0007.html...
2024/07/31 05:35:38 | stdout | No large trees found
2024/07/31 05:35:38 | stdout | Looking for large trees in text/part0006.html...
2024/07/31 05:35:38 | stdout | Splitting markup on page breaks and flow limits, if any...
2024/07/31 05:35:38 | stdout | 67% Running EPUB Output plugin
2024/07/31 05:35:38 | stdout | Creating EPUB Output...
2024/07/31 05:35:38 | stdout | Trimming 'images/00002.jpeg' from manifest
2024/07/31 05:35:38 | stdout | Trimming unused files from manifest...
2024/07/31 05:35:38 | stdout | Cleaning up manifest...
2024/07/31 05:35:38 | stdout | Removing fake margins...
2024/07/31 05:35:38 | stdout | Source base font size is 12.00000pt
2024/07/31 05:35:38 | stdout | Flattening CSS and remapping font sizes...
2024/07/31 05:35:38 | stdout | Detecting structure...
2024/07/31 05:35:38 | stdout | Merging user specified metadata...
2024/07/31 05:35:37 | stdout | 34% Running transforms on e-book...
2024/07/31 05:35:37 | stdout | Parsing all content...
2024/07/31 05:35:37 | stdout | Found KF8 MOBI of type 'standalone'
2024/07/31 05:35:37 | stdout | on /cwa-book-ingest/A. A. Milne/Winnie-the-Pooh - A. A. Milne.azw3
2024/07/31 05:35:37 | stdout | InputFormatPlugin: MOBI Input running
2024/07/31 05:35:37 | stdout | 1% Converting input to HTML...
2024/07/31 05:35:36 | stdout |  
2024/07/31 05:35:36 | stdout | [new-book-processor]: START_CON: Converting Winnie-the-Pooh - A. A. Milne.azw3...
2024/07/31 05:35:36 | stdout |  
2024/07/31 05:35:36 | stdout | [new-book-processor]: Converting file from to epub format...
2024/07/31 05:35:36 | stdout | [new-book-processor]: No epub files found in the current directory. Starting conversion process...
2024/07/31 05:35:36 | stdout |  
2024/07/31 05:35:36 | stdout | [books-to-process]: New files detected - /cwa-book-ingest/A. A. Milne/Winnie-the-Pooh - A. A. Milne.azw3

On the topic of permissions, as a test I changed from running it as me (UID 1026/GID 100) to root (0/0), and it successfully dealt with processing and importing all subfolders, and deleting everything.

However: filename is still just book id, and there's no cover displayed because of this.

Odd. I've just changed the IDs back... and the UID is showing as 0 on boot up in the docker logs, but it is definitely 1026 in settings. GID was changed (back to 100).

And when set back to me, the importer is again only able to go one folder deep.

Further cover testing: I can manually add a cover, and it displays fine.

TheDMV2 commented 1 month ago

Further testing: I've done a complete wipe of everything (all folders, and docker image, etc), and redownloaded / rebuilt with compose.

PUID: resolved; it's back to showing 1026

Recursive subfolders: nope, just the first subfolder.

Covers & book downloading & book deleting: yes! This is working now.

Uploading: Not quite working; azw3 file isn't processed, and metadata can't be saved. File IS uploaded.

Error editing book: [Errno 13] Permission denied: '/etc/calibre-web-automator/metadata_change_logs/20240731093016-2.json'

Tried web uploading an epub, no difference. Tried downloading or reading - back to the broken 404 errors. Wiped my cwa library again, turned off uploads, re-copied the azw3 book (just to the root ingest folder), and it worked fine, covers display and reading online works fine.

However, if I try to edit the metadata - even if I don't actually edit anything, and just hit save - I get the same error:

Error editing book: [Errno 13] Permission denied: '/etc/calibre-web-automator/metadata_change_logs/20240731094355-1.json'

[2024-07-31 09:46:06,443] ERROR {cps.metadata_provider.amazon:127} 503 Server Error:  for url: https://www.amazon.com/s?k=The+Long+Way+to+a+Small,+Angry+Planet&i=digital-text&sprefix=The+Long+Way+to+a+Small,+Angry+Planet%2Cdigital-text&ref=nb_sb_noss
[2024-07-31 09:46:06,443]  INFO {scholarly:96} Getting https://scholar.google.com/scholar?hl=en&q=The%20Long%20Way%20to%20a%20Small%252C%20Angry%20Planet&as_vis=0&as_sdt=0,33
[2024-07-31 09:46:08,218]  INFO {httpx:1026} HTTP Request: GET https://scholar.google.com/scholar?hl=en&q=The%20Long%20Way%20to%20a%20Small%252C%20Angry%20Planet&as_vis=0&as_sdt=0,33 "HTTP/1.1 200 OK"
[2024-07-31 09:46:20,919] ERROR {cps.editbooks:229} [Errno 13] Permission denied: '/etc/calibre-web-automator/metadata_change_logs/20240731094620-2.json'

jmarmstrong1207 commented 1 month ago

Could you provide your docker compose for reference?

jmarmstrong1207 commented 1 month ago

And could you check if regular Calibre web gives you an issue?

jmarmstrong1207 commented 1 month ago

I'm not sure how Synology works, but it looks like it's putting @eaDir and other @ tags into the directory pathname, messing up the auto importing system and possibly also regular Calibre webs web uploading. This sounds more of a Synology issue

If even web uploading isn't working, then the problem isn't due to this project

ayalon commented 1 month ago

I don't think so. I have the same issue also with 1.2.3 The problems are with the permissions. I have set

    environment:
      - PUID=1024
      - PGID=101

But the folder "/etc/calibre-web-automator" and all its files have the following owner: drwxr-xr-x 1 1000 100 296 Jul 29 22:28

Therefore the docker image does not adapt the permissions of folder. I can fix it manually, but after updating the docker image the permissions are lost again.

crocodilestick commented 1 month ago

I don't think so. I have the same issue also with 1.2.3 The problems are with the permissions. I have set
    environment:
      - PUID=1024
      - PGID=101
But the folder "/etc/calibre-web-automator" and all its files have the following owner: drwxr-xr-x 1 1000 100 296 Jul 29 22:28

Therefore the docker image does not adapt the permissions of folder. I can fix it manually, but after updating the docker image the permissions are lost again.

Hey @ayalon , that should be fixed in the next update when our Dockerfile is finished but may I ask what reason you have to change the persmissions from the default?

ayalon commented 1 month ago

The permissions are bound to the host system. If you use Unraid or Synology, this is the documented calibre way of ensuring, that the host system (where I have a user and a group as documented above) can access the files. And they do not match the abc user and GROUP of your container.

I don't think, that you need to create custom docker image to solve the issue. Calibre uses the linuxserver.io and this base image already supports custom user UID / GUID. You just have to make sure, that your folder with all your files is chowned with the values provided to the docker image.

ayalon commented 1 month ago

I have fixed it meanwhile by adding a Custom Script to the the image: https://docs.linuxserver.io/general/container-customization/#custom-scripts - /home/foo/appdata/my-custom-files:/custom-cont-init.d:ro

In this script I change the permissions of the folder. Then the change persists.

crocodilestick commented 1 month ago

The permissions are bound to the host system. If you use Unraid or Synology, this is the documented calibre way of ensuring, that the host system (where I have a user and a group as documented above) can access the files. And they do not match the abc user and GROUP of your container.

I don't think, that you need to create custom docker image to solve the issue. Calibre uses the linuxserver.io and this base image already supports custom user UID / GUID. You just have to make sure, that your folder with all your files is chowned with the values provided to the docker image.

@ayalon Ah okay good to know, we weren't planning on creating a custom docker images, the images so far have been made without a Dockerfile and so the new files are unaffected by the environment variables you enter into your docker compose. This should be fixed with the next update when the images from now on will be built with the Dockerfile currently in testing

Also would you mind sharing your custom init script so that it may also be able help @TheDMV2 until the new images release?

kainzilla commented 1 month ago

This should be fixed with the next update when the images from now on will be built with the Dockerfile currently in testing

Thanks for working on this! Synology isn't the only scenario where specific UID / GIDs are needed, as an example OpenShift kubernetes wants containers to run as the namespace UID / GIDs, and podman rootless containers also do a UID / GID remap as a security measure - this fix should allow CWA to run on those platforms without issue :partying_face:

iroQuai commented 1 month ago

I'm running into the same problems on Synology. Looking forward to the update that fixes this, because now editing metadata is impossible (error 13)

crocodilestick commented 1 month ago

I'm running into the same problems on Synology. Looking forward to the update that fixes this, because now editing metadata is impossible (error 13)

Hey @iroQuai , I'm just putting the finishing touches on V2.0.0 which will be released soon (next couple days) however in the meantime feel free to use the release candidate build using the following docker compose (fully stable but only a couple minor features missing):

---
services:
  calibre-web-automated:
    image: crocodilestick/calibre-web-automated:release-candidate
    container_name: cwa-v2.0.0-rc
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=UTC
      - DOCKER_MODS=linuxserver/mods:universal-calibre
    volumes:
      - . :/config
      - . :/cwa-book-ingest
      - . :/calibre-library
      #- /path/to/where/you/keep/your/books:/books #Optional
      #- /path/to/your/gmail/credentials.json:/app/calibre-web/gmail.json #Optional
    ports:
      - 8084:8083 # Change the first number to change the port you want to access the Web UI, not the second
    restart: unless-stopped

iroQuai commented 1 month ago

I wanted to wait on the new release but was too impatient. Just tried the RC you mentioned. It does fix the permission problems for me, so now I can edit metadata!

There is a bigger issue for me now: import doesn't work anymore! Logs mention everything works like usual and file gets deleted from the ingest folder. Only thing: the file doesn't show up in CWA or in the books folder!

Is there a place within the container those files might still be?

crocodilestick commented 1 month ago

Hey @iroQuai , the RC has been tested for a while now but a few different people and no one else has had that issue. I would suggest making sure you're only running one instance of CWA and that you haven't accidentally made multiple libraries

iroQuai commented 1 month ago

I'll start over and do a clean install to test again! Will report back

0x09AF commented 3 days ago

I know this has been marked as Completed but I am having issues with deploying v2.0.1. My set up id docker in LXC with mounted Synology NAS as a NFS share. internal container's root user has no permissions to the share as well as chown is not allowed. I've had lots of permissions issues setting up Paperless-NGX in docker and resolved them by creating folders locally to the container (where chown is allowed) and then movind them to the NFS share.
That trick doesn't work here as the container installs necessary packages and starts as the root user and then changes to abc every container start. Is there a way to change something about the container itself or adapt it to my setup so that we don't do chown? I would rather not change anything NFS-related as I have 10+ containers mapped and working.
docker logs

. . .
calibre-web-automated  | [custom-init] No custom files found, skipping...
calibre-web-automated  | [cwa-set-perms] Recursively setting ownership of everything in /config, /calibre-library, /app/calibre-web-automated to abc:abc...
calibre-web-automated  | [cwa-set-perms] Successfully set permissions for '/config'!
calibre-web-automated  | chown: changing ownership of '/calibre-library/metadata.db': Operation not permitted
calibre-web-automated  | chown: changing ownership of '/calibre-library': Operation not permitted
calibre-web-automated  | [cwa-set-perms] Service could not successfully set permissions for '/calibre-library' (see errors above).
. . .

My compose file

---
services:
  calibre-web-automated:
    image: crocodilestick/calibre-web-automated:v2.0.1
    container_name: calibre-web-automated
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
      - DOCKER_MODS=linuxserver/mods:universal-calibre
    volumes:
      - ./config:/config
      - /data/books_in:/cwa-book-ingest
      - /data/media/books:/calibre-library
    ports:
      - 8084:8083 # Change the first number to change the port you want to access the Web UI, not the second
    restart: unless-stopped

ls -ln /data/

. . .
drwxr-xr-x 1  1000  1000    0 Sep  4 09:47  books_in
. . .

ls -ln /data/media/

. . .
drwxr-xr-x 1  1000  1000    22 Sep  4 09:23 books
. . .

mount

. . .
10.10.1.20:/volume1/media on /data type nfs4 (rw,nosuid,nodev,noexec,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.10.1.25,local_lock=none,addr=10.10.1.20)
. . .

EDIT: add LXC user and group $ id -u; id -g

1000
1000

Thank you

TheDMV2 commented 3 days ago

You should probably update your PID/GID settings to your PID/GIDs first.

0x09AF commented 2 days ago

You should probably update your PID/GID settings to your PID/GIDs first.

What should they be? I set them to 1000:1000. They all match across the board. I updated my previous post with my user's ID and group ID

TheDMV2 commented 1 day ago

You should probably update your PID/GID settings to your PID/GIDs first.

What should they be? I set them to 1000:1000. They all match across the board. I updated my previous post with my user's ID and group ID

Let me word that more clearly.

In Calibre-Web-Automated, there are two environment settings which set Calibre-Web-Automated's user ID and group ID:

environment:

PUID=1000
PGID=1000

Calibre-Web-Automated's PUID and PGID should set to the same user ID and Group ID as your account in your Synology device (or whatever account you want Calibre-Web-Automated to run as).

To find your IDs: https://www.google.com/search?q=how+to+find+my+synology+user+id

https://www.synoforum.com/resources/how-to-find-uid-userid-and-gid-groupid.77/

SSH into your Synology NAS and type "id". That's it.

0x09AF commented 1 day ago

That's a good one, thanks for advise. This is from Synology: $ id -u; id -g

1000
100

I updated my docker-compose file:

. . .
    environment:
      - PUID=1000
      - PGID=100
. . .

pretty much the same log output after the change:

. . . 
calibre-web-automated  | [custom-init] No custom files found, skipping...
calibre-web-automated  | [cwa-set-perms] Recursively setting ownership of everything in /config, /calibre-library, /app/calibre-web-automated to abc:abc...
calibre-web-automated  | [cwa-set-perms] Successfully set permissions for '/config'!
calibre-web-automated  | chown: changing ownership of '/calibre-library/metadata.db': Operation not permitted
calibre-web-automated  | [cwa-set-perms] Service could not successfully set permissions for '/calibre-library' (see errors above).
calibre-web-automated  | chown: changing ownership of '/calibre-library': Operation not permitted
calibre-web-automated  | [cwa-set-perms] Successfully set permissions for '/app/calibre-web-automated'!
. . .

Just a side note, all the other LSIO-based, and even others non-LSIO docker containers in the same LXC are set to PUID and PGID 1000:1000 and they work fine. I just haven't noticed they ever try to chown and hence never caused issues with my setup...

iroQuai commented 1 day ago

It's best practice on Synology to create a seperate group and user for docker with only access to the services it actually needs. This is a great writeup on how to do that:

https://drfrankenstein.co.uk/step-2-setting-up-a-restricted-docker-user-and-obtaining-ids/

Would you maybe try that and report back?

0x09AF commented 20 hours ago

Thanks for the guide. I think having that "restricted" user made it even worse inside the container... Set UID to 1031 and GID to 65536 to match the Synology new user. The logs:

calibre-web-automated  | ───────────────────────────────────────
calibre-web-automated  | GID/UID
calibre-web-automated  | ───────────────────────────────────────
calibre-web-automated  | 
calibre-web-automated  | User UID:    1031
calibre-web-automated  | User GID:    65536
calibre-web-automated  | ───────────────────────────────────────
calibre-web-automated  | 
calibre-web-automated  | chown: changing ownership of '/app': Invalid argument
calibre-web-automated  | **** Permissions could not be set. This is probably because your volume mounts are remote or read-only. ****
calibre-web-automated  | **** The app may not work properly and we will not provide support for it. ****
calibre-web-automated  | chown: changing ownership of '/config': Invalid argument
calibre-web-automated  | **** Permissions could not be set. This is probably because your volume mounts are remote or read-only. ****
calibre-web-automated  | **** The app may not work properly and we will not provide support for it. ****
calibre-web-automated  | chown: changing ownership of '/defaults': Invalid argument
calibre-web-automated  | **** Permissions could not be set. This is probably because your volume mounts are remote or read-only. ****
calibre-web-automated  | **** The app may not work properly and we will not provide support for it. ****
calibre-web-automated  | chown: changing ownership of '/config': Invalid argument
calibre-web-automated  | chown: changing ownership of '/config/client_secrets.json': Invalid argument
calibre-web-automated  | chown: changing ownership of '/config/app.db': Invalid argument
calibre-web-automated  | chown: changing ownership of '/config/.CALIBRE_RELEASE': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web/cps/cache': Invalid argument
calibre-web-automated  | **** Permissions could not be set. This is probably because your volume mounts are remote or read-only. ****
calibre-web-automated  | **** The app may not work properly and we will not provide support for it. ****
. . .
calibre-web-automated  | **** Calibre already installed, skipping ****
calibre-web-automated  | [custom-init] No custom files found, skipping...
calibre-web-automated  | [cwa-set-perms] Recursively setting ownership of everything in /config, /calibre-library, /app/calibre-web-automated to abc:abc...
calibre-web-automated  | chown: changing ownership of '/config/client_secrets.json': Invalid argument
calibre-web-automated  | chown: changing ownership of '/config/app.db': Invalid argument
calibre-web-automated  | chown: changing ownership of '/config/.CALIBRE_RELEASE': Invalid argument
calibre-web-automated  | chown: changing ownership of '/config': Invalid argument
calibre-web-automated  | [cwa-set-perms] Service could not successfully set permissions for '/config' (see errors above).
calibre-web-automated  | chown: changing ownership of '/calibre-library/metadata.db': Invalid argument
calibre-web-automated  | chown: changing ownership of '/calibre-library': Invalid argument
calibre-web-automated  | [cwa-set-perms] Service could not successfully set permissions for '/calibre-library' (see errors above).
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/metadata_temp': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/requirements.txt': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/empty_library/app.db': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/empty_library/metadata.db': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/empty_library': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/metadata_change_logs': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/dirs.json': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/.github/FUNDING.yml': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/.github': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/scripts/cwa_db.py': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/scripts/check-cwa-install.sh': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/scripts/auto-library.py': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/scripts/convert-library.py': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/scripts/cover-enforcer.py': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/scripts/setup-cwa.sh': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/scripts/new-book-processor.py': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/scripts': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated/cwa-import': Invalid argument
calibre-web-automated  | chown: changing ownership of '/app/calibre-web-automated': Invalid argument
calibre-web-automated  | [cwa-set-perms] Service could not successfully set permissions for '/app/calibre-web-automated' (see errors above).
calibre-web-automated  | [cwa-set-perms] Ending service...
calibre-web-automated  | s6-applyuidgid: fatal: unable to set supplementary group list: Invalid argument
calibre-web-automated  | [auto-library]: Existing library found at /calibre-library, mounting now...
calibre-web-automated  | [auto-library] Updating dirs.json with new library location...
calibre-web-automated  | [auto-library]: Updating Settings Database with library location...
calibre-web-automated  | [auto-library] Library location successfully set to: /calibre-library
calibre-web-automated  | ========== STARTING METADATA CHANGE DETECTOR ===========
calibre-web-automated  | [metadata-change-detector]: Watching folder: /app/calibre-web-automated/metadata_change_logs
calibre-web-automated  | ========== STARTING BOOKS-TO-PROCESS DETECTOR ==========
calibre-web-automated  | ============== STARTING NEW BOOK DETECTOR ==============
calibre-web-automated  | s6-applyuidgid: fatal: unable to set supplementary group list: Invalid argument
calibre-web-automated  | [books-to-process]: Watching folder: /cwa-book-ingest
calibre-web-automated  | [new-book-detector]: Watching folder: /app/calibre-web-automated/cwa-import
calibre-web-automated  | s6-applyuidgid: fatal: unable to set supplementary group list: Invalid argument
calibre-web-automated  | s6-applyuidgid: fatal: unable to set supplementary group list: Invalid argument
calibre-web-automated  | s6-applyuidgid: fatal: unable to set supplementary group list: Invalid argument
. . .

crocodilestick / Calibre-Web-Automated

[Fixed in next version/ Release Candidate] File permission issues on NAS systems #30