Invalid cross-device link during upload

[pthiben]

Hello,

Thanks a lot for your work on calibre-web, it's an amazing piece of software !

I think this bug is related to calibre-web / python shutils, but as a disclaimer I'm running with a docker version in WSL. It looks awfully similar to https://github.com/janeczku/calibre-web/issues/2062

Calibre Web     0.6.23 - d3233b4b9cf4ec540ff31fba787865cb95810ae3 - 2024-08-05T18:42:13+02:00
Python  3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0]
Platform    Linux 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64

My /tmp and /books are mounted on different drives.

overlay        1055762868  26174148 975885248   3% /
tmpfs               65536         0     65536   0% /dev
shm                 65536         0     65536   0% /dev/shm
E:              976760828 602444084 374316744  62% /books

I tried changing the TMPDIR environment variable to point to /books/tmp, but it didn't seem to have propagated to calibre (might be a specific docker issue)

My Repro:

1. Download ebook from https://www.gutenberg.org/ebooks/25344 (I have the same problem with any ebook with cover that I upload)
2. Upload to calibre-web

I'm getting the error

Error: Rename title from: '/tmp/calibre_web/00cdb14c52903caf841ec90322ccd515' to '/books/Nathaniel Hawthorne/The Scarlet Letter (338)' failed with error: [Errno 1] Operation not permitted

The book has actually been moved /books/Nathaniel Hawthorne/The Scarlet Letter (338) But the cover is moved to /books/Nathaniel Hawthorne/The Scarlet Letter /books/Nathaniel Hawthorne/The Scarlet Letter seems to be the path expected by calibre-web because if I move manually the book there, fetch metadata and save, things work just fine. The book's URL ends up being admin/book/338 I'm not sure why the cover is succeeding but the book doesn't. Could we do a copy+delete when we detect that the move failed ?

Here's the log.

[2024-09-12 19:29:11,867] DEBUG {cps.uploader:258} Temporary file: /tmp/calibre_web/00cdb14c52903caf841ec90322ccd515
[2024-09-12 19:29:12,663] ERROR {cps.helper:49} Rename title from /tmp/calibre_web/00cdb14c52903caf841ec90322ccd515 to /books/Nathaniel Hawthorne/The Scarlet Letter (338) failed with error: [Errno 1] Operation not permitted
Traceback (most recent call last):
  File "/usr/lib/python3.10/shutil.py", line 816, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/calibre_web/00cdb14c52903caf841ec90322ccd515' -> '/books/Nathaniel Hawthorne/The Scarlet Letter (338)/The Scarlet Letter.epub'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/calibre-web/cps/helper.py", line 537, in move_files_on_change
    shutil.move(original_filepath, os.path.join(new_path, db_filename))
  File "/usr/lib/python3.10/shutil.py", line 836, in move
    copy_function(src, real_dst)
  File "/usr/lib/python3.10/shutil.py", line 435, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.10/shutil.py", line 374, in copystat
    lookup("utime")(dst, ns=(st.st_atime_ns, st.st_mtime_ns),
PermissionError: [Errno 1] Operation not permitted

[OzzieIsaacs]

In general the code is correct. It uses move to move the file, a rename command would only work on the same drive, this was the case in the past but is now solved. The move command tries to copy the metadata which fails. I can't reproduce this, I think you need some complicated use case like the one in the closed issue or something in conjunction with docker (as in your case and in many similar cases) Please try to update to the newest nightly version. Calibre-Web should then give the move another try without coping the metadata. It's possible that this will still occur on other actions (rename authors, title, upload single file or cover)

[mendhak]

Focusing specifically on tmp, it might also be worth you sharing your docker compose, which ought to include how you're mapping your /tmp. Worth noting that docker can let you mount tmpfs but that's a Linux only feature, and it should work in WSL2, might be worth looking at that thread to see how they're doing it. Apologies if I've misunderstood anything.

[pthiben]

Nightly works, thank you so much !

FYI, docker-compose when I tried with TMPDIR is

  calibre-web:
    image: lscr.io/linuxserver/calibre-web:nightly
    container_name: calibre-web
    environment:
      - PUID=${DOCKER_PUID}
      - PGID=${DOCKER_PGID}
      - TZ=${DOCKER_TZ}
      - TMPDIR=/books/tmp
    volumes:
      - ${DOCKER_CONFIG_PATH}/calibre-web:/config
      - ${MEDIA_PATH}/books/calibre:/books
    ports:
      - 8083:8083
      - 9001:9001
      - 465:465
      - 5299:5299
    privileged: true

https://docs.python.org/3/library/tempfile.html states that python should not use /tmp, but

 The default directory is chosen from a platform-dependent list, but the user of the application can control the directory location by setting the TMPDIR, TEMP or TMP environment variables

Dir is 777, and I tried other directories but nothing worked