Closed JpMaster0 closed 6 months ago
gilesknap
commented
8 months ago Sorry for late reply I missed this.
I think the cause is that writing to a sqlite db may not work on an nfs mount. there are some command line options to allow your to move the db directory to a seperate mount - I can't recall which right now but gphotos-sync --help will tell you.
JpMaster0
commented
7 months ago Had this project laying on ice, now giving it a try again..
I think the cause is that writing to a sqlite db may not work on an nfs mount. there are some command line options to allow your to move the db directory to a seperate mount - I can't recall which right now but
gphotos-sync --helpwill tell you.
I think the command "--dbpath" is specifying the location of the database. It's the one on the end of my command, or am I wrong?
$ sudo docker run --rm -v ~/.config/gphotos-sync:/config -v /nfs:/storage -v ~/db:/db -p 8080:8080 -it ghcr.io/gilesknap/gphotos-sync /storage --progress --db-path /db
any other ideas or ways to find informative logs?
gilesknap
commented
7 months ago Hi @JpMaster0 yes the db-path is specified in your command line. What filesystem is /db ?? If it is your container filesystem then I think that could also be an issue. If so try mounting /db to something local on your host running the container.
To get more logs use --log-level DEBUG and see the log file that should probably appear in /db if I recall.
JpMaster0
commented
7 months ago Thanks for answering Giles!
/db is in the home folder of my host. it should be ext4. it is passed to the container via -v ~/db:/db
$ df -T ./db
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv ext4 11218472 6920408 3706400 66% /the log level option is helping i guess. i noticed that /storage is detected as a tempfs and i'm guessing this is the fault...?
After a while the folder seems to fill up and the process comes to a halt but when i look into the folder /nfs/gphoto there is nothing new. (i've missed the ./gphoto in my commands before but the issue is the same.. so it has nothing to do with the nfs share?!)
I've cut the log a bit to reduce reoccurring lines
$ sudo docker run --rm -v ~/.config/gphotos-sync:/config -v /nfs/gphoto:/storage -v ~/db:/db -p 8080:8080 -it ghcr.io/gilesknap/gphotos-sync /storage --progress --db-path /db --log-level DEBUG
12-04 20:49:46 WARNING gphotos-sync 3.1.3 2023-12-04 20:49:46.416610
12-04 20:49:46 DEBUG MINIMUM_DATE = 1800-01-01 00:00:00
12-04 20:49:46 INFO Target filesystem /storage is tmpfs
12-04 20:49:46 DEBUG Checking if is filesystem supports symbolic links...
12-04 20:49:46 DEBUG attempting to symlink /storage/test_src_434988099 to /storage/test_dst_1258840432
12-04 20:49:46 DEBUG Checking if File system supports unicode filenames...
12-04 20:49:46 INFO Filesystem supports Unicode filenames
12-04 20:49:46 DEBUG Checking if File system is case insensitive...
12-04 20:49:46 INFO Case sensitive file system found
12-04 20:49:46 INFO Max Path Length: 4096
12-04 20:49:46 INFO Max filename length: 255
12-04 20:49:46 INFO version: 3.1.3, database schema version 5.7
12-04 20:49:46 DEBUG no bad_ids file, bad ids list is empty
12-04 20:49:46 WARNING Indexing Google Photos Files ...
12-04 20:49:46 INFO searching for media start=2023-10-20 13:26:21, end=None, videos=True
12-04 20:49:46 DEBUG mediaItems.search with body:
{'pageToken': None, 'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate': {'year': 2023, 'month': 10, 'day': 20}, 'endDate': {'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']}, 'includeArchivedMedia': False}}
12-04 20:49:47 INFO Indexed 1 photos/2023/12/IMG20231202125341.jpg
...
12-04 20:49:48 INFO Indexed 81 photos/2023/10/dji_fly_20231022_162132_325_1697985492562_video.mp4
12-04 20:49:48 DEBUG Skipped Index (already indexed) 1 photos/2023/10/IMG20231020152621.jpg
...
12-04 20:49:48 DEBUG Skipped Index (already indexed) 4 photos/2023/10/IMG20231020112523.jpg
12-04 20:49:48 DEBUG search_media parsed 85 media_items with 100 PAGE_SIZE
12-04 20:49:48 WARNING indexed 81 items
12-04 20:49:48 WARNING Downloading Photos ...
12-04 20:49:49 INFO downloading 1 photos/2023/09/IMG20230901173529.jpg
...
12-04 20:49:49 INFO downloading 20 photos/2023/08/IMG20230819140316.jpg
12-04 20:49:51 DEBUG COMPLETED 1 downloading photos/2023/08/IMG20230819141037.jpg
...
12-04 20:51:14 DEBUG COMPLETED 119 downloading photos/2023/08/dji_fly_20230813_190316_222_1691947175788_photo.jpg
12-04 20:51:14 INFO downloading 139 photos/2023/08/dji_fly_20230813_184232_205_1691945995921_photo.jpg
12-04 20:51:15 ERROR FAILURE 1 downloading photos/2023/08/dji_fly_20230813_185552_220_1691945935992_photo.jpg - [Errno 28] No space left on device
12-04 20:51:26 ERROR FAILURE 2 downloading photos/2023/08/dji_fly_20230816_174052_268_1692201255998_video.mp4 - [Errno 28] No space left on device
12-04 20:51:26 INFO Saving Database ...
12-04 20:51:26 INFO Database Saved.
12-04 20:51:26 ERROR
Process failed.
Traceback (most recent call last):
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 174, in download_photo_media
self.download_batch(batch)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 215, in download_batch
self.download_file(media_item, media_item_json)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 246, in download_file
self.do_download_complete(done_list)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 347, in do_download_complete
raise e
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 279, in do_download_file
shutil.copyfileobj(response.raw, temp_file)
File "/usr/local/lib/python3.10/shutil.py", line 198, in copyfileobj
fdst_write(buf)
File "/usr/local/lib/python3.10/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
OSError: [Errno 28] No space left on device
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 507, in main
self.start(args)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 447, in start
self.do_sync(args)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 415, in do_sync
files_downloaded = self.google_photos_down.download_photo_media()
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 178, in download_photo_media
self.do_download_complete(futures_left)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 347, in do_download_complete
raise e
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 279, in do_download_file
shutil.copyfileobj(response.raw, temp_file)
File "/usr/local/lib/python3.10/shutil.py", line 198, in copyfileobj
fdst_write(buf)
File "/usr/local/lib/python3.10/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
OSError: [Errno 28] No space left on device
12-04 20:51:26 WARNING Done. edit: I've tried to download to a folder also in the home directory and it straight up works. so the tempfs is the fault here..
$ df -T /nfs/gphoto/
Filesystem Type 1K-blocks Used Available Use% Mounted on
192.168.2.143:/mnt/t31B/gphotobackup nfs4 4647862016 128 4647861888 1% /nfs/gphotoOK I might have an answer for you here. The code downloads to a temporary file first and then moves it to your storage folder when done. It looks like the line that tries to write the stream from the server to the temporary file is failing.
Can you also mount /tmp to your host or at least make sure that there is a reasonable sized and writeable temp file system inside your container?
JpMaster0
commented
6 months ago Figured it out eventually but still not sure what really fixed my error. I am using a docker compose setup now.
compose.yaml:
services:
gsync:
image: ghcr.io/gilesknap/gphotos-sync
command: /storage1 --progress --db-path /db
ports:
- 8080:8080
volumes:
- /home/USER/db:/db
- /home/USER/.config/gphotos-sync:/config
- type: volume
source: nfsshare
target: /storage1
volume:
nocopy: true
volumes:
nfsshare:
driver_opts:
type: "nfs"
o: "addr=192.168.2.143,nolock,soft,rw"
device: ":/mnt/t31B/gphotobackup"had to remove a old docker volume to make this work, as my first attempt had bad parameters and docker compose wasn't replacing it with new parameters :)
Also I am using docker compose here, not docker-compose
(first remove old container: docker ps -a, docker rm X
than remove old volume: docker volume ls, docker volume rm X)
now it just works with a docker compose up
gilesknap
commented
6 months ago Thanks for letting me know. It looks like you did not need to do anything with /tmp - but then that would normally be writeable inside the container.
Sorry if this issue is sole caused by my personal storage-situation. If so please tell me :)
I can't get the container to work when writing the downloaded media to a nfs-share. Main error as follows:
[Errno 28] No space left on deviceMaybe you've encountered the same problem as i did and you found a solution or have any ideas how to tackle this.Command to start the show:
output:
debug steps already done: