Fix documentation on creating virtual environments

[baelim]

This has been working like clockwork for 20 downloads at at time. Repeated attempts always hit 20.

I tried running the container both in my mounted disk and local (where symlink is supported) but it always fails after 20 downloads. I checked the console and quota limits on the console is default to 10k for the library API and I can see the dashboard spike when creating the initial index.

What am I missing? The only work around I can think of is a cron job to run every 5 minutes or use docker-compose with restart. The whole thing feels like a new Google Policy as I'm just joining this club.

07-10 03:35:20 gphotos_sync.Main WARNING gphotos-sync 0.1.dev1+g2f6bcbb 2022-07-10 03:35:20.758595 07-10 03:35:20 gphotos_sync.Utils DEBUG MINIMUM_DATE = 1981-01-01 00:00:00 07-10 03:35:21 gphotos_sync.Checks INFO Target filesystem /storage is fuseblk 07-10 03:35:21 gphotos_sync.Checks DEBUG Checking if is filesystem supports symbolic links... 07-10 03:35:21 gphotos_sync.Checks DEBUG attempting to symlink /storage/test_src_3804034856 to /storage/test_dst_903085907 07-10 03:35:21 gphotos_sync.Checks ERROR Symbolic links not supported 07-10 03:35:21 gphotos_sync.Checks ERROR Albums are not going to be synced - requires symlinks 07-10 03:35:21 gphotos_sync.Checks DEBUG Checking if File system supports unicode filenames... 07-10 03:35:21 gphotos_sync.Checks INFO Filesystem supports Unicode filenames 07-10 03:35:21 gphotos_sync.Checks DEBUG Checking if File system is case insensitive... 07-10 03:35:21 gphotos_sync.Checks INFO Case insensitive file system found 07-10 03:35:21 gphotos_sync.Checks INFO Max Path Length: 4096 07-10 03:35:21 gphotos_sync.Checks INFO Max filename length: 255 07-10 03:35:21 gphotos_sync.Main INFO version: 0.1.dev1+g2f6bcbb, database schema version 5.7 07-10 03:35:22 gphotos_sync.BadIds DEBUG no bad_ids file, bad ids list is empty 07-10 03:35:22 gphotos_sync.GooglePhotosIndex WARNING Indexing Google Photos Files ... 07-10 03:35:22 gphotos_sync.GooglePhotosIndex INFO searching for media start=2022-07-09 21:33:55, end=None, videos=True 07-10 03:35:22 gphotos_sync.GooglePhotosIndex DEBUG mediaItems.search with body: {'pageToken': None, 'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate': {'year': 2022, 'month': 7, 'day': 9}, 'endDate': {'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']}, 'includeArchivedMedia': False}} 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 1 photos/2022/07/img_2341.mov 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 2 photos/2022/07/20220709_171727.mp4 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 3 photos/2022/07/20220709_171553.mp4 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 4 photos/2022/07/screenshot_20220709-152412_kakaotalk.jpg 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 5 photos/2022/07/20220709_131224.mp4 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 6 photos/2022/07/20220709_130819.jpg 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 7 photos/2022/07/img_2340.heic 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 8 photos/2022/07/img_2339.heic 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG Skipped Index (already indexed) 9 photos/2022/07/img_2338.mov 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG search_media parsed 9 media_items with 100 PAGE_SIZE 07-10 03:35:23 gphotos_sync.GooglePhotosIndex DEBUG mediaItems.search with body: {'pageToken': 'CmwKRHR5cGUuZ29vZ2xlYXBpcy5jb20vZ29vZ2xlLnBob3Rvcy5saWJyYXJ5LnYxLlNlYXJjaE1lZGlhSXRlbXNSZXF1ZXN0EiQQZCIgChQSEgoHCOYPEAcYCRIHCLgXEAEYARoDCgEAOgMKAQASjAFBSF91UTQwUUlhZ0tpWEc2MkgtRWJrX2NObnFQUElfZGJuWGNjelFVR2FQQXJfMGk4UkxNWEpjaW5CV1BkMWxIel9VX1huUjktcE1ORnptVlRtMG1kZ1BQTWJSQ3lWOUQ5ZjZWaTA3VXdoUGVHTGE3anJTLUh6TjQxZ0F3M1FxWlItVHpkNTRGX1hkVhobLUNxTWZSWWVzMGIwdXFWbTRvZ0ZQRnBzSEVJ', 'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate': {'year': 2022, 'month': 7, 'day': 9}, 'endDate': {'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']}, 'includeArchivedMedia': False}} 07-10 03:35:23 gphotos_sync.GooglePhotosIndex WARNING indexed 0 items 07-10 03:35:26 gphotos_sync.GooglePhotosDownload WARNING Downloading Photos ... 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 61 photos/2022/06/img_2302.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 62 photos/2022/06/img_2301.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 63 photos/2022/06/20220629_165043.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 64 photos/2022/06/20220629_165042.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 65 photos/2022/06/20220629_165041.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 66 photos/2022/06/20220629_165040.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 67 photos/2022/06/20220629_165037.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 68 photos/2022/06/20220629_165036.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 69 photos/2022/06/20220629_165035.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 70 photos/2022/06/20220629_165033.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 71 photos/2022/06/img_0879.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 72 photos/2022/06/img_0878.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 73 photos/2022/06/img_0877.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 74 photos/2022/06/img_0876.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 75 photos/2022/06/img_0875.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 76 photos/2022/06/img_0874.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 77 photos/2022/06/img_0873.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 78 photos/2022/06/20220629_073556.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 79 photos/2022/06/20220629_073552.jpg 07-10 03:35:27 gphotos_sync.GooglePhotosDownload DEBUG SKIPPED download (file exists) 80 photos/2022/06/20220629_073550.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 1 photos/2022/06/20220629_073549.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 2 photos/2022/06/20220629_073547.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 3 photos/2022/06/20220629_073548.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 4 photos/2022/06/img_0872.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 5 photos/2022/06/img_0871.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 6 photos/2022/06/img_0870.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 7 photos/2022/06/img_2300.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 8 photos/2022/06/img_2299.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 9 photos/2022/06/img_2298.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 10 photos/2022/06/img_2297.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 11 photos/2022/06/img_0868.jpg 07-10 03:35:28 gphotos_sync.GooglePhotosDownload INFO downloading 12 photos/2022/06/img_0867.jpg 07-10 03:35:29 gphotos_sync.GooglePhotosDownload INFO downloading 13 photos/2022/06/img_0866.jpg 07-10 03:35:29 gphotos_sync.GooglePhotosDownload INFO downloading 14 photos/2022/06/img_2296.mov 07-10 03:35:29 gphotos_sync.GooglePhotosDownload INFO downloading 15 photos/2022/06/img_2295.heic 07-10 03:35:29 gphotos_sync.GooglePhotosDownload INFO downloading 16 photos/2022/06/img_2294.heic 07-10 03:35:29 gphotos_sync.GooglePhotosDownload INFO downloading 17 photos/2022/06/img_2293.heic 07-10 03:35:29 gphotos_sync.GooglePhotosDownload INFO downloading 18 photos/2022/06/img_2292.heic 07-10 03:35:29 gphotos_sync.GooglePhotosDownload INFO downloading 19 photos/2022/06/20220626_170435.jpg 07-10 03:35:29 gphotos_sync.GooglePhotosDownload INFO downloading 20 photos/2022/06/20220626_170434.jpg 07-10 03:35:31 gphotos_sync.GooglePhotosDownload ERROR FAILURE 1 downloading photos/2022/06/20220629_073549.jpg - [Errno 1] Operation not permitted 07-10 03:35:31 gphotos_sync.GooglePhotosDownload ERROR FAILURE 2 downloading photos/2022/06/20220629_073549.jpg - [Errno 1] Operation not permitted 07-10 03:35:31 gphotos_sync.LocalData INFO Saving Database ... 07-10 03:35:33 gphotos_sync.LocalData INFO Database Saved. 07-10 03:35:33 gphotos_sync.Main ERROR Process failed. Traceback (most recent call last): File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 493, in main self.start(args) File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 433, in start self.do_sync(args) File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 401, in do_sync files_downloaded = self.google_photos_down.download_photo_media() File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 178, in download_photo_media self.do_download_complete(futures_left) File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 347, in do_download_complete raise e File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 174, in download_photo_media self.download_batch(batch) File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 215, in download_batch self.download_file(media_item, media_item_json) File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 246, in download_file self.do_download_complete(done_list) File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 347, in do_download_complete raise e File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GooglePhotosDownload.py", line 285, in do_download_file os.utime( PermissionError: [Errno 1] Operation not permitted 07-10 03:35:33 gphotos_sync.Main WARNING Done.

[gilesknap]

Hi @baelim. I think your issue is that there is a problem committing to the database. The download process will update the DB every 20 files or so as far as I recall.

I'm going to guess that this is something to do with the filesystem you are using. Many network filesystems are not compatible with sqlite3. The workaround for this is to store the database in a local filesystem, maybe your home directory? Use the --dbpath option to specify.

Hope this helps.

[baelim]

Thank you for the quick response. I've been trying different permutations of the container and doing some back reading on file formats to try to understand the underlying issue. Your assessment seems correct but the commit issues is prevalent also when running in the local drive.

And to be clear what I mean by local it's $HOME/Desktop/Photos.

So in terms of permutations:

• Ran the container as usual and got 20 downloads
• Ran the container as local only and got 20 downloads
• Ran the local install using the --db-path variable and got no downloads (output below)

So now I'm wondering if file format is at fault.
Local mSD is on ext4 (pi3) and the external drive is on exFAT or NTFS mounted by fuseblk. Tried mounting another external drive but I'm not having much luck getting it not use fuseblk to rule out the latter.

07-11 00:03:01 gphotos_sync.Main WARNING gphotos-sync 3.0.3 2022-07-11 00:03:01.323599 07-11 00:03:01 gphotos_sync.Utils DEBUG MINIMUM_DATE = 1981-01-01 00:00:00 07-11 00:03:01 gphotos_sync.Checks INFO Target filesystem /mnt/usb1/photos is fuseblk 07-11 00:03:01 gphotos_sync.Checks DEBUG Checking if is filesystem supports symbolic links... 07-11 00:03:01 gphotos_sync.Checks DEBUG attempting to symlink /mnt/usb1/photos/test_src_3888756174 to /mnt/usb1/photos/test_dst_2183533860 07-11 00:03:01 gphotos_sync.Checks ERROR Symbolic links not supported 07-11 00:03:01 gphotos_sync.Checks ERROR Albums are not going to be synced - requires symlinks 07-11 00:03:01 gphotos_sync.Checks DEBUG Checking if File system supports unicode filenames... 07-11 00:03:01 gphotos_sync.Checks INFO Filesystem does not support Unicode filenames 07-11 00:03:01 gphotos_sync.Checks DEBUG Checking if File system is case insensitive... 07-11 00:03:01 gphotos_sync.Checks INFO Case insensitive file system found 07-11 00:03:01 gphotos_sync.Checks INFO Max Path Length: 4096 07-11 00:03:01 gphotos_sync.Checks INFO Max filename length: 255 07-11 00:03:02 gphotos_sync.Main INFO version: 3.0.3, database schema version 5.7 07-11 00:03:02 gphotos_sync.Main ERROR Process failed. Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/gphotos_sync/Main.py", line 496, in main self.setup(args, db_path) File "/usr/local/lib/python3.7/dist-packages/gphotos_sync/Main.py", line 345, in setup self.auth.authorize() File "/usr/local/lib/python3.7/dist-packages/gphotos_sync/authorize.py", line 118, in authorize respect_retry_after_header=True, TypeError: __init__() got an unexpected keyword argument 'allowed_methods' 07-11 00:03:02 gphotos_sync.Main WARNING Done.

[gilesknap]

The error you are getting when using --dbpath is related to out if date dependencies I believe. There is another open issue discussing this. The best way to fix it is to create a virtual env for gphotos-sync. I'll be pushing a fix for this soonish.

[gilesknap]

See https://github.com/gilesknap/gphotos-sync/issues/365 for fixes to got an unexpected keyword argument 'allowed_methods'

[baelim]

Added the suggested fix from #365 and got some odd outputs.
Still crashed at 20 downloads but some of those downloads looked like hash values.

Went back and created a fresh image with the latest raspberry os and it's working like it should.

[gilesknap]

Thanks for reporting. I need to change the docs to create a venv to avoid these issues and also pin some dependencies in setup.cfg

[baelim]

Just a few notes when trying to create my virtual environment

As a newbie on venv I kept creating the folder in root i.e.: python3 -m venv /gphotos-sync Then it would fail, complete it on sudo and then fail on me at the cron job due to permission issues

pip install of gphotos-sync failed too naturally. ran sudo and it installed in my actual environment ha!

I'd recommend documentation to include full path i.e.: python3 -m venv $HOME/gphotos-sync

Also recommend to confirm gphotos install in environment using pip list as well as gphotos-sync --version

Finally the should match the local install instructions i.e.: $HOME/gphotos-sync/bin/...

Thank you for all the hard work!

Martin

On Mon, Jul 18, 2022 at 4:47 PM Giles Knap @.***> wrote:

Thanks for reporting. I need to change the docs to create a venv to avoid these issues and also pin some dependencies in setup.cfg

— Reply to this email directly, view it on GitHub https://github.com/gilesknap/gphotos-sync/issues/374#issuecomment-1188293570, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKA2VMFETS64D76H7L5ZX3VUW7EPANCNFSM53EK4KSQ . You are receiving this because you modified the open/close state.Message ID: @.***>

-- Martin

[gilesknap]

Thanks. I'll make changes as you suggest.

reopening and changing the title.

[gilesknap]

All dependencies are now pinned moving docs issues to #417