search

gilesknap / gphotos-sync

Google Photos and Albums backup with Google Photos Library API
Apache License 2.0
1.97k stars 161 forks source link

don't download photos of shared albums #432

Closed rui-nar closed 1 year ago

rui-nar commented 1 year ago

Hi Is it possible not to download the photos of shared albums when using the --skip-shared-albums option?

My usecase is the following: I have an automatic backup of the photos taken with my smartphone in original quality with an app into my NAS. Since I do my sorting & deleting from Google Photos, I'm looking to "synchronise" my local copy with the "truth" which is my Google Photos referential (I'm looking into using the --compare-folder option) but first I need a proper mirror of Google Photos into my NAS of ONLY my photos and my albums .. not shared albums or other people's photos ...

I hope I was clear Rui

gilesknap commented 1 year ago

As far as I recall, --skip-shared-albums works as you require. I don't use it on my own library but I do have a system test that verifies that it works (at least on the test account).

Using --compare-folder successfully may be pretty hard. It is probably over complicated by my attempts to cope with cameras from the 90s that frequently used the same filenames and often trashed any metadata. That's why I deprecated it.

Best of luck!

rui-nar commented 1 year ago

From my runs the skip-shared-albums is downloading all photos, not just mine hence me writing to you 😜 Would you mind checking it for me on your end ?

As for the compare option, I'm only going back a few years so it shouldn't be a problem.

I'm loving this project btw 👍🏼👍🏼

On Wed, May 10, 2023, 22:02 Giles Knap @.***> wrote:

As far as I recall, --skip-shared-albums works as you require. I don't use it on my own library but I do have a system test that verifies that it works (at least on the test account).

Using --compare-folder successfully may be pretty hard. It is probably over complicated by my attempts to cope with cameras from the 90s that frequently used the same filenames and often trashed any metadata. That's why I deprecated it.

Best of luck!

— Reply to this email directly, view it on GitHub https://github.com/gilesknap/gphotos-sync/issues/432#issuecomment-1542735040, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACP6EX24SWPCMJKKDNWA7O3XFPX3XANCNFSM6AAAAAAX45QDGM . You are receiving this because you authored the thread.Message ID: @.***>

gilesknap commented 1 year ago

Sure I'll verify against my account (which has many shared albums). I'll get back to you this weekend hopefully.

rui-nar commented 1 year ago

still interested on the results of your tests btw ;)

gilesknap commented 1 year ago

Sorry. Will be able to look at this today.

gilesknap commented 1 year ago

Logging of tests

using 2023 photos only. According to Google Photos Web I have:

Test 1 - with no shared


(venv) [giles@ws1 gphoto-test]$ gphotos-sync no-shared/ --start-date 2023-01-01 --skip-shared-albums --progress
05-16 14:07:29 WARNING  gphotos-sync 3.1.3.dev0+ga631449.d20230516 2023-05-16 14:07:29.504828 
 5-16 14:07:29 WARNING  Indexing Google Photos Files ... 
05-16 14:07:58 WARNING  indexed 1577 items ..
 5-16 14:07:58 WARNING  Downloading Photos ... 
05-16 14:15:46 WARNING  Downloaded 1940 Items, Failed 0, Already Downloaded 0 
 5-16 14:15:46 WARNING  Indexing Albums ... 
05-16 14:35:04 WARNING  Indexed 468 Albums ..
05-16 14:35:04 WARNING  Downloading Photos ... 
05-16 14:35:05 WARNING  Downloaded 1943 Items, Failed 0, Already Downloaded 1940 
05-16 14:35:05 WARNING  Creating album folder links to media ... 
05-16 14:35:23 WARNING  Created 893 new album folder links 
05-16 14:35:23 WARNING  Done.

Initial number of images is 1940 as expected TODO: 3 extra files came down with albums - investigate

Albums created:

(venv) [giles@ws1 no-shared]$ ls albums/* | wc
    515    1991   12248
(venv) [giles@ws1 no-shared]$ ls albums/1999/0803\ Trip\ to\ San\ Francisco\ and\ Las\ Vegas/
(venv) [giles@ws1 no-shared]$ 

(venv) [giles@ws1 no-shared]$ ls albums/2023/0215\ Latest\ works\ by\ Noah/
'0141_Scanned Document-3.jpg'   0150_swim.jpg                                                 '0159_Scanned Document-1.jpg'
'0142_Scanned Document-7.jpg'   0151_awards-2.jpg                                             '0160_Scanned Document-2.jpg'
'0143_Scanned Document-5.jpg'   0152_awards-1.jpg                                              0161_IMG_20230202_0001.jpg
'0144_Scanned Document-9.jpg'   0153_1673704825830-42118ba4-ef47-4bd9-9f5c-be186620fc29_.jpg   0162_PXL_20230207_222258401.jpg
'0145_Scanned Document-8.jpg'   0154_1673704893896-aa4f079a-eb5c-4ba9-8f99-62ba9357e409.jpg    0163_new-2.jpg
'0146_Scanned Document-4.jpg'   0155_1673802139435-ed51621e-6a77-4797-a49c-78c01459e36d_.jpg   0164_new-1.jpg
'0147_Scanned Document-1.jpg'   0156_IMG_6789.HEIC                                             0165_new2.jpg
'0148_Scanned Document-6.jpg'   0157_PXL_20230127_170708525.jpg                                0166_PXL_20230215_223643463.jpg
'0149_Scanned Document-2.jpg'   0158_Beshianlong.jpg                                           0167_PXL_20230215_223807405.jpg
(venv) [giles@ws1 no-shared]$

Hmm - a links folder was created for every album in my library - but only files dated past 2023 are linked (so that is a lot of empty folders)

Albums in 2023 is 7 as expected

(venv) [giles@ws1 no-shared]$ ls albums/2023/0
0115 Birdworld - meeting Emus/             0326 Hockney/                              0515 Millie!/
0215 Latest works by Noah/                 0416 Dinosnores at Natural History Museum/ 
0218 Jurassic Coast (West Lulworth)/       0418 Ivy's 11th Birthday/
rui-nar commented 1 year ago

Interesting and thanks for your test. So if I understand correctly, for you it worked (despite the 3 extra files).

Side question, how do you explain the difference between the 1577 indexed items and 1940 downloaded items ?

On Tue, May 16, 2023, 15:45 Giles Knap @.***> wrote:

Logging of tests

using 2023 photos only. According to Google Photos Web I have:

  • 1940 photos
  • 7 albums (one of which is auto)
  • 2 named shared albums
  • 1 shared album contains 145 photos not mine
  • 1 shared album contains 15 mine, 1 not mine
  • there are other unamed shared folders (which should be ignored - TODO verify this)

Test 1 - with no shared

(venv) @.*** gphoto-test]$ gphotos-sync no-shared/ --start-date 2023-01-01 --skip-shared-albums --progress 05-16 14:07:29 WARNING gphotos-sync 3.1.3.dev0+ga631449.d20230516 2023-05-16 14:07:29.504828 5-16 14:07:29 WARNING Indexing Google Photos Files ... 05-16 14:07:58 WARNING indexed 1577 items .. 5-16 14:07:58 WARNING Downloading Photos ... 05-16 14:15:46 WARNING Downloaded 1940 Items, Failed 0, Already Downloaded 0 5-16 14:15:46 WARNING Indexing Albums ... 05-16 14:35:04 WARNING Indexed 468 Albums .. 05-16 14:35:04 WARNING Downloading Photos ... 05-16 14:35:05 WARNING Downloaded 1943 Items, Failed 0, Already Downloaded 1940 05-16 14:35:05 WARNING Creating album folder links to media ... 05-16 14:35:23 WARNING Created 893 new album folder links 05-16 14:35:23 WARNING Done.

TODO: 3 extra files came down with albums - investigate

Albums created:

(venv) @. no-shared]$ ls albums/ | wc 515 1991 12248 (venv) **@. no-shared]$ ls albums/1999/0803\ Trip\ to\ San\ Francisco\ and\ Las\ Vegas/ (venv) @. no-shared]$

(venv) @. no-shared]$ ls albums/2023/0215\ Latest\ works\ by\ Noah/'0141_Scanned Document-3.jpg' 0150_swim.jpg '0159_Scanned Document-1.jpg''0142_Scanned Document-7.jpg' 0151_awards-2.jpg '0160_Scanned Document-2.jpg''0143_Scanned Document-5.jpg' 0152_awards-1.jpg 0161_IMG_20230202_0001.jpg'0144_Scanned Document-9.jpg' 01531673704825830-42118ba4-ef47-4bd9-9f5c-be186620fc29.jpg 0162_PXL_20230207_222258401.jpg'0145_Scanned Document-8.jpg' 0154_1673704893896-aa4f079a-eb5c-4ba9-8f99-62ba9357e409.jpg 0163_new-2.jpg'0146_Scanned Document-4.jpg' 01551673802139435-ed51621e-6a77-4797-a49c-78c01459e36d.jpg 0164_new-1.jpg'0147_Scanned Document-1.jpg' 0156_IMG_6789.HEIC 0165_new2.jpg'0148_Scanned Document-6.jpg' 0157_PXL_20230127_170708525.jpg 0166_PXL_20230215_223643463.jpg'0149_Scanned Document-2.jpg' 0158_Beshianlong.jpg 0167_PXL_20230215_223807405.jpg (venv) @. no-shared]$

Hmm - a links folder was created for every album in my library - but only files dated past 2023 are linked (so that is a lot of empty folders)

— Reply to this email directly, view it on GitHub https://github.com/gilesknap/gphotos-sync/issues/432#issuecomment-1549706260, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACP6EXZ6PVM2TPSEBOZSJGTXGOAIVANCNFSM6AAAAAAX45QDGM . You are receiving this because you authored the thread.Message ID: @.***>

gilesknap commented 1 year ago

Test 2 with Shared Albums


(venv) [giles@ws1 gphoto-test]$ gphotos-sync with-shared/ --start-date 2023-01-01  --progress
05-16 14:25:57 WARNING  gphotos-sync 3.1.3.dev0+ga631449.d20230516 2023-05-16 14:25:57.530072 
 5-16 14:25:57 WARNING  Indexing Google Photos Files ... 
05-16 14:26:31 WARNING  indexed 1940 items ..
 5-16 14:26:31 WARNING  Downloading Photos ... 
05-16 14:31:57 WARNING  Downloaded 1940 Items, Failed 0, Already Downloaded 0 
 5-16 14:31:57 WARNING  Indexing Shared (titled) Albums ... 
05-16 14:35:28 WARNING  Indexed 485 Shared (titled) Albums ..
 5-16 14:35:28 WARNING  Indexing Albums ... 
05-16 14:51:19 WARNING  Indexed 468 Albums ..
 5-16 14:51:19 WARNING  Downloading Photos ... 
05-16 14:52:13 WARNING  Downloaded 2130 Items, Failed 0, Already Downloaded 1940 
05-16 14:52:13 WARNING  Creating album folder links to media ... 
05-16 14:52:32 WARNING  Created 1115 new album folder links 
05-16 14:52:32 WARNING  Done.

TODO: back later to analyse these...

gilesknap commented 1 year ago

@rui-nar I don't have any immediate answers. But I now have two folders of files to do a detailed investigation with. I have to run now but will get back to this later today.

gilesknap commented 1 year ago

OK - I do have an answer for the 1577 vs 1940. This was just because I ran and aborted gphotos-sync in that folder before the test and it had already started indexing when I aborted (I wanted to restart with --progress). You can see the correct numbers in Test 2.

back in a couple of hours.

gilesknap commented 1 year ago

with-shared downloaded an extra 2130-1940 = 190 files whereas I thought I had 146 shared files.

listing the files in the extra album folders that appear in albums/2023 gives 227 extra files.

Neither of these figures add up.

rui-nar commented 1 year ago

Development is hard 😂

I'll try to run some tests tonight like the ones you did to see if I share your view. Are you using a particular version or just the latest status of the repository?

On Tue, May 16, 2023, 17:02 Giles Knap @.***> wrote:

with-shared downloaded an extra 2130-1940 = 190 files whereas I thought I had 146 shared files.

listing the files in the extra album folders that appear in albums/2023 gives 227 extra files.

Neither of these figures add up.

— Reply to this email directly, view it on GitHub https://github.com/gilesknap/gphotos-sync/issues/432#issuecomment-1549849426, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACP6EX7PTLZTWRPUMS2OQL3XGOJGRANCNFSM6AAAAAAX45QDGM . You are receiving this because you were mentioned.Message ID: @.***>

gilesknap commented 1 year ago

Albums that appear in the shared sync:

(venv) [giles@ws1 with-shared]$ ls -l albums/2023
total 88
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0115 Birdworld - meeting Emus'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0215 Latest works by Noah'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0218 Charmouth, Corfe Castle & Lyme Regis'
drwxr-xr-x 2 giles dialout 20480 May 16 15:28 '0218 Jurassic Coast (West Lulworth)'
drwxr-xr-x 2 giles dialout 12288 May 16 15:28 '0219 James 11th Birthday @ GoApe'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0326 Hockney'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0326 Hockney in Kings X'
drwxr-xr-x 2 giles dialout 12288 May 16 15:28 '0416 Dinosnores at Natural History Museum'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 "0418 Ivy's 11th Birthday"
drwxr-xr-x 2 giles dialout 20480 May 16 15:28 '0515 Millie!'

Albums that appear in non-shared sync

drwxr-xr-x 2 giles dialout  4096 May 16 14:35 '0115 Birdworld - meeting Emus'
drwxr-xr-x 2 giles dialout  4096 May 16 14:35 '0215 Latest works by Noah'
drwxr-xr-x 2 giles dialout 16384 May 16 14:35 '0218 Jurassic Coast (West Lulworth)'
drwxr-xr-x 2 giles dialout  4096 May 16 14:35 '0326 Hockney'
drwxr-xr-x 2 giles dialout 12288 May 16 14:35 '0416 Dinosnores at Natural History Museum'
drwxr-xr-x 2 giles dialout  4096 May 16 14:35 "0418 Ivy's 11th Birthday"
drwxr-xr-x 2 giles dialout 20480 May 16 14:35 '0515 Millie!'

EXTRA ALBUMS

drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0218 Charmouth, Corfe Castle & Lyme Regis'
drwxr-xr-x 2 giles dialout 12288 May 16 15:28 '0219 James 11th Birthday @ GoApe'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0326 Hockney in Kings X'

The first above is an auto generated album (whch apparently counts as a shared one even though its only in my library) so it contains links to pre-existing photos and can be ignored.

Second is fully shared with 145 images Third is fully shared with 42 images

This adds up to 187 images extra which is 3 less than reported in the log (so the same 3 that the non-shared sync did probably).

Now checking the difference between actual photos downloaded in each case:


(venv) [giles@ws1 no-shared]$ find photos/ -type f | wc
   1943    1960   69317
(venv) [giles@ws1 no-shared]$ cd ../with-shared/
(venv) [giles@ws1 with-shared]$ find photos/ -type f | wc
   2130    2189   74915
(venv) [giles@ws1 with-shared]$

2130 - 1943 = 187 which is exactly right.

EDIT: forgot to say that this tallies with the photo count in these 2 albums in the Web GUI

gilesknap commented 1 year ago

Final Synopsis

For my library the 2023 shared/no-shared albums do behave as expected with 187 less photos when using --skip-shared-albums.

Things to note:

gilesknap commented 1 year ago

@rui-nar These tests were run against the main branch.

Let me know your results.

rui-nar commented 1 year ago

hey @gilesknap I have to apologise Giles, I ran all the tests you did and I have the SAME results then you. So I re-ran all my previous commands (which I store in a text file) and all results were in line with expectations. I cannot explain this !!! I know I upgraded the version to the latest at some point in time but I was sure that I had re-tested everything.

So sorry for having wasted your time.

Btw, the --compare-folder worked flawlessly even if I took some time to understand that the differences I was seeing were archived pictures in GPhotos which I can't seem to filter out in the Google Photos search.

Anyways. Awesome project !! Almost able to fulfill my wife's usecase ;)

gilesknap commented 1 year ago

Well it is possible that you simply got inconsistent behaviour from the Google API. I was reasonably convinced that this might have been broken so its good to have verified it properly.

Regarding the archived photos - there is a gphotos-sync option that is supposed to include them --archived. Otherwise they are not supposed to be included. Is that not your experience (this is probably for a different issue if you want to open one)

rui-nar commented 1 year ago

Wrt to the archived photos not being filtered I was referring to the Google Photos web interface. Your tool correctly filters out the archived but then to compare to the web interface I have to search for archived and remove that count from the total count. Nothing complicated, just annoying.

As I mentioned, I'm happy with what I'm getting from gphotos-sync.

I may open a separate issue about SSL errors forcing retries but they're sufficiently rare to not be too annoying for the time being.

On Wed, May 17, 2023, 14:13 Giles Knap @.***> wrote:

Well it is possible that you simply got inconsistent behaviour from the Google API. I was reasonably convinced that this might have been broken so its good to have verified it properly.

Regarding the archived photos - there is a gphotos-sync option that is supposed to include them --archived. Otherwise they are not supposed to be included. Is that not your experience (this is probably for a different issue if you want to open one)

— Reply to this email directly, view it on GitHub https://github.com/gilesknap/gphotos-sync/issues/432#issuecomment-1551282604, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACP6EX2CKP7LPVKRFNDAXGDXGS6FNANCNFSM6AAAAAAX45QDGM . You are receiving this because you were mentioned.Message ID: @.***>