search

steilerDev / icloud-photos-sync

One-way sync engine for the iCloud Photos Library into the native file system
https://icps.steiler.dev/
GNU General Public License v3.0
348 stars 8 forks source link

Initial Sync stuck at the last ~14.000 assets #355

Open halbtuerke opened 1 year ago

halbtuerke commented 1 year ago

Describe the issue

Since roughly 2 days, the initial sync somehow is stuck on the last ~14.000 assets of ~100.000.

Syncing assets, by keeping 85624 and removing 18 local assets, as well as adding 15259 remote assets...
 ■■                                       6% | Elapsed: 59m4s | 924/15259 assets downloaded

Syncing assets, by keeping 85766 and removing 16 local assets, as well as adding 15117 remote assets...
 ■■■                                      6% | Elapsed: 59m2s | 1057/15117 assets downloaded

Syncing assets, by keeping 86054 and removing 11 local assets, as well as adding 14830 remote assets...
 ■■■■                                     9% | Elapsed: 58m29s | 1375/14830 assets downloaded

... 2 days later

Syncing assets, by keeping 86090 and removing 17 local assets, as well as adding 14828 remote assets...
 ■■                                       5% | Elapsed: 37m30s | 764/14828 assets downloaded

It's always downloading assets for roughly an hour and then after the SYNC_NETWORK error, it's showing roughly the same numbers again.

I have attached the terminal output and the debug log:

Is there any other information I can provide you with?

Thanks

How to reproduce the behavior?

No response

Error Code

no error code

Relevant log output

Logs are attached

Operating system

macOS 13.6

Execution environment

node

icloud-photos-sync version

1.2.0

Checklist

steilerDev commented 1 year ago

The sync failing every hour is expected - Apple's URL tokens expire after that and currently the easy solution is to fail and retry automatically.

The weird thing though is that in each execution run it is downloading a couple of thousand assets, but the number of local assets does not increase when they are loaded in the next run (and only a small numbers of files fails validation and get deleted). How many files do you see in your _All Photos folder / does the disk usage increase?

And from which run are those logs?

halbtuerke commented 1 year ago

How many files do you see in your _All Photos folder / does the disk usage increase?

Command 23:30 7:30 diff
du -k ./_All-Photos 581683716 581714664 +30.948
du -k ./_Shared-Photos 320221000 319965012 -255.988
find ./_All-Photos -type f \| wc -l 57772 57772 0
find ./_Shared-Photos -type f \| wc -l 28538 28538 0

So it seems like there is absolutely no progress being made. 😢

And from which run are those logs?

Those logs were from yesterday 2023-09-28 10:35 - 15:30

steilerDev commented 1 year ago

There are a couple of weird things here:

The only thing I could think of is that we are downloading files that already exist. This should not happen because diffing should notice that the local and remote file are the same. The worst case scenario would be, that we are having file naming conflicts (the UUID I'm using for naming files is not actually a UUID). Because if we are at the point of writing files to disk I'm simply overwriting them.

Let me add some additional logging (once I fixed a current bug) and then we can validate if this is the case.

steilerDev commented 1 year ago

1.3.0-beta.1 should log a warning, in case my assumption is correct. Could you try and run it and report back @halbtuerke

halbtuerke commented 1 year ago

There are ~5700 entries of the Skipping download of... message in the logs.

Here are some examples:

[2023-10-02T07:57:02.537Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AfOyvgkFncw-xHiRSYu7n29m3sqnAf-yidbOU19sKfsoNbTD9FIeHNwO/${f}?o=ApvIB_fNfcgWYjScTntc1sfsc2pxFfP_D6i2wBAu7U0o&v=1&x=3&a=CAogxl53NxRm_H_cVBh6-uEz-xjGKSBR35_dkx2vdp7H-KYSaxC7gYr7rjEYu97l_K4xIgEAUgRm3sqnWgQeHNwOaiX806682rfBo4apszTjNcOS9BPGOvz1fV4wjj0Y5mwuo-YsuuJqciUz59usNOWiKxQXidWo9O0vwoNNPE_hGITLf37cgqmh7E6S53Q9&e=1696236924&fl=&r=ee1fb29d-fbb9-451b-8e9d-32ace7cd8650-1&k=UlvQlG5UPJobZd9K4cEVPQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=rRdq9SAFtvrI90vjknj_i5wyTv0 to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AfOyvgkFncw-xHiRSYu7n29m3sqn.jpeg as file already exists
[2023-10-02T07:57:02.537Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/ATikmtq01iWPgrlBu7F7cZoGF8puAXnR5pMDm4qpFgmK9l4a-KTG5I1F/${f}?o=Attq_hQjW7VbmQoUZJWT_uv0hNTjXofzcEnzyURVBJgb&v=1&x=3&a=CAogJiKdhmSw6vT5_hGi53LRYiS0e-ihbd2EWdIFt1GEX3QSaxCPjYr7rjEYj-rl_K4xIgEAUgQGF8puWgTG5I1FaiUwyucBLpKv9cWYqOcK43O9DwPvLq5U-oIbBPpSnpGnPZStYtkVciWRjedkurPGdZxI9p8Hj2wY10Wn0rAHFrPA1dQ_1jY1YZgtyvy4&e=1696236926&fl=&r=fcc97b55-40bf-417d-8c29-cddd3c12719b-1&k=eYSamczhdroOMzPRK5559Q&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=y1Psi2x-COswIdxTRaAx1GRnwLQ to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/ATikmtq01iWPgrlBu7F7cZoGF8pu.heic as file already exists
[2023-10-02T07:57:02.538Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/ATc43vPaLgGwVIQzj3y5qrv5dOs3AQZHCga0rMmVHwgVFJgtGf6U5XWM/${f}?o=AgllOklBGLP2mjIUKArMnR_0oCgNu9O52Y0t2olXhqJB&v=1&x=3&a=CAogLuahuEe79xUydVKDG0LU4Han6ypGKZFzBsCdBHfVUa0SaxCujIr7rjEYrunl_K4xIgEAUgT5dOs3WgSU5XWMaiUNsqtBveMFKGS-XKFg8y3PwyLA2aE7kFiBEIGZSlod2lQ5cLZociUYX9OSkAkw9Cc_rYAXhspaeSmzVGc0Ad8WnSy2zEkCQD8aF59H&e=1696236926&fl=&r=0cc2ba43-2842-490f-b9f0-f79b9bc3cc31-1&k=wfGpmLpqmv0h5kM2F7ciig&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=DvEPMc6iCpOr519ENpDVBwuiSn8 to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/ATc43vPaLgGwVIQzj3y5qrv5dOs3.jpeg as file already exists
[2023-10-02T07:57:02.538Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AeOi5QkJ-TwQ71lep6v0JuyaYHGyAR5n7eJS4zw4tNp1nfki8MX8AuWW/${f}?o=AnQbFf4e_7z6t4jXOGGvl31via885YWQnDDgir9L0nlG&v=1&x=3&a=CAogP6nBxipY34m_dMGFCEc9NWLWduToGxdfMoshMqN5kMoSaxCkkYr7rjEYpO7l_K4xIgEAUgSaYHGyWgT8AuWWaiWCYg-8QCkFClQDR36-Nwd3WYeVL3mmnNaBTskZ6BhhAEZ00CqDciUWu952ICeBQlHWyX8KqyiWv9BKSLLrPP1c5hZ3aHSvjubT4SAm&e=1696236926&fl=&r=b96de211-f7bf-4ec6-9155-d6ed78ec6e95-1&k=GMbNOAJ95h2E515oSWT5eg&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=u9eQPOvBxi-nNKTNZh6TU0DhPN4 to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AeOi5QkJ-TwQ71lep6v0JuyaYHGy.heic as file already exists
[2023-10-02T07:57:02.538Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AVU3zstvhywey0aYlhDLSfZ0PbcDAchQsVizMI4_NcRjSYNSexPGKbUt/${f}?o=AlcuePXHnL1wGUYSAQWaKTJKskF0IjQ3jjskABiBvm9U&v=1&x=3&a=CAogat3EDNpFic2ja-X7zewaDzC-pcTZwznLcm6NEqAu2KoSaxCxkor7rjEYse_l_K4xIgEAUgR0PbcDWgTGKbUtaiXNHR5zwfvb5WP9J44n3SUVFMSgzlqZ9ZfI7eqSCaTGjQqCV5vkciXYslTpDvJMVb37HG3e6WoxEkpc---uxXaHNEW5lNTiqPbE47_z&e=1696236926&fl=&r=9661e640-24e9-489d-9680-2e947274a683-1&k=WyWy2iJlL6_imPTPa6yBkA&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=LB0izroFNDSGfqATy-SniNk7d0g to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AVU3zstvhywey0aYlhDLSfZ0PbcD.heic as file already exists
[2023-10-02T07:57:02.538Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AV7HjHt5i2eAnxTmJE9qdVlxXDn0AVeEEN9FYv-YD1YH-ymx7G8BwMms/${f}?o=Ao4JAbj6T0ABCXJZxyqmoAfD5B2mk6q-HCHDwo-z82e9&v=1&x=3&a=CAog9SH1UwgBi-QqmXNLFCY96X6zDmP2xqExmWz5qDx4gYgSaxD2j4r7rjEY9uzl_K4xIgEAUgRxXDn0WgQBwMmsaiVmOcVACkvnfKPOmKOedgKq8oAnz3_hdeUe0iekFCxcDcLIVrieciXD1XcyMsgiXPRm-j0agnLv2RWtcrbBL7CG8Crm2NLCADe7tbDF&e=1696236926&fl=&r=6a78c4e7-c28b-48d8-b86e-15e23b1971ec-1&k=DrA7y0vcTb4KmNk67Nc8cg&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=6AGMVhvsZuQO0vmp49f0ghwd5FQ to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AV7HjHt5i2eAnxTmJE9qdVlxXDn0.jpeg as file already exists
[2023-10-02T07:57:02.558Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/ATIb4L3pzTAgNA2Wz0BXaIEqTKYlATZHeRTIB8RXyaPeLdWDkmZvM5YM/${f}?o=ArnG-GmGAHK8V0v3cYLnrmBRv_e5pJVenq_uAqp8KRG8&v=1&x=3&a=CAogIqmOEHX48DyQtgQnTa1JNd7HaKp3ZeZ4efXYfr_vbjMSaxDeqIr7rjEY3oXm_K4xIgEAUgQqTKYlWgRvM5YMaiUJlGNm-rJ73d0Yp-jCnDTVjjSq_Tr3Il-QdnLNpal6R3KHNA3pciXIxgKhn3uc19PUV58MravqwDMuzMeebPjC0kEcIcqwbqUewYML&e=1696236929&fl=&r=6171002c-11e2-4b1c-a6e0-d0a26b5218e9-1&k=PIStNQ-0dlFgE8BHqtsVdw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=14TlSBRJ7d2eAAuCI5kqbOVmn-I to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/ATIb4L3pzTAgNA2Wz0BXaIEqTKYl.jpeg as file already exists
[2023-10-02T07:57:02.560Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AaPtUcO-Pr2VeNgDB0whOvQMFbipASrM_vI9hQqUmksK8Xw1NyF8I29U/${f}?o=AsXkB_vcV3srdXM-szlq-GOH3k6MlPTUbocnXtEURyzE&v=1&x=3&a=CAogkA0XPMWzssXEsa_BHGFU7BI03IRSJgDK4XB-2YS_we4SaxDYq4r7rjEY2Ijm_K4xIgEAUgQMFbipWgR8I29UaiXWY8_dHFYLO8bnca_Ze29gmt2rGIGMSPrLgQiM-jkdoncTS88iciX9yW5KB5U97uiQtWZ3op_cPxdgT-Jk0QS4b6Vzzu6K_EkpngfX&e=1696236930&fl=&r=42537e58-8749-4827-8df6-d9313ace6372-1&k=C1CXgq-f4NyJJZbfEc8dJA&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=hewYachcjHB3LhgJ8clBXwjf7CY to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AaPtUcO-Pr2VeNgDB0whOvQMFbip.jpeg as file already exists
[2023-10-02T07:57:02.560Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AdPMjgH05zOBx1M_9FjpVvlwk_yoARlIQTdSByFJkFb-AeGe2-5u52BP/${f}?o=AhFtkxsGCkjPyUG40eo-HZ-0ApvaGlt19a73NLVLwN1O&v=1&x=3&a=CAogx62UWsiWvSoMcyAQo02UoZFvpiVtdzWDQnnYxr5OwoYSaxCRrYr7rjEYkYrm_K4xIgEAUgRwk_yoWgRu52BPaiVHL9c1JpLbpT9gP2a0C9_-3t3t5vochOnwXe0UUj_wf7NlG5d3ciWchKtQUJSm9LE2kirHuygviGNP5tx5z2dIiwoF3_rqMNSttJiq&e=1696236930&fl=&r=f0518c0a-5feb-492d-8c4a-e7cc4ae8fe6d-1&k=is29TzrbF6fm8sKvPwAITQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=Yoyl92a2MirHL-MkXxFshjHNOAo to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AdPMjgH05zOBx1M_9FjpVvlwk_yo.jpeg as file already exists
[2023-10-02T07:57:02.560Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/ARkKPPyn_yQ3QCmRhnDHr-9gn88hAQ19TVsN7pgaJnh0FiR0s-PCtjjw/${f}?o=Alk8KMlvrWO0oAwttUwQg7b0pThZEb_zw9JadiyzTMUz&v=1&x=3&a=CAogbmsr3--pnB03bMMNLQJpkMBIBR2zcOR7OedCcYVcnuUSaxDdr4r7rjEY3Yzm_K4xIgEAUgRgn88hWgTCtjjwaiWvs774yPfmQb5Maga5PnEMG4UcP7p_CfQZWcvnB0H8bA_BgW1ZciVPt6UJihCbXrjqnS0btyAmctcup5179h1RsLkxXndW1lA9vcFO&e=1696236930&fl=&r=9eeb359e-a219-4b3f-8266-ef712a4274c9-1&k=-YSA5-aVwrd1BlWtFstGZQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=eOaCMg2RUnCAQmhOpOb1Insgu28 to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/ARkKPPyn_yQ3QCmRhnDHr-9gn88h.jpeg as file already exists
[2023-10-02T07:57:02.561Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/ARQmBHEtYnXU4jwoQrQ20khA-JQbAarefYd2p8jc6u65QpuQsrpxdOiO/${f}?o=ApW3SLpIeIX-JxEXG8Wd6CZztTQ4JZWTk9CPDcp-dg2O&v=1&x=3&a=CAoghk-s4ryxwJE1DrPAeTeOMSqdGSTnuj7I9pA6QCF4Wq4SaxDytIr7rjEY8pHm_K4xIgEAUgRA-JQbWgRxdOiOaiWnylSRSW8kLxQnlKuMtcRbkkRM7r3_O396Fz0Id6o-PKZkkKNDciXUMiHRLCbSVSwDrpY5tsgrgYtrWeUuyR4v1W2pNVdl46g6VFx4&e=1696236931&fl=&r=df060f69-36d0-41fe-87e2-6394426f36d4-1&k=jVoB1K4S0Ruoz0zYRJC_Fg&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=DhCL8XNBi7jZS8nmh1URvUPaFXI to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/ARQmBHEtYnXU4jwoQrQ20khA-JQb.jpeg as file already exists
[2023-10-02T07:57:02.561Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AXxv6TmcecKdj84QhEEmyh1V4E-sAciSZBhUmkmM677ePJPnen-Swjnv/${f}?o=AiAD5yrwbjwfXcSvTOp1rWgnNeE0bmZatuo0kO8wF5iV&v=1&x=3&a=CAog2NQLuEZzcYK4WYcDg8rPUpIecS0dKeqN30k0S3Hl9jMSaxCSvIr7rjEYkpnm_K4xIgEAUgRV4E-sWgSSwjnvaiVR3Ze_cq-G25jr5DwP0KFSohwiQYB6xatHenpiC8iYIBEfYUsrciXuTmvNd8pdrGSm92qv4oJYoUBUj71Q1YpUiU4ShiiZxE9JhLxH&e=1696236932&fl=&r=a79882c3-8e6c-46a5-851d-4bd2743d7e1e-1&k=4TiMl4sHpxwB7qchw-hEvQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=7QUP8kDInfY0r42H6wVOLfeMKN0 to /Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AXxv6TmcecKdj84QhEEmyh1V4E-s.mov as file already exists
[2023-10-02T07:57:53.083Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AW5I0mhjHlvqlWkMwNcHFvfuFVTFAQor9ZUuYL3I4MnYD3HDkaSZSFw6/${f}?o=AoyClGsOPkb7a2I0fKQt4UOAHZZGkRf5kN9CBLUYc3nv&v=1&x=3&a=CAogdb5YBkLEPOyMaDlxlKqsv_4ukr0CcJpaIYpybJ-CfCASaxCOiYv7rjEYjubm_K4xIgEAUgTuFVTFWgSZSFw6aiXEiQpB5OWxxZCAH69EaJNBguraXmCtrT7-YW2KzAB5nuT7vneEciWqubPxHmN1Dl7aJadpeOzOorvLDR_WNcj5nfhMZwgfnANH-mpk&e=1696236942&fl=&r=e4cfef16-1e5f-44ff-a9ba-8e597bb85d65-1&k=SpXXjBiyD8_R2CajzsKo9Q&ckc=com.apple.photos.cloud&ckz=SharedSync-50A3C536-9BE0-457A-8A26-A8FD65DE083A&y=1&p=58&s=jqIJRxr_cgP7X_C4Z0eeWz-0yTM to /Volumes/Extreme SSD/icloud-photos-sync/_Shared-Photos/AW5I0mhjHlvqlWkMwNcHFvfuFVTF.heic as file already exists
[2023-10-02T07:57:53.083Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AcUVSByyL23BpgEUQ_JMn8lXHGs_ATj-u7PL5-WsgcOxF0ad7u_muAU0/${f}?o=ApVav2ebI8dOdk6266fvYEAjUMtmwptgqqQFl4kj-IAF&v=1&x=3&a=CAogOvaooR5f0tfPFVpOMTGPxI24dNxV-PyrSPNzNRPVktkSaxDShov7rjEY0uPm_K4xIgEAUgRXHGs_WgTmuAU0aiWIW-kds3ByT2_GcuGuV3woPgu8bWK9DdqxnEMKQPcRZxaCgSe4ciVJ35gQMpQOOsVvwMiglTt3H7Iy7WjluaSyZirSnW_8ARpbtj4x&e=1696236941&fl=&r=75119965-e1af-4d16-ab4e-17ff4da3803e-1&k=2HV54FkjXzuc8L46Wmn1WQ&ckc=com.apple.photos.cloud&ckz=SharedSync-50A3C536-9BE0-457A-8A26-A8FD65DE083A&y=1&p=58&s=q17ErsjMVSN6wDlHGh52k67RQrY to /Volumes/Extreme SSD/icloud-photos-sync/_Shared-Photos/AcUVSByyL23BpgEUQ_JMn8lXHGs_.heic as file already exists
[2023-10-02T07:57:53.083Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AQ2UBfAA4TfovvOiFMJSHG3POVRuAew7MUBsrJKXuKYMmuBZZQsh45Pm/${f}?o=AkH2KWDeI_409wNA-KnAd3ULPPzgYApchNT91z7D50PV&v=1&x=3&a=CAogRPq1Tnhr9zSQt8TdOPKbaI8txiDl7gK6y4GzWijmxn0SaxDZhov7rjEY2ePm_K4xIgEAUgTPOVRuWgQh45PmaiWszJnHGk5kqEb2LDzOZy9DctMJQUji8TlC50qyML8suVKwMm_qciXsgvKzvMPehhxM6VaI3pSerSrZBUOkPMHsxH-goh5mXvolBCmi&e=1696236941&fl=&r=75119965-e1af-4d16-ab4e-17ff4da3803e-1&k=KeFP81qhY5D8iPHoB24b9Q&ckc=com.apple.photos.cloud&ckz=SharedSync-50A3C536-9BE0-457A-8A26-A8FD65DE083A&y=1&p=58&s=VyQEUceKUvnBBmMj5plVxOLjkDU to /Volumes/Extreme SSD/icloud-photos-sync/_Shared-Photos/AQ2UBfAA4TfovvOiFMJSHG3POVRu.heic as file already exists
[2023-10-02T07:57:53.083Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AULqwlReOizZfb5jAcpYqoZWK7rNAauV4UnoWLVA7V-VxF_016IB-VzC/${f}?o=AvjJSB09cRYoVw32FWt5QMmBaCwMTfExxnKoqWqrRlme&v=1&x=3&a=CAoguRXJiyKKZCK1Ljz9qadIU2sZprKtuvYhiuZulE69icISaxDfhov7rjEY3-Pm_K4xIgEAUgRWK7rNWgQB-VzCaiUeo8SL3U4kRUnjfpRT8eyuHbb9JqPAeG0MP2MW4htI4Hzu6JaIciWpOVEzI4sB3IgFoN8tPY9XBkAln_qP7GVd9zFs_l25nmwFkg2I&e=1696236941&fl=&r=75119965-e1af-4d16-ab4e-17ff4da3803e-1&k=oXddsGknUzqVIqWEEUP1Bw&ckc=com.apple.photos.cloud&ckz=SharedSync-50A3C536-9BE0-457A-8A26-A8FD65DE083A&y=1&p=58&s=tJHBx32QxUn0BCArVFo68BDyShQ to /Volumes/Extreme SSD/icloud-photos-sync/_Shared-Photos/AULqwlReOizZfb5jAcpYqoZWK7rN.heic as file already exists
[2023-10-02T07:57:53.083Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/AWivSppuKBSK3Xs9Qj8dbtNTcg0cAbLCk62VzgaXLyulf5swyeoVFVdv/${f}?o=AosLFDvT3HlHhvGwLavB40AGuF_xZ76YZddbcdN--yAw&v=1&x=3&a=CAogkaIkYlNtNc5cgMXVDfyheFqvb-pe-lvscbG5_Rtt0wUSaxDmhov7rjEY5uPm_K4xIgEAUgRTcg0cWgQVFVdvaiW1HyNrPyLjfQgE6iMhS5ldsuMMvI0-HUXzxX5iqSPQyldtvtsVciWLZfU-8TGNPCpzim8oobh2Zq8luDP4wpiLA1lrhuJ3W3fkX_Da&e=1696236941&fl=&r=75119965-e1af-4d16-ab4e-17ff4da3803e-1&k=ccPtzagNUroskCgXfd_QzA&ckc=com.apple.photos.cloud&ckz=SharedSync-50A3C536-9BE0-457A-8A26-A8FD65DE083A&y=1&p=58&s=Y_gI7EUoO-jt9m-pfiSnPBSQ5yk to /Volumes/Extreme SSD/icloud-photos-sync/_Shared-Photos/AWivSppuKBSK3Xs9Qj8dbtNTcg0c.heic as file already exists
[2023-10-02T07:57:53.084Z] WARN NetworkManager: Skipping download of https://cvws.icloud-content.com/B/ASr6HovI9ZB7cgD7C-fRWgcsBpPtAZuk9Jt6_OQv5oDmv0P8oVgFPpSZ/${f}?o=AtdiRyU39ZFHr6tuO4d1aBAAZaFXbLKx-qr1XRX_SknD&v=1&x=3&a=CAogJ_PnZzkBvGfxKbzU0vBoFlMzH2TPd3TiJqAQxefgvF4SaxDthov7rjEY7ePm_K4xIgEAUgQsBpPtWgQFPpSZaiX8ZPEHlF5lTU_P50YL1JtYLjVzVIsrVchItG4R6adSaCkJNG9_ciV062f8Ha0UxHmoQGau_8VBTWWwy0EdSpwtn-63uONdJEKmrtSO&e=1696236941&fl=&r=75119965-e1af-4d16-ab4e-17ff4da3803e-1&k=eBv6rmUnVqBybJE0_6nW8w&ckc=com.apple.photos.cloud&ckz=SharedSync-50A3C536-9BE0-457A-8A26-A8FD65DE083A&y=1&p=58&s=068gXnkhALSmVYNOQ6PBZ9L03_o to /Volumes/Extreme SSD/icloud-photos-sync/_Shared-Photos/ASr6HovI9ZB7cgD7C-fRWgcsBpPt.heic as file already exists

Interesting enough there is still some progress being made in the last hours, but that could only be a coincidence:

Syncing assets, by keeping 86269 and removing 12 local assets, as well as adding 14751 remote assets...
 ■■■■■■                                   15% | Elapsed: 59m34s | 2281/14751 assets downloaded
Syncing assets, by keeping 87135 and removing 19 local assets, as well as adding 13887 remote assets...
 ■■■■■■■                                  16% | Elapsed: 58m52s | 2335/13887 assets downloaded
Syncing assets, by keeping 88016 and removing 20 local assets, as well as adding 13006 remote assets...
 ■■■■■■■                                  18% | Elapsed: 58m49s | 2399/13006 assets downloaded
Syncing assets, by keeping 88948 and removing 20 local assets, as well as adding 12074 remote assets...
 ■■■■■■■■■                                22% | Elapsed: 56m11s | 2733/12074 assets downloaded
steilerDev commented 1 year ago

That's good and bad news!

This means our assumption is correct, however I think this will not be addressed fully until #354 - however it is an important consideration!

You should see more progress now because already existing files will be skipped - before the change it would re-download the file.

Now the important question would be: Are those duplicates or distinct files that have the same ID. Let me clean up the warnings and create a separate folder, where those conflicted files are written to - you can then check if those are duplicates (and it would be safe to ignore) or actually different files that we need to keep.

steilerDev commented 1 year ago

I've just added some rough (and untested) debugging logic to v1.3.0-nightly.4 that should help us understand the issue better:

If a duplicate is detected (file already exists) the tool will create a new folder within the data dir (duplicates) and:

If you could then check the duplicate filenames to see if those are actually the same file or different ones (i.e. is it a naming conflict, or a de-duplication problem)

halbtuerke commented 1 year ago

Seems like the logic is not quite right. I'm getting the following error message and the sync immediately fails:

Syncing assets, by keeping 92316 and removing 0 local assets, as well as adding 8726 remote assets...
                                          0% | Elapsed: 0s | 0/8726 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by EEXIST: file already exists, link '/Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AeOi5QkJ-TwQ71lep6v0JuyaYHGy.heic' -> '/Volumes/Extreme SSD/icloud-photos-sync/duplicates/AeOi5QkJ-TwQ71lep6v0JuyaYHGy.heic-orig'
Refreshing iCloud connection & retrying (attempt #5)...
steilerDev commented 1 year ago

ah of course...one sec....and I guess there is already some stuff in the duplciate folder?

halbtuerke commented 1 year ago

Yes, there are 12 items in the duplicates folder:

 AeOi5QkJ-TwQ71lep6v0JuyaYHGy.heic-dup-68595
 AeOi5QkJ-TwQ71lep6v0JuyaYHGy.heic-orig
 AfOyvgkFncw-xHiRSYu7n29m3sqn.jpeg-dup-439576
 AfOyvgkFncw-xHiRSYu7n29m3sqn.jpeg-orig
 ATc43vPaLgGwVIQzj3y5qrv5dOs3.jpeg-dup-319799
 ATc43vPaLgGwVIQzj3y5qrv5dOs3.jpeg-orig
 ATikmtq01iWPgrlBu7F7cZoGF8pu.heic-dup-172507
 ATikmtq01iWPgrlBu7F7cZoGF8pu.heic-orig
 AV7HjHt5i2eAnxTmJE9qdVlxXDn0.jpeg-dup-299436
 AV7HjHt5i2eAnxTmJE9qdVlxXDn0.jpeg-orig
 AVU3zstvhywey0aYlhDLSfZ0PbcD.heic-dup-866613
 AVU3zstvhywey0aYlhDLSfZ0PbcD.heic-orig
steilerDev commented 1 year ago

And how is the content of those files looking? Are e.g. AeOi5QkJ-TwQ71lep6v0JuyaYHGy.heic-dup-68595 and AeOi5QkJ-TwQ71lep6v0JuyaYHGy.heic-orig the same file?

steilerDev commented 1 year ago

Also: try 1.3.0-nightly.5 (which is currently getting deployed)

halbtuerke commented 1 year ago

The files are looking the same and they also have the same file sizes.

For easier comparison it would be helpful if the files would be called AeOi5QkJ-TwQ71lep6v0JuyaYHGy-dup-68595.heic and AeOi5QkJ-TwQ71lep6v0JuyaYHGy-orig.heic, otherwise I have to rename them manually in order to open them.

steilerDev commented 1 year ago

Try 1.3.0-nightly.6 (once deployed) :)

halbtuerke commented 1 year ago

Unfortunately I'm still getting

Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by EEXIST: file already exists, mkdir '/Volumes/Extreme SSD/icloud-photos-sync/duplicates'
Refreshing iCloud connection & retrying (attempt #2)...

and

Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by EEXIST: file already exists, link '/Volumes/Extreme SSD/icloud-photos-sync/_All-Photos/AfOyvgkFncw-xHiRSYu7n29m3sqn.jpeg' -> '/Volumes/Extreme SSD/icloud-photos-sync/duplicates/AfOyvgkFncw-xHiRSYu7n29m3sqn-orig.jpeg'
Refreshing iCloud connection & retrying (attempt #3)...
steilerDev commented 1 year ago

Have a try with 1.3.0-nightly.7 - this should handle the errors gracefully.

I also added additional logging to the diffing algorithm, to try and understand why those files end up in the queue to download (even though they are already on disk) - I you could search the logs for the duplicated file IDs and share the relevant pieces, this should help.

halbtuerke commented 1 year ago

The most recent version is unfortunately hard crashing and therefore it's not producing any logs in which those duplicated IDs are shown:

Syncing assets, by keeping 92314 and removing 0 local assets, as well as adding 8798 remote assets...
                                          0% | Elapsed: 0s | 0/8798 assets downloadednode:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: EEXIST: file already exists, mkdir '/Volumes/Extreme SSD/icloud-photos-sync/duplicates'] {
  errno: -17,
  code: 'EEXIST',
  syscall: 'mkdir',
  path: '/Volumes/Extreme SSD/icloud-photos-sync/duplicates'
}

Node.js v20.8.0
halbtuerke commented 1 year ago

The most recent version is unfortunately hard crashing

After some more investigations I figured out that the crash was caused by the newer Node.js version I have installed on my machine. After installing and using the current LTS, the crashes disappeared.

If you could search the logs for the duplicated file IDs and share the relevant pieces, this should help.

Here are the log lines for one of the duplicates (Aa_fDRoooKJJY-0nHI30XCM9rfi). The sync ran for ~9 hours and is now stuck at the last 8722 assets:

   200806  [2023-10-06T22:46:21.467Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/AfItei6RTLo19UyN2h091dhYLsPXAckdzfuZ5sxghoCg0gB0FyrpAU65/${f}?o=AmZ-Ys62bZnPrC_v8We2vANZKL4rrhur6MbKAfvQSPDD&v=1&x=3&a=CAogIxdKM7eRaPlbGl3WJ5l8lKjjzzOio7OukGWduTk6GMgSaxDSy-u3sDEY0qjHubAxIgEAUgRYLsPXWgTpAU65aiVN7lR9SKsok3lm_PzVjkQIoOYoRNBAvZ4PcLb2GODPDNNiLLNrciVwhbvE2WZ_XxlSyCxeYoGJFUW_TQAhtot6L83EDF4UOgS68IiT&e=1696632788&fl=&r=31b8f91d-c030-41f7-9814-4dd83c11175f-1&k=2yePyBKJJ8UYKtGXkrp85Q&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=9KanIi9qCI7Y6CdMXPx88j5mRJA
   200807  [2023-10-06T22:46:21.467Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/AfItei6RTLo19UyN2h091dhYLsPXAckdzfuZ5sxghoCg0gB0FyrpAU65/${f}?o=AmZ-Ys62bZnPrC_v8We2vANZKL4rrhur6MbKAfvQSPDD&v=1&x=3&a=CAogIxdKM7eRaPlbGl3WJ5l8lKjjzzOio7OukGWduTk6GMgSaxDSy-u3sDEY0qjHubAxIgEAUgRYLsPXWgTpAU65aiVN7lR9SKsok3lm_PzVjkQIoOYoRNBAvZ4PcLb2GODPDNNiLLNrciVwhbvE2WZ_XxlSyCxeYoGJFUW_TQAhtot6L83EDF4UOgS68IiT&e=1696632788&fl=&r=31b8f91d-c030-41f7-9814-4dd83c11175f-1&k=2yePyBKJJ8UYKtGXkrp85Q&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=9KanIi9qCI7Y6CdMXPx88j5mRJA
   200808: [2023-10-06T22:46:21.472Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-369127.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-369127.heic and linking original file Aa_fDRoooKJJY-0nHI30XCM9rfi--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
   200809: [2023-10-06T22:46:21.472Z] DEBUG NetworkManager: Starting download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AnCcyqOmsb2xQTljccFFYi74MqV-1jig5vzO0w6yFd6_&v=1&x=3&a=CAogmi1CkwuVf4jSZ1lBKU1MB2XYAX9ckKQWEG94i4P1nZ0SaxD8y-u3sDEY_KjHubAxIgEAUgQ9rfi-WgS9OF6NaiV-Fc1cUpgsOLD6OtKID5KAUoXlusj6X8Q-PwGAnVuMfpSXdqG6ciWko1zSP5rJaxuKcaoDiIDlF1xrUkp2tdwt76y5gIoIG4FHd6nq&e=1696632788&fl=&r=31b8f91d-c030-41f7-9814-4dd83c11175f-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=TJMXDFosE0s5J9gDOfhX6Xx3PTc
   200810: [2023-10-06T22:46:21.979Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AnCcyqOmsb2xQTljccFFYi74MqV-1jig5vzO0w6yFd6_&v=1&x=3&a=CAogmi1CkwuVf4jSZ1lBKU1MB2XYAX9ckKQWEG94i4P1nZ0SaxD8y-u3sDEY_KjHubAxIgEAUgQ9rfi-WgS9OF6NaiV-Fc1cUpgsOLD6OtKID5KAUoXlusj6X8Q-PwGAnVuMfpSXdqG6ciWko1zSP5rJaxuKcaoDiIDlF1xrUkp2tdwt76y5gIoIG4FHd6nq&e=1696632788&fl=&r=31b8f91d-c030-41f7-9814-4dd83c11175f-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=TJMXDFosE0s5J9gDOfhX6Xx3PTc to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-369127.heic
   200811: [2023-10-06T22:46:23.787Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AnCcyqOmsb2xQTljccFFYi74MqV-1jig5vzO0w6yFd6_&v=1&x=3&a=CAogmi1CkwuVf4jSZ1lBKU1MB2XYAX9ckKQWEG94i4P1nZ0SaxD8y-u3sDEY_KjHubAxIgEAUgQ9rfi-WgS9OF6NaiV-Fc1cUpgsOLD6OtKID5KAUoXlusj6X8Q-PwGAnVuMfpSXdqG6ciWko1zSP5rJaxuKcaoDiIDlF1xrUkp2tdwt76y5gIoIG4FHd6nq&e=1696632788&fl=&r=31b8f91d-c030-41f7-9814-4dd83c11175f-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=TJMXDFosE0s5J9gDOfhX6Xx3PTc

   200811  [2023-10-06T22:46:23.791Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-357440.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-357440.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
   200812  [2023-10-06T22:46:23.791Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-357440.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-357440.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates

   402749  [2023-10-06T23:55:44.108Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AjC3fE4S9V3fRIJiARbrUft_sNqyrHkxSENHF-Qg9Li6&v=1&x=3&a=CAogVXagR3msFCuo87l8d8JDLrsrya7x0sfpsxCWjFwDx34SaxDzyeS5sDEY86bAu7AxIgEAUgTQ6AqTWgTt6wTaaiUp52oJ28Dy0r-FH-Il-NqNJl6SyDexFRilBBibQYE7qL3EKPJGciWkO98MBMx1wQINC6uDNlNSCsfJ1ltDagnuDIXCdSNd3i8IYPEZ&e=1696636867&fl=&r=e36c9381-d6ea-446a-8921-6d06da3b9939-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=86AhpaoTMrlzFh14SA7iqG7bTvY
   402750  [2023-10-06T23:55:44.108Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AjC3fE4S9V3fRIJiARbrUft_sNqyrHkxSENHF-Qg9Li6&v=1&x=3&a=CAogVXagR3msFCuo87l8d8JDLrsrya7x0sfpsxCWjFwDx34SaxDzyeS5sDEY86bAu7AxIgEAUgTQ6AqTWgTt6wTaaiUp52oJ28Dy0r-FH-Il-NqNJl6SyDexFRilBBibQYE7qL3EKPJGciWkO98MBMx1wQINC6uDNlNSCsfJ1ltDagnuDIXCdSNd3i8IYPEZ&e=1696636867&fl=&r=e36c9381-d6ea-446a-8921-6d06da3b9939-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=86AhpaoTMrlzFh14SA7iqG7bTvY
   402751: [2023-10-06T23:55:44.111Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-901792.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-901792.heic and linking original file Aa_fDRoooKJJY-0nHI30XCM9rfi--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
   402752: [2023-10-06T23:55:44.111Z] DEBUG NetworkManager: Starting download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=Av8LhBhzr-llIWrMzyfrvVVrxPlHKwJbR6Wy_b5w38Tb&v=1&x=3&a=CAog_yt9Ew7ZXmRS0GyKwwDxgGuaJQrMfAME10Jjug1GSxsSaxChyuS5sDEYoafAu7AxIgEAUgQ9rfi-WgS9OF6NaiW_b4Y-MeDSh2hkShozsPZURjQjQtaJ7M9cDFG84zy-iZJWWedhciURt0lqf0lX8ZTCj2WePZAWKBoG0lKw_lboOoxCd_Q9lWazmS6H&e=1696636867&fl=&r=e36c9381-d6ea-446a-8921-6d06da3b9939-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=NWIE1f4O6f8UZ80XNn4l-DLx1LY

   402752  [2023-10-06T23:55:44.480Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/AZZWSm6gsVdoSmzLK2OINh6mkmy3AVp_SHXYXgYqepvku1EjoLxEmuFu/${f}?o=AmkkuEUSRhrK4cauU1akNi5h3jwWsp4tPGO6ANYG-vzD&v=1&x=3&a=CAog64NRKbF1ZDsz1N_djWC21gaa3Qbpm9x_E7Q52pmeSu0SaxCDyuS5sDEYg6fAu7AxIgEAUgSmkmy3WgREmuFuaiWUyCoTon5_mYqFMkujfUZnGC-0E3AuIuWUnb0rGv8J43zdEqMEciXvMLjd0QqoO0GibVWzdkE9o7pR5AHGJCXTFeTIDOZDWO3vvApA&e=1696636867&fl=&r=e36c9381-d6ea-446a-8921-6d06da3b9939-1&k=EtBJ5EAWb4kWKl-RiKhb3g&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=F4Uxg1pmSYQkbEhly3O36MorLx0 to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AZZWSm6gsVdoSmzLK2OINh6mkmy3-dup-59159.mov
   402753  [2023-10-06T23:55:44.480Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/AZZWSm6gsVdoSmzLK2OINh6mkmy3AVp_SHXYXgYqepvku1EjoLxEmuFu/${f}?o=AmkkuEUSRhrK4cauU1akNi5h3jwWsp4tPGO6ANYG-vzD&v=1&x=3&a=CAog64NRKbF1ZDsz1N_djWC21gaa3Qbpm9x_E7Q52pmeSu0SaxCDyuS5sDEYg6fAu7AxIgEAUgSmkmy3WgREmuFuaiWUyCoTon5_mYqFMkujfUZnGC-0E3AuIuWUnb0rGv8J43zdEqMEciXvMLjd0QqoO0GibVWzdkE9o7pR5AHGJCXTFeTIDOZDWO3vvApA&e=1696636867&fl=&r=e36c9381-d6ea-446a-8921-6d06da3b9939-1&k=EtBJ5EAWb4kWKl-RiKhb3g&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=F4Uxg1pmSYQkbEhly3O36MorLx0 to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AZZWSm6gsVdoSmzLK2OINh6mkmy3-dup-59159.mov
   402754: [2023-10-06T23:55:44.540Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=Av8LhBhzr-llIWrMzyfrvVVrxPlHKwJbR6Wy_b5w38Tb&v=1&x=3&a=CAog_yt9Ew7ZXmRS0GyKwwDxgGuaJQrMfAME10Jjug1GSxsSaxChyuS5sDEYoafAu7AxIgEAUgQ9rfi-WgS9OF6NaiW_b4Y-MeDSh2hkShozsPZURjQjQtaJ7M9cDFG84zy-iZJWWedhciURt0lqf0lX8ZTCj2WePZAWKBoG0lKw_lboOoxCd_Q9lWazmS6H&e=1696636867&fl=&r=e36c9381-d6ea-446a-8921-6d06da3b9939-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=NWIE1f4O6f8UZ80XNn4l-DLx1LY to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-901792.heic
   402755: [2023-10-06T23:55:45.157Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=Av8LhBhzr-llIWrMzyfrvVVrxPlHKwJbR6Wy_b5w38Tb&v=1&x=3&a=CAog_yt9Ew7ZXmRS0GyKwwDxgGuaJQrMfAME10Jjug1GSxsSaxChyuS5sDEYoafAu7AxIgEAUgQ9rfi-WgS9OF6NaiW_b4Y-MeDSh2hkShozsPZURjQjQtaJ7M9cDFG84zy-iZJWWedhciURt0lqf0lX8ZTCj2WePZAWKBoG0lKw_lboOoxCd_Q9lWazmS6H&e=1696636867&fl=&r=e36c9381-d6ea-446a-8921-6d06da3b9939-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=NWIE1f4O6f8UZ80XNn4l-DLx1LY

   402755  [2023-10-06T23:55:45.161Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-581831.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-581831.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
   402756  [2023-10-06T23:55:45.161Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-581831.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-581831.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates

   604706  [2023-10-07T01:05:20.376Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AjPiuJHmDp7lzCZgptjx8KTePf7onRUPyyNt4-dJkf02&v=1&x=3&a=CAogmzEvYGD5BBjZkaxYjZ1_Bir6Jo2ZZpr4vtDT2gmg214SaxDA9eC7sDEYwNK8vbAxIgEAUgTQ6AqTWgTt6wTaaiXccDvHRNzmgrRWFijreJnuqcN2jlVV7EZJh2cwE6l4qm15MZOQciVgSXX6PWMtmmT_h7gTmpvxoxy-Ftb-amYqCicJsf254foOVs0b&e=1696641001&fl=&r=c8323a30-f10d-4db5-82e5-f92bb78339e9-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=dHfLDtyrxNNX3RvYrpX-f456B2E
   604707  [2023-10-07T01:05:20.376Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AjPiuJHmDp7lzCZgptjx8KTePf7onRUPyyNt4-dJkf02&v=1&x=3&a=CAogmzEvYGD5BBjZkaxYjZ1_Bir6Jo2ZZpr4vtDT2gmg214SaxDA9eC7sDEYwNK8vbAxIgEAUgTQ6AqTWgTt6wTaaiXccDvHRNzmgrRWFijreJnuqcN2jlVV7EZJh2cwE6l4qm15MZOQciVgSXX6PWMtmmT_h7gTmpvxoxy-Ftb-amYqCicJsf254foOVs0b&e=1696641001&fl=&r=c8323a30-f10d-4db5-82e5-f92bb78339e9-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=dHfLDtyrxNNX3RvYrpX-f456B2E
   604708: [2023-10-07T01:05:20.379Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-885537.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-885537.heic and linking original file Aa_fDRoooKJJY-0nHI30XCM9rfi--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
   604709: [2023-10-07T01:05:20.379Z] DEBUG NetworkManager: Starting download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AqPASkZ6B-XeGp7043ynJ0KcwUMuP4-LHUHtBQ5Od2kM&v=1&x=3&a=CAogebcMeoUNry9yjCZ0ltiSnx6VbKDzM2zBFREZRl3LOwASaxDf9eC7sDEY39K8vbAxIgEAUgQ9rfi-WgS9OF6NaiUZ4JQFuN_IrJrm9n6WtpM-hYgZ4cOvrEZNkaIp8MCvNnseZ7MKciWYbAXHnb1crN-JusBrrUzJBQTvfzIds1JLMwd7EO_Xl8fybAC9&e=1696641001&fl=&r=c8323a30-f10d-4db5-82e5-f92bb78339e9-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=SA2pCUHion2TCOY6yQYQLxWD-i0
   604710: [2023-10-07T01:05:20.821Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AqPASkZ6B-XeGp7043ynJ0KcwUMuP4-LHUHtBQ5Od2kM&v=1&x=3&a=CAogebcMeoUNry9yjCZ0ltiSnx6VbKDzM2zBFREZRl3LOwASaxDf9eC7sDEY39K8vbAxIgEAUgQ9rfi-WgS9OF6NaiUZ4JQFuN_IrJrm9n6WtpM-hYgZ4cOvrEZNkaIp8MCvNnseZ7MKciWYbAXHnb1crN-JusBrrUzJBQTvfzIds1JLMwd7EO_Xl8fybAC9&e=1696641001&fl=&r=c8323a30-f10d-4db5-82e5-f92bb78339e9-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=SA2pCUHion2TCOY6yQYQLxWD-i0 to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-885537.heic
   604711: [2023-10-07T01:05:21.376Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AqPASkZ6B-XeGp7043ynJ0KcwUMuP4-LHUHtBQ5Od2kM&v=1&x=3&a=CAogebcMeoUNry9yjCZ0ltiSnx6VbKDzM2zBFREZRl3LOwASaxDf9eC7sDEY39K8vbAxIgEAUgQ9rfi-WgS9OF6NaiUZ4JQFuN_IrJrm9n6WtpM-hYgZ4cOvrEZNkaIp8MCvNnseZ7MKciWYbAXHnb1crN-JusBrrUzJBQTvfzIds1JLMwd7EO_Xl8fybAC9&e=1696641001&fl=&r=c8323a30-f10d-4db5-82e5-f92bb78339e9-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=SA2pCUHion2TCOY6yQYQLxWD-i0

   604711  [2023-10-07T01:05:21.380Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-899640.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-899640.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
   604712  [2023-10-07T01:05:21.380Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-899640.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-899640.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates

   806216  [2023-10-07T02:14:08.917Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AnIXSQV-hEbOQU6u8z8hBdxTHKQkszG1VP_3yC2LWxHo&v=1&x=3&a=CAog17MuVEcoez6a12oDzPDwwdSwsuHmRjZb8dY2nYeZuIgSaxCo8d29sDEYqM65v7AxIgEAUgTQ6AqTWgTt6wTaaiVHYY6JwIFeFJmlMrwnPFyXmtqaB3kcgKtDjN75kLBtvNG-z3B9ciWauxtLlSh1tXc0-ml8VjjMNrypMS8w0yP2K-qDaK8YBbk8iAed&e=1696645146&fl=&r=be1d549f-fa42-4e4a-8b49-5f7b0db5d5f3-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=ze_11mYgI5TYYq8r-blH-B6Ee4Y
   806217  [2023-10-07T02:14:08.917Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AnIXSQV-hEbOQU6u8z8hBdxTHKQkszG1VP_3yC2LWxHo&v=1&x=3&a=CAog17MuVEcoez6a12oDzPDwwdSwsuHmRjZb8dY2nYeZuIgSaxCo8d29sDEYqM65v7AxIgEAUgTQ6AqTWgTt6wTaaiVHYY6JwIFeFJmlMrwnPFyXmtqaB3kcgKtDjN75kLBtvNG-z3B9ciWauxtLlSh1tXc0-ml8VjjMNrypMS8w0yP2K-qDaK8YBbk8iAed&e=1696645146&fl=&r=be1d549f-fa42-4e4a-8b49-5f7b0db5d5f3-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=ze_11mYgI5TYYq8r-blH-B6Ee4Y
   806218: [2023-10-07T02:14:08.923Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-698092.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-698092.heic and linking original file Aa_fDRoooKJJY-0nHI30XCM9rfi--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
   806219: [2023-10-07T02:14:08.923Z] DEBUG NetworkManager: Starting download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AhSfIG-khdBMTY5PETD4ZuLxD5BA7JS09E_4zz02uQPl&v=1&x=3&a=CAogiLJ60ysKG964vhVGC1IeR5UmVoNDXppvJq5zfKN74TgSaxDG8d29sDEYxs65v7AxIgEAUgQ9rfi-WgS9OF6NaiX9dgrE5ZzWzJLiyjUpIvgQTLFkgkkZRVfrRnJ71Pqxc9Ge1YR5ciWLHC_DjfJxjmyPrUKCP8G5-WgWplrL5puP7bbPmIW95W_FEkI0&e=1696645146&fl=&r=be1d549f-fa42-4e4a-8b49-5f7b0db5d5f3-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=rj7VSPt9VNX4xcVg1AAeLPAzon4
   806220: [2023-10-07T02:14:09.347Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AhSfIG-khdBMTY5PETD4ZuLxD5BA7JS09E_4zz02uQPl&v=1&x=3&a=CAogiLJ60ysKG964vhVGC1IeR5UmVoNDXppvJq5zfKN74TgSaxDG8d29sDEYxs65v7AxIgEAUgQ9rfi-WgS9OF6NaiX9dgrE5ZzWzJLiyjUpIvgQTLFkgkkZRVfrRnJ71Pqxc9Ge1YR5ciWLHC_DjfJxjmyPrUKCP8G5-WgWplrL5puP7bbPmIW95W_FEkI0&e=1696645146&fl=&r=be1d549f-fa42-4e4a-8b49-5f7b0db5d5f3-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=rj7VSPt9VNX4xcVg1AAeLPAzon4 to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-698092.heic
   806221: [2023-10-07T02:14:09.909Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AhSfIG-khdBMTY5PETD4ZuLxD5BA7JS09E_4zz02uQPl&v=1&x=3&a=CAogiLJ60ysKG964vhVGC1IeR5UmVoNDXppvJq5zfKN74TgSaxDG8d29sDEYxs65v7AxIgEAUgQ9rfi-WgS9OF6NaiX9dgrE5ZzWzJLiyjUpIvgQTLFkgkkZRVfrRnJ71Pqxc9Ge1YR5ciWLHC_DjfJxjmyPrUKCP8G5-WgWplrL5puP7bbPmIW95W_FEkI0&e=1696645146&fl=&r=be1d549f-fa42-4e4a-8b49-5f7b0db5d5f3-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=rj7VSPt9VNX4xcVg1AAeLPAzon4

   806221  [2023-10-07T02:14:09.913Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-35360.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-35360.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
   806222  [2023-10-07T02:14:09.913Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-35360.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-35360.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates

  1008144  [2023-10-07T03:22:36.058Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=Am0XJVnFmyOCQVAvmcZMAsQ_GW-6uKZSmZrOdvyw6X7h&v=1&x=3&a=CAogx9TRzeNfaKP7e8-7nORtAZnutUGJWpbyIuTDEwvm5OoSaxDntti_sDEY55O0wbAxIgEAUgTQ6AqTWgTt6wTaaiXedIPjDLR6pEzF38HlZg_eke6F92SYO4b-BcoDfxxYnxByATE_ciWalRwbvxQP-Noj1DjTN6SYrpwh4GZ8u_Se5wNqVt_DI9si9PIX&e=1696649251&fl=&r=8e11f3e2-9796-411b-8773-3eac2ad47a03-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=Ua5iMJZNR35ML2zI4RZmzOJ0J8s
  1008145  [2023-10-07T03:22:36.058Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=Am0XJVnFmyOCQVAvmcZMAsQ_GW-6uKZSmZrOdvyw6X7h&v=1&x=3&a=CAogx9TRzeNfaKP7e8-7nORtAZnutUGJWpbyIuTDEwvm5OoSaxDntti_sDEY55O0wbAxIgEAUgTQ6AqTWgTt6wTaaiXedIPjDLR6pEzF38HlZg_eke6F92SYO4b-BcoDfxxYnxByATE_ciWalRwbvxQP-Noj1DjTN6SYrpwh4GZ8u_Se5wNqVt_DI9si9PIX&e=1696649251&fl=&r=8e11f3e2-9796-411b-8773-3eac2ad47a03-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=Ua5iMJZNR35ML2zI4RZmzOJ0J8s
  1008146: [2023-10-07T03:22:36.062Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-665359.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-665359.heic and linking original file Aa_fDRoooKJJY-0nHI30XCM9rfi--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
  1008147: [2023-10-07T03:22:36.062Z] DEBUG NetworkManager: Starting download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AnjBPW1JcT0KkA0HtXmHq9pFeoyq-HICLm0-kFy5wOFs&v=1&x=3&a=CAogN5wtCOlSefdsCADhlnnAkMUqkX9PoOVTl4tE3Ll9UakSaxCTt9i_sDEYk5S0wbAxIgEAUgQ9rfi-WgS9OF6NaiVJnqd7wTfb4nfRY1okvzC1xHoDW69T2Sz6pruekDgYjYPYjBlGciXj7k5bq6baQkRt3i9tKRSQD1nVyibPLvirC_9PVYIvRoIiGgdF&e=1696649251&fl=&r=8e11f3e2-9796-411b-8773-3eac2ad47a03-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=kFzURHMGW5SA8C5Ox4vYxzgZ2ps
  1008148: [2023-10-07T03:22:36.484Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AnjBPW1JcT0KkA0HtXmHq9pFeoyq-HICLm0-kFy5wOFs&v=1&x=3&a=CAogN5wtCOlSefdsCADhlnnAkMUqkX9PoOVTl4tE3Ll9UakSaxCTt9i_sDEYk5S0wbAxIgEAUgQ9rfi-WgS9OF6NaiVJnqd7wTfb4nfRY1okvzC1xHoDW69T2Sz6pruekDgYjYPYjBlGciXj7k5bq6baQkRt3i9tKRSQD1nVyibPLvirC_9PVYIvRoIiGgdF&e=1696649251&fl=&r=8e11f3e2-9796-411b-8773-3eac2ad47a03-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=kFzURHMGW5SA8C5Ox4vYxzgZ2ps to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-665359.heic

  1008148  [2023-10-07T03:22:36.517Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/AZZWSm6gsVdoSmzLK2OINh6mkmy3AVp_SHXYXgYqepvku1EjoLxEmuFu/${f}?o=AjLa8FX28hO0PzHncIi4XVI7gNyp3D3PsBCAh1ksj5Te&v=1&x=3&a=CAog3AsqS-koQ9T5GA93KtNnFmzWcIf5vCHW8EL73viJ9wESaxD2tti_sDEY9pO0wbAxIgEAUgSmkmy3WgREmuFuaiVw6H9TztO1cgVdJ9T6OUeRViNEevwagWkFHetyM9R1ZgSKiuYJciXW5dZzhJnK4_2Ij9Az-1MsygKKdfQM7yLG8rOm0DF2y-SfIyjd&e=1696649251&fl=&r=8e11f3e2-9796-411b-8773-3eac2ad47a03-1&k=EtBJ5EAWb4kWKl-RiKhb3g&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=2KoAbpZKsb5AQfHv_OZ3bJi7Acg to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AZZWSm6gsVdoSmzLK2OINh6mkmy3-dup-141970.mov
  1008149  [2023-10-07T03:22:36.517Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/AZZWSm6gsVdoSmzLK2OINh6mkmy3AVp_SHXYXgYqepvku1EjoLxEmuFu/${f}?o=AjLa8FX28hO0PzHncIi4XVI7gNyp3D3PsBCAh1ksj5Te&v=1&x=3&a=CAog3AsqS-koQ9T5GA93KtNnFmzWcIf5vCHW8EL73viJ9wESaxD2tti_sDEY9pO0wbAxIgEAUgSmkmy3WgREmuFuaiVw6H9TztO1cgVdJ9T6OUeRViNEevwagWkFHetyM9R1ZgSKiuYJciXW5dZzhJnK4_2Ij9Az-1MsygKKdfQM7yLG8rOm0DF2y-SfIyjd&e=1696649251&fl=&r=8e11f3e2-9796-411b-8773-3eac2ad47a03-1&k=EtBJ5EAWb4kWKl-RiKhb3g&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=2KoAbpZKsb5AQfHv_OZ3bJi7Acg to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AZZWSm6gsVdoSmzLK2OINh6mkmy3-dup-141970.mov
  1008150: [2023-10-07T03:22:37.215Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AnjBPW1JcT0KkA0HtXmHq9pFeoyq-HICLm0-kFy5wOFs&v=1&x=3&a=CAogN5wtCOlSefdsCADhlnnAkMUqkX9PoOVTl4tE3Ll9UakSaxCTt9i_sDEYk5S0wbAxIgEAUgQ9rfi-WgS9OF6NaiVJnqd7wTfb4nfRY1okvzC1xHoDW69T2Sz6pruekDgYjYPYjBlGciXj7k5bq6baQkRt3i9tKRSQD1nVyibPLvirC_9PVYIvRoIiGgdF&e=1696649251&fl=&r=8e11f3e2-9796-411b-8773-3eac2ad47a03-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=kFzURHMGW5SA8C5Ox4vYxzgZ2ps

  1008150  [2023-10-07T03:22:37.222Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-200160.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-200160.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
  1008151  [2023-10-07T03:22:37.222Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-200160.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-200160.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates

  1209904  [2023-10-07T04:31:29.796Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=Aj_qorT6ybmsBpEp6f7KpZqBcZcgxQh4H8Fzw3bDA6_J&v=1&x=3&a=CAogig_Xta2ouQ2rPnp7_T8dmG9KNcOyPkFG3YbcbUjHc64SaxDy-9TBsDEY8tiww7AxIgEAUgTQ6AqTWgTt6wTaaiVswoU09SUbh1NBQBAkUsPp4krTXFQvtRTyl-G2YIlcmXht5T1UciXuWTqjsDC0zGF2sM3HPl6fsrwONCb-j_lHVY2SEJ21uN_s5RZX&e=1696653388&fl=&r=c5e78d2e-2b0b-4f81-b248-005895e45672-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=vINAoU-_LyAqzQETSQerlF3qU3c
  1209905  [2023-10-07T04:31:29.796Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=Aj_qorT6ybmsBpEp6f7KpZqBcZcgxQh4H8Fzw3bDA6_J&v=1&x=3&a=CAogig_Xta2ouQ2rPnp7_T8dmG9KNcOyPkFG3YbcbUjHc64SaxDy-9TBsDEY8tiww7AxIgEAUgTQ6AqTWgTt6wTaaiVswoU09SUbh1NBQBAkUsPp4krTXFQvtRTyl-G2YIlcmXht5T1UciXuWTqjsDC0zGF2sM3HPl6fsrwONCb-j_lHVY2SEJ21uN_s5RZX&e=1696653388&fl=&r=c5e78d2e-2b0b-4f81-b248-005895e45672-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=vINAoU-_LyAqzQETSQerlF3qU3c
  1209906: [2023-10-07T04:31:29.803Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-957426.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-957426.heic and linking original file Aa_fDRoooKJJY-0nHI30XCM9rfi--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
  1209907: [2023-10-07T04:31:29.803Z] DEBUG NetworkManager: Starting download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=Ag6SX7EDZ0cuWOZtKSKCcuw3Nwoc_yZOOPi4_MXp-V3P&v=1&x=3&a=CAogWN9m6tH321CcC5U4fjkpT_NejD3f4lH8XeVpxRX-PAYSaxCR_NTBsDEYkdmww7AxIgEAUgQ9rfi-WgS9OF6NaiXsnrFbSY7fC16aOJUPHHekKbzb5DN9M9q5gXUEUEoLrkEsIqjnciXPuVx8cc1i_ja3LYq6Vh3L4diGLWGbEJNPQdERIX1YbcfdNpOT&e=1696653388&fl=&r=c5e78d2e-2b0b-4f81-b248-005895e45672-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=6CPhG2ULFoM9H4yz8A0jrOJB0rw

  1209907  [2023-10-07T04:31:30.147Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/AZZWSm6gsVdoSmzLK2OINh6mkmy3AVp_SHXYXgYqepvku1EjoLxEmuFu/${f}?o=AigkmWxH6mmufZTuqAnvZXgOO6jJwdMia5XcC5clZ67K&v=1&x=3&a=CAogDo9N12PvBAZbX_C_ZT1f3iO0s4ivajz3PxbtRBaTyaASaxD8-9TBsDEY_Niww7AxIgEAUgSmkmy3WgREmuFuaiVsVvqaD-7x-VHOK8akI2GVnDLkK2kih1fuob56Kl3qr9A-t6VhciW9MfZIm54vX_23bhsJUnVpnmrV8UDGfRHuTCU-gPQdGoasztLN&e=1696653388&fl=&r=c5e78d2e-2b0b-4f81-b248-005895e45672-1&k=EtBJ5EAWb4kWKl-RiKhb3g&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=DP8yxkj_yTComqYK5FY2OUaKDJE to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AZZWSm6gsVdoSmzLK2OINh6mkmy3-dup-425077.mov
  1209908  [2023-10-07T04:31:30.147Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/AZZWSm6gsVdoSmzLK2OINh6mkmy3AVp_SHXYXgYqepvku1EjoLxEmuFu/${f}?o=AigkmWxH6mmufZTuqAnvZXgOO6jJwdMia5XcC5clZ67K&v=1&x=3&a=CAogDo9N12PvBAZbX_C_ZT1f3iO0s4ivajz3PxbtRBaTyaASaxD8-9TBsDEY_Niww7AxIgEAUgSmkmy3WgREmuFuaiVsVvqaD-7x-VHOK8akI2GVnDLkK2kih1fuob56Kl3qr9A-t6VhciW9MfZIm54vX_23bhsJUnVpnmrV8UDGfRHuTCU-gPQdGoasztLN&e=1696653388&fl=&r=c5e78d2e-2b0b-4f81-b248-005895e45672-1&k=EtBJ5EAWb4kWKl-RiKhb3g&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=DP8yxkj_yTComqYK5FY2OUaKDJE to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AZZWSm6gsVdoSmzLK2OINh6mkmy3-dup-425077.mov
  1209909: [2023-10-07T04:31:30.417Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=Ag6SX7EDZ0cuWOZtKSKCcuw3Nwoc_yZOOPi4_MXp-V3P&v=1&x=3&a=CAogWN9m6tH321CcC5U4fjkpT_NejD3f4lH8XeVpxRX-PAYSaxCR_NTBsDEYkdmww7AxIgEAUgQ9rfi-WgS9OF6NaiXsnrFbSY7fC16aOJUPHHekKbzb5DN9M9q5gXUEUEoLrkEsIqjnciXPuVx8cc1i_ja3LYq6Vh3L4diGLWGbEJNPQdERIX1YbcfdNpOT&e=1696653388&fl=&r=c5e78d2e-2b0b-4f81-b248-005895e45672-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=6CPhG2ULFoM9H4yz8A0jrOJB0rw to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-957426.heic
  1209910: [2023-10-07T04:31:32.448Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=Ag6SX7EDZ0cuWOZtKSKCcuw3Nwoc_yZOOPi4_MXp-V3P&v=1&x=3&a=CAogWN9m6tH321CcC5U4fjkpT_NejD3f4lH8XeVpxRX-PAYSaxCR_NTBsDEYkdmww7AxIgEAUgQ9rfi-WgS9OF6NaiXsnrFbSY7fC16aOJUPHHekKbzb5DN9M9q5gXUEUEoLrkEsIqjnciXPuVx8cc1i_ja3LYq6Vh3L4diGLWGbEJNPQdERIX1YbcfdNpOT&e=1696653388&fl=&r=c5e78d2e-2b0b-4f81-b248-005895e45672-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=6CPhG2ULFoM9H4yz8A0jrOJB0rw

  1209910  [2023-10-07T04:31:32.456Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-685695.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-685695.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
  1209911  [2023-10-07T04:31:32.456Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-685695.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-685695.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates

  1411859  [2023-10-07T05:40:53.920Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AjnCSpKSaU4jIm6AIWuQ4nOC4QSMneeHf6h6pgAcQ6r5&v=1&x=3&a=CAogQ9Viqbr_uAzE18nGYfXvovX96V4XnHSv-adRHAkyHDkSaxCx7tDDsDEYscusxbAxIgEAUgTQ6AqTWgTt6wTaaiV15PBMWF_-5SpbI6QrkYM--17AeyxwRx4fAGC3tq8GuE2hZ3vVciXQUJwu23xmHPuPMFVuLErm8K0j_Xf33Kj1XcU_ttNViIigJdci&e=1696657515&fl=&r=c04d359c-e77c-43d7-b9bb-14cdd13c1fb4-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=_mcDd82l7KOr4mGqkLZj4UXpeuM
  1411860  [2023-10-07T05:40:53.920Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AjnCSpKSaU4jIm6AIWuQ4nOC4QSMneeHf6h6pgAcQ6r5&v=1&x=3&a=CAogQ9Viqbr_uAzE18nGYfXvovX96V4XnHSv-adRHAkyHDkSaxCx7tDDsDEYscusxbAxIgEAUgTQ6AqTWgTt6wTaaiV15PBMWF_-5SpbI6QrkYM--17AeyxwRx4fAGC3tq8GuE2hZ3vVciXQUJwu23xmHPuPMFVuLErm8K0j_Xf33Kj1XcU_ttNViIigJdci&e=1696657515&fl=&r=c04d359c-e77c-43d7-b9bb-14cdd13c1fb4-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=_mcDd82l7KOr4mGqkLZj4UXpeuM
  1411861: [2023-10-07T05:40:53.922Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-719637.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-719637.heic and linking original file Aa_fDRoooKJJY-0nHI30XCM9rfi--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
  1411862: [2023-10-07T05:40:53.922Z] DEBUG NetworkManager: Starting download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AjUBKWS0K8ubRxHQCs7J6k-RleAY4ShCdhmg3DfxxNzh&v=1&x=3&a=CAog9hn3NlCt0Tx6TemAqbyIqLxlPx0mzGkphlbtxH5Mu1wSaxDQ7tDDsDEY0MusxbAxIgEAUgQ9rfi-WgS9OF6NaiXNha-LakU2K_xFa3ca7Bw2ztNAo-kU-KlHvVLgM6_bM0YQLEgNciU4oqHPxSnbeo4iAXJt0da8hqoJa5gchYYYxHjR6HMKH9sh8Ynd&e=1696657515&fl=&r=c04d359c-e77c-43d7-b9bb-14cdd13c1fb4-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=fIm8kCXUUEgun0Xe7u7CRsCJR8s
  1411863: [2023-10-07T05:40:54.332Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AjUBKWS0K8ubRxHQCs7J6k-RleAY4ShCdhmg3DfxxNzh&v=1&x=3&a=CAog9hn3NlCt0Tx6TemAqbyIqLxlPx0mzGkphlbtxH5Mu1wSaxDQ7tDDsDEY0MusxbAxIgEAUgQ9rfi-WgS9OF6NaiXNha-LakU2K_xFa3ca7Bw2ztNAo-kU-KlHvVLgM6_bM0YQLEgNciU4oqHPxSnbeo4iAXJt0da8hqoJa5gchYYYxHjR6HMKH9sh8Ynd&e=1696657515&fl=&r=c04d359c-e77c-43d7-b9bb-14cdd13c1fb4-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=fIm8kCXUUEgun0Xe7u7CRsCJR8s to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-719637.heic
  1411864: [2023-10-07T05:40:54.931Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AjUBKWS0K8ubRxHQCs7J6k-RleAY4ShCdhmg3DfxxNzh&v=1&x=3&a=CAog9hn3NlCt0Tx6TemAqbyIqLxlPx0mzGkphlbtxH5Mu1wSaxDQ7tDDsDEY0MusxbAxIgEAUgQ9rfi-WgS9OF6NaiXNha-LakU2K_xFa3ca7Bw2ztNAo-kU-KlHvVLgM6_bM0YQLEgNciU4oqHPxSnbeo4iAXJt0da8hqoJa5gchYYYxHjR6HMKH9sh8Ynd&e=1696657515&fl=&r=c04d359c-e77c-43d7-b9bb-14cdd13c1fb4-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=fIm8kCXUUEgun0Xe7u7CRsCJR8s

  1411864  [2023-10-07T05:40:54.939Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-350150.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-350150.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
  1411865  [2023-10-07T05:40:54.939Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-350150.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-350150.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates

  1613474  [2023-10-07T06:49:53.150Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AmG9EiV6UOjHIlOG4KQYfrAVCr8WIiBjSzdVCvHmCkyJ&v=1&x=3&a=CAogioM6WWWP25ddANjrTeKT1UrNRJ57lj92TxNBIOkEaOQSaxCooc_FsDEYqP6qx7AxIgEAUgTQ6AqTWgTt6wTaaiXdDNS49jIY93WIMiX-HaWkqS664EbOYrxeYrBtliRMgfW07zr8ciViEs1JnKdMBYR4oBsLocRvxf1s9D78FCJTpxiFjEud5U0NsSvC&e=1696661684&fl=&r=d54c382a-5bb5-433e-a62e-c0e6778f8df7-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=G8xr2kPFJ4Ij1-fVGlVgPE0cmf8
  1613475  [2023-10-07T06:49:53.150Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Ac_ngH2YWYZ_xI_lYwSyGknQ6AqTAch_3To-MHigmis3w5XE-NPt6wTa/${f}?o=AmG9EiV6UOjHIlOG4KQYfrAVCr8WIiBjSzdVCvHmCkyJ&v=1&x=3&a=CAogioM6WWWP25ddANjrTeKT1UrNRJ57lj92TxNBIOkEaOQSaxCooc_FsDEYqP6qx7AxIgEAUgTQ6AqTWgTt6wTaaiXdDNS49jIY93WIMiX-HaWkqS664EbOYrxeYrBtliRMgfW07zr8ciViEs1JnKdMBYR4oBsLocRvxf1s9D78FCJTpxiFjEud5U0NsSvC&e=1696661684&fl=&r=d54c382a-5bb5-433e-a62e-c0e6778f8df7-1&k=BR8ttrz7bHEHuvyO_4jKgQ&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=G8xr2kPFJ4Ij1-fVGlVgPE0cmf8
  1613476: [2023-10-07T06:49:53.155Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-80254.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-80254.heic and linking original file Aa_fDRoooKJJY-0nHI30XCM9rfi--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
  1613477: [2023-10-07T06:49:53.155Z] DEBUG NetworkManager: Starting download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AkZRQdL0g8zBl0Et_8TBYiOlixe7_fbHf67vUnKazhrz&v=1&x=3&a=CAogLOM0-GJEygZWm-mcyLoshfnm6Cjne8xthD4jOtZ2suMSaxDKoc_FsDEYyv6qx7AxIgEAUgQ9rfi-WgS9OF6NaiWyATAJMGzbABkx68_Aj3Ae1mXA-8XmD4ttoBLwFluYBlhZis9ZciXJmX4x6K-NvzxUTZokcAITpwpuhu_KIJgohA5ZcIFl7Ov7-7Uk&e=1696661684&fl=&r=d54c382a-5bb5-433e-a62e-c0e6778f8df7-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=jmbFgI__I-E5_mnYEEuUpnXk53E

  1613477  [2023-10-07T06:49:53.545Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/AZZWSm6gsVdoSmzLK2OINh6mkmy3AVp_SHXYXgYqepvku1EjoLxEmuFu/${f}?o=Auk8petY0mO6II1laczBetIXe9weLCdbkUcqXHqBdJ1X&v=1&x=3&a=CAogcB_UIVsqZ0VHmjX6YEuIdL80660oaiUknfsUwH0Mf8MSaxCzoc_FsDEYs_6qx7AxIgEAUgSmkmy3WgREmuFuaiWBr8XL24vve84WYXzsP81EVxjc_c8LGXOk7Ok74f1ezsjh1pagciXSUO-eaLzmUmfd6uZBWZfI-SS9IPaFrEf8tkXuJq7smwQrYF8N&e=1696661684&fl=&r=d54c382a-5bb5-433e-a62e-c0e6778f8df7-1&k=EtBJ5EAWb4kWKl-RiKhb3g&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=sR3yR6EOYTSgKr9iAQQXWxTlk9s to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AZZWSm6gsVdoSmzLK2OINh6mkmy3-dup-305011.mov
  1613478  [2023-10-07T06:49:53.545Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/AZZWSm6gsVdoSmzLK2OINh6mkmy3AVp_SHXYXgYqepvku1EjoLxEmuFu/${f}?o=Auk8petY0mO6II1laczBetIXe9weLCdbkUcqXHqBdJ1X&v=1&x=3&a=CAogcB_UIVsqZ0VHmjX6YEuIdL80660oaiUknfsUwH0Mf8MSaxCzoc_FsDEYs_6qx7AxIgEAUgSmkmy3WgREmuFuaiWBr8XL24vve84WYXzsP81EVxjc_c8LGXOk7Ok74f1ezsjh1pagciXSUO-eaLzmUmfd6uZBWZfI-SS9IPaFrEf8tkXuJq7smwQrYF8N&e=1696661684&fl=&r=d54c382a-5bb5-433e-a62e-c0e6778f8df7-1&k=EtBJ5EAWb4kWKl-RiKhb3g&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=sR3yR6EOYTSgKr9iAQQXWxTlk9s to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AZZWSm6gsVdoSmzLK2OINh6mkmy3-dup-305011.mov
  1613479: [2023-10-07T06:49:53.584Z] DEBUG NetworkManager: Starting to write https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AkZRQdL0g8zBl0Et_8TBYiOlixe7_fbHf67vUnKazhrz&v=1&x=3&a=CAogLOM0-GJEygZWm-mcyLoshfnm6Cjne8xthD4jOtZ2suMSaxDKoc_FsDEYyv6qx7AxIgEAUgQ9rfi-WgS9OF6NaiWyATAJMGzbABkx68_Aj3Ae1mXA-8XmD4ttoBLwFluYBlhZis9ZciXJmX4x6K-NvzxUTZokcAITpwpuhu_KIJgohA5ZcIFl7Ov7-7Uk&e=1696661684&fl=&r=d54c382a-5bb5-433e-a62e-c0e6778f8df7-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=jmbFgI__I-E5_mnYEEuUpnXk53E to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/Aa_fDRoooKJJY-0nHI30XCM9rfi--dup-80254.heic
  1613480: [2023-10-07T06:49:54.394Z] DEBUG NetworkManager: Finished download of https://cvws.icloud-content.com/B/Aa_fDRoooKJJY-0nHI30XCM9rfi-AeHznfCvqP0c10kik4I4KI69OF6N/${f}?o=AkZRQdL0g8zBl0Et_8TBYiOlixe7_fbHf67vUnKazhrz&v=1&x=3&a=CAogLOM0-GJEygZWm-mcyLoshfnm6Cjne8xthD4jOtZ2suMSaxDKoc_FsDEYyv6qx7AxIgEAUgQ9rfi-WgS9OF6NaiWyATAJMGzbABkx68_Aj3Ae1mXA-8XmD4ttoBLwFluYBlhZis9ZciXJmX4x6K-NvzxUTZokcAITpwpuhu_KIJgohA5ZcIFl7Ov7-7Uk&e=1696661684&fl=&r=d54c382a-5bb5-433e-a62e-c0e6778f8df7-1&k=ImcQTbItcwDlrjnLY3QjLw&ckc=com.apple.photos.cloud&ckz=PrimarySync&y=1&p=58&s=jmbFgI__I-E5_mnYEEuUpnXk53E
  1613480  [2023-10-07T06:49:54.402Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-108783.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-108783.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates
  1613481  [2023-10-07T06:49:54.402Z] WARN NetworkManager: /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-108783.heic already exists - saving duplicate to /Volumes/Extreme SSD/icloud-photos-sync/duplicates/AUJ0QTGAj6TeCZ-NTHIICD-ne4c--dup-108783.heic and linking original file AUJ0QTGAj6TeCZ-NTHIICD-ne4c--orig.heic to /Volumes/Extreme SSD/icloud-photos-sync/duplicates

Let me know if you need any additional information.

steilerDev commented 1 year ago

I think I also made a wrong assumption how chaining promises work - should be fixed now as well.

Is there any other reference to the file in your logs?

Could you also please share how the 'duplicates' folder looks like (ls -al). Are all duplicate files the exact same? maybe you can use diff to compare them down to the byte level?

halbtuerke commented 1 year ago

Is there any other reference to the file in your logs?

No, these are all the occurrences.

Could you also please share how the 'duplicates' folder looks like (ls -al).

Here is the output of ls -la: https://gist.github.com/halbtuerke/ca17c2d2a1915eb951363e8a0712ce70

Are all duplicate files the exact same? maybe you can use diff to compare them down to the byte level?

As requested I have compared the files on a byte level. For that I wrote the following Python script, which determines all the unique IDs and then compares all duplicates of the individual IDs. The result is that almost all of those duplicates are completely the same. 

#!/usr/bin/env python3

import os
from collections import defaultdict
import filecmp

def compare_files_by_prefix(directory_path):
    # Initialize a dictionary to hold file prefixes and corresponding file names
    files_by_prefix = defaultdict(list)

    # Iterate through the directory to populate the files_by_prefix dictionary
    for file_name in os.listdir(directory_path):
        prefix = file_name.split('-orig')[0].split('-dup')[0]
        files_by_prefix[prefix].append(os.path.join(directory_path, file_name))

    # Initialize a dictionary to hold comparison results
    comparison_results = {}

    # Compare files by prefix
    for prefix, file_list in files_by_prefix.items():
        print(prefix)
        # Skip if there's only one file with this prefix
        if len(file_list) <= 1:
            comparison_results[prefix] = "Only one file with this prefix."
            continue

        # Compare the files on a byte level
        are_identical = all(filecmp.cmp(file_list[0], other_file, shallow=False) for other_file in file_list[1:])

        comparison_results[prefix] = "Identical" if are_identical else "Different"

    return comparison_results

def print_comparison_results(results):
    for prefix, status in results.items():
        print(f"Files with prefix '{prefix}': {status}")

# For demonstration purposes, let's assume a directory path.
# Note: This is a placeholder, and the actual directory path should be provided.
directory_path = "/Volumes/Extreme SSD/icloud-photos-sync/duplicates"

# Get the comparison results
results = compare_files_by_prefix(directory_path)

# Print the results
print_comparison_results(results)

There are only 2 cases where there are actual differences, but only with single duplicates:

Files with prefix 'AY5TDE3mVEXwtU1Oyo2RYM1xV07g': Different
Files with prefix 'AW0tYkIVqOpECWEBGzW4hZELQRGV': Different
Files with prefix 'AcfheUEZsdw7922tT4y2C7y6D4T0': Only one file with this prefix.
Files with prefix 'AUfNRVZbCaJmzXgX_5U4CBX7cQTG': Only one file with this prefix.

CleanShot 2023-10-09 at 22 54 47@2x CleanShot 2023-10-09 at 22 54 04@2x

In both cases it seems like those files are broken and cannot be opened.

So in general it seems like we are mostly downloading legit duplicates.

steilerDev commented 1 year ago

Okay - so these are legit duplicates - and I have no idea why the diffing algorithm does not catch them.

In 1.3.0-nightly.10 I've reverted the debug logic and it should now skip already existing files. This should unblock your sync executions and I will need to address this properly with the planned improvements of the epic in #354.

steilerDev commented 1 year ago

@halbtuerke did you succeed syncing your assets with the workaround introduced in the latest version?

halbtuerke commented 1 year ago

Yes, the initial synchronization was successful. However, I encounter issues when attempting to re-synchronize for updating photos. Below are the error messages displayed:

Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ICLOUD_PHOTOS_FETCH_RECORDS: Unable to fetch records (album 'All photos') caused by Request failed with status code 408
Refreshing iCloud connection & retrying (attempt #35)...

I am using version 1.3.0.

steilerDev commented 1 year ago

Is this the 'full' log? Meaning: does it go straight to attempt 35?

halbtuerke commented 1 year ago

No, there were 34 previous attempts. I just terminate after 35 attempts. 🥲

steilerDev commented 1 year ago

And what's the timing of those 408's? could you set retryCount to 1, enable debugging logs, re-run and share the logs and output here?

halbtuerke commented 1 year ago

Command used:

FORCE=true LOG_LEVEL=debug icloud-photos-sync \
-u USER \
-p PASSWORD \
-d "/Volumes/Extreme SSD/icloud-photos-sync" \
--enable-crash-reporting \
--metadata-rate '1/20' \
--max-retries 1 \
--export-metrics \
sync;

Output of the run (started at 7:05 PM):

Welcome to icloud-photos-sync, v.1.3.0!
Made with <3 by steilerDev
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Starting sync at 10/30/2023, 7:07:40 PM
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...

Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ICLOUD_PHOTOS_FETCH_RECORDS: Unable to fetch records (album 'All photos') caused by Request failed with status code 408
Refreshing iCloud connection & retrying (attempt #2)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
Error: APP_SYNC: Sync failed caused by SYNC_MAX_RETRY: Sync did not complete successfully within expected amount of tries (2) (error code: 3bc11b53-5f84-44e3-81b1-a7ce4178ddfc)

You can find the logs at https://gist.github.com/halbtuerke/406cced473934ba7684c1d7bfe4269e1