search

mandarons / icloud-drive-docker

Dockerized iCloud Client - make a local copy of your iCloud documents and photos, and keep it automatically up-to-date.
BSD 3-Clause "New" or "Revised" License
962 stars 46 forks source link

[BUG] Not syncing partial anymore with Error 500 because of special characters in filenames #198

Open nicx opened 4 months ago

nicx commented 4 months ago

I am using iCloud-drive-docker for some months without problems. Suddenly one of my instances ar no more syncing all files/photos. In the Log I see the error

2024-02-29 07:57:13,333 :: ERROR :: icloudpy.base :: base.py :: 186 :: Authentication required for Account. (500)

I did a full reauthentication after deleting the session_data, but the error is still there after about 15 minutes of working (and syncing drive files) correctly. The newest photos are missing.

I do not have this problem with my other instances of iCloud-drive-docker running.

What can I do to get more information about the reason for this problem?

nicx commented 4 months ago

@mandarons here are additional DEBUG logs regarding this error:

2024-03-01 08:20:02,055 :: DEBUG :: requests.sessions.http :: base.py :: 75 :: GET https://cvws.icloud-content.com/B/AeC9tiYgopF7MT09KhleZegy7lMNAUtr4s0aTNFbcqsPF87oiQLDybxm/2015_07_Geh.-Abr.+Juli+2015+mit+Geh.-Erho%CC%88+3%2C0+%25+u.+RR+04-06.pdf?o=AibojzXG2BT40soVqvOup0Uu62qjhKCksjWxnGeLZ3sV&v=1&x=3&a=CAogFBzYyVbCSDiVJSyCXAT1RvnILTgT1EpDdaT__ia0qo4SbRD5vonH3zEY-ZvlyN8xIgEAUgQy7lMNWgTDybxmaibLUHdsppyqoGez7oAnLA4A1jvD0DVkf0y2gKpGZVqxcpO0PyGdsnImY4EDhM6WY1UC8R12xrHeBXkg8GtFZ_pGSFP2hohSpmJxT4vPlgE&e=1709281201&fl=&r=5dd7feb5-4d2c-441f-b05f-5606b1487eac-1&k=DPiw8lui3ZSLmUV0ZGnjUw&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=F5AeVxJbrpDUVCqQMrbw813nroQ 
2024-03-01 08:20:02,056 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 1019 :: Starting new HTTPS connection (28): cvws.icloud-content.com:443
2024-03-01 08:20:02,284 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 474 :: https://cvws.icloud-content.com:443 "GET /B/AeC9tiYgopF7MT09KhleZegy7lMNAUtr4s0aTNFbcqsPF87oiQLDybxm/2015_07_Geh.-Abr.+Juli+2015+mit+Geh.-Erho%CC%88+3%2C0+%25+u.+RR+04-06.pdf?o=AibojzXG2BT40soVqvOup0Uu62qjhKCksjWxnGeLZ3sV&v=1&x=3&a=CAogFBzYyVbCSDiVJSyCXAT1RvnILTgT1EpDdaT__ia0qo4SbRD5vonH3zEY-ZvlyN8xIgEAUgQy7lMNWgTDybxmaibLUHdsppyqoGez7oAnLA4A1jvD0DVkf0y2gKpGZVqxcpO0PyGdsnImY4EDhM6WY1UC8R12xrHeBXkg8GtFZ_pGSFP2hohSpmJxT4vPlgE&e=1709281201&fl=&r=5dd7feb5-4d2c-441f-b05f-5606b1487eac-1&k=DPiw8lui3ZSLmUV0ZGnjUw&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=F5AeVxJbrpDUVCqQMrbw813nroQ HTTP/1.1" 500 0
2024-03-01 08:20:02,285 :: DEBUG :: icloudpy.base :: base.py :: 94 :: Saved session data to file
2024-03-01 08:20:02,285 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/asdfasdfasdfde
2024-03-01 08:20:02,285 :: ERROR :: icloudpy.base :: base.py :: 186 :: Authentication required for Account. (500)

or here:

2024-03-01 08:20:01,734 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/asdfasdfasdfde
2024-03-01 08:20:01,734 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 142 :: {'document_id': '7F18DA79-269B-46DF-9EDC-FE026210582E', 'item_id': 'CJC_vaYFEAAiEH8Y2nkmm0bfntz-AmIQWC4', 'owner_dsid': 1422876560, 'data_token': {'url': 'https://cvws.icloud-content.com/B/AeC9tiYgopF7MT09KhleZegy7lMNAUtr4s0aTNFbcqsPF87oiQLDybxm/2015_07_Geh.-Abr.+Juli+2015+mit+Geh.-Erho%CC%88+3%2C0+%25+u.+RR+04-06.pdf?o=AibojzXG2BT40soVqvOup0Uu62qjhKCksjWxnGeLZ3sV&v=1&x=3&a=CAogFBzYyVbCSDiVJSyCXAT1RvnILTgT1EpDdaT__ia0qo4SbRD5vonH3zEY-ZvlyN8xIgEAUgQy7lMNWgTDybxmaibLUHdsppyqoGez7oAnLA4A1jvD0DVkf0y2gKpGZVqxcpO0PyGdsnImY4EDhM6WY1UC8R12xrHeBXkg8GtFZ_pGSFP2hohSpmJxT4vPlgE&e=1709281201&fl=&r=5dd7feb5-4d2c-441f-b05f-5606b1487eac-1&k=DPiw8lui3ZSLmUV0ZGnjUw&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=F5AeVxJbrpDUVCqQMrbw813nroQ', 'token': 'CAogNcfvUh3AUwM8cKCh+3CdRZllEluFV0Tnb3FQ/uVI+fwSbRD5vonH3zEY+ZvlyN8xIgEAUgQy7lMNWgTDybxmaibvSSSLUgcpSycOstcR8a5G3dNhyIxHUw8kRUQAn5K/bJDJ1Au/enImHK549QniWyhNPe9z4cNZEiJhIgXaji51j3Qoskl9w7go7gYch6c=', 'signature': 'AeC9tiYgopF7MT09KhleZegy7lMN', 'wrapping_key': 'DPiw8lui3ZSLmUV0ZGnjUw==', 'reference_signature': 'AUtr4s0aTNFbcqsPF87oiQLDybxm'}, 'double_etag': 'byg::byf'}
2024-03-01 08:20:01,735 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 75 :: GET https://cvws.icloud-content.com/B/AeC9tiYgopF7MT09KhleZegy7lMNAUtr4s0aTNFbcqsPF87oiQLDybxm/2015_07_Geh.-Abr.+Juli+2015+mit+Geh.-Erho%CC%88+3%2C0+%25+u.+RR+04-06.pdf?o=AibojzXG2BT40soVqvOup0Uu62qjhKCksjWxnGeLZ3sV&v=1&x=3&a=CAogFBzYyVbCSDiVJSyCXAT1RvnILTgT1EpDdaT__ia0qo4SbRD5vonH3zEY-ZvlyN8xIgEAUgQy7lMNWgTDybxmaibLUHdsppyqoGez7oAnLA4A1jvD0DVkf0y2gKpGZVqxcpO0PyGdsnImY4EDhM6WY1UC8R12xrHeBXkg8GtFZ_pGSFP2hohSpmJxT4vPlgE&e=1709281201&fl=&r=5dd7feb5-4d2c-441f-b05f-5606b1487eac-1&k=DPiw8lui3ZSLmUV0ZGnjUw&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=F5AeVxJbrpDUVCqQMrbw813nroQ 
2024-03-01 08:20:01,736 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 291 :: Resetting dropped connection: cvws.icloud-content.com
2024-03-01 08:20:02,053 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 474 :: https://cvws.icloud-content.com:443 "GET /B/AeC9tiYgopF7MT09KhleZegy7lMNAUtr4s0aTNFbcqsPF87oiQLDybxm/2015_07_Geh.-Abr.+Juli+2015+mit+Geh.-Erho%CC%88+3%2C0+%25+u.+RR+04-06.pdf?o=AibojzXG2BT40soVqvOup0Uu62qjhKCksjWxnGeLZ3sV&v=1&x=3&a=CAogFBzYyVbCSDiVJSyCXAT1RvnILTgT1EpDdaT__ia0qo4SbRD5vonH3zEY-ZvlyN8xIgEAUgQy7lMNWgTDybxmaibLUHdsppyqoGez7oAnLA4A1jvD0DVkf0y2gKpGZVqxcpO0PyGdsnImY4EDhM6WY1UC8R12xrHeBXkg8GtFZ_pGSFP2hohSpmJxT4vPlgE&e=1709281201&fl=&r=5dd7feb5-4d2c-441f-b05f-5606b1487eac-1&k=DPiw8lui3ZSLmUV0ZGnjUw&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=F5AeVxJbrpDUVCqQMrbw813nroQ HTTP/1.1" 500 0
2024-03-01 08:20:02,053 :: DEBUG :: icloudpy.base :: base.py :: 94 :: Saved session data to file
2024-03-01 08:20:02,054 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/asdfasdfasdfde
2024-03-01 08:20:02,054 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 127 :: Internal Server Error (500). Retrying ...
2024-03-01 08:20:02,055 :: DEBUG :: requests.sessions.http :: base.py :: 75 :: GET https://cvws.icloud-content.com/B/AeC9tiYgopF7MT09KhleZegy7lMNAUtr4s0aTNFbcqsPF87oiQLDybxm/2015_07_Geh.-Abr.+Juli+2015+mit+Geh.-Erho%CC%88+3%2C0+%25+u.+RR+04-06.pdf?o=AibojzXG2BT40soVqvOup0Uu62qjhKCksjWxnGeLZ3sV&v=1&x=3&a=CAogFBzYyVbCSDiVJSyCXAT1RvnILTgT1EpDdaT__ia0qo4SbRD5vonH3zEY-ZvlyN8xIgEAUgQy7lMNWgTDybxmaibLUHdsppyqoGez7oAnLA4A1jvD0DVkf0y2gKpGZVqxcpO0PyGdsnImY4EDhM6WY1UC8R12xrHeBXkg8GtFZ_pGSFP2hohSpmJxT4vPlgE&e=1709281201&fl=&r=5dd7feb5-4d2c-441f-b05f-5606b1487eac-1&k=DPiw8lui3ZSLmUV0ZGnjUw&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=F5AeVxJbrpDUVCqQMrbw813nroQ 
2024-03-01 08:20:02,056 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 1019 :: Starting new HTTPS connection (28): cvws.icloud-content.com:443
2024-03-01 08:20:02,284 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 474 :: https://cvws.icloud-content.com:443 "GET /B/AeC9tiYgopF7MT09KhleZegy7lMNAUtr4s0aTNFbcqsPF87oiQLDybxm/2015_07_Geh.-Abr.+Juli+2015+mit+Geh.-Erho%CC%88+3%2C0+%25+u.+RR+04-06.pdf?o=AibojzXG2BT40soVqvOup0Uu62qjhKCksjWxnGeLZ3sV&v=1&x=3&a=CAogFBzYyVbCSDiVJSyCXAT1RvnILTgT1EpDdaT__ia0qo4SbRD5vonH3zEY-ZvlyN8xIgEAUgQy7lMNWgTDybxmaibLUHdsppyqoGez7oAnLA4A1jvD0DVkf0y2gKpGZVqxcpO0PyGdsnImY4EDhM6WY1UC8R12xrHeBXkg8GtFZ_pGSFP2hohSpmJxT4vPlgE&e=1709281201&fl=&r=5dd7feb5-4d2c-441f-b05f-5606b1487eac-1&k=DPiw8lui3ZSLmUV0ZGnjUw&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=F5AeVxJbrpDUVCqQMrbw813nroQ HTTP/1.1" 500 0
2024-03-01 08:20:02,285 :: DEBUG :: icloudpy.base :: base.py :: 94 :: Saved session data to file
2024-03-01 08:20:02,285 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/asdfasdfasdfde
2024-03-01 08:20:02,285 :: ERROR :: icloudpy.base :: base.py :: 186 :: Authentication required for Account. (500)
nicx commented 4 months ago

@mandarons any hint?

mandarons commented 3 months ago

The newest photos are missing.

How are these different from older ones? Part of shared album/library?

I do not have this problem with my other instances of iCloud-drive-docker running.

Can you please do detailed comparison (config.yaml, logs, docker command/compose file etc.)? If everything looks exactly the same, I am guessing server isn't allowing to use multiple new logins from same IP (your home external IP).

nicx commented 3 months ago

@mandarons

The newest photos are missing.

How are these different from older ones? Part of shared album/library?

I did some more research and it seems it's not only photo related... the sync just interrupts and ether is a lot of different data missing. BTW I am not using shared albums/libraries so far. And if you look at my previous log file I posted already, you can see the error is regarding a file on iCloud Drive, not a photo ;)

I do not have this problem with my other instances of iCloud-drive-docker running.

Can you please do detailed comparison (config.yaml, logs, docker command/compose file etc.)? If everything looks exactly the same, I am guessing server isn't allowing to use multiple new logins from same IP (your home external IP).

Currently I have 2 instances running for 2 different iCloud users. Everything I exactly the same besides username and password. The only difference is that the problematic account has a lot of data to sync.

Could it be a problem of the filename of the file which is synced? Could it be a problem of the amount of data to be synced? What else could I do to find out the reason?

nicx commented 3 months ago

@mandarons I did some more tests:

I have removed a suspicious folder from iCloud that contains a lot of files with special characters in the file name. The synchronization now works completely without any errors.

However, since the files that have now been removed are synchronized directly via Apple iCloud (from my Mac to Apple iCloud) without any problems, I would say that there is a bug in iCloud-drive-docker or in the icloudpy library. The affected file names can be seen in the log file above.

Could you please have a look? Thanks!

nicx commented 3 months ago

@mandarons are you alive? :) Could I do anything to help you?

mandarons commented 3 months ago

@nicx yes, I am alive - busy with my day job. :(

I have removed a suspicious folder from iCloud that contains a lot of files with special characters in the file name.

I think, this is it. Thanks for helping to narrow it down. Yes - it's a bug somewhere (icloudpy/icloud-drive-docker).

Can you share equivalent filename that you see on your disk for 2015_07_Geh.-Abr.+Juli+2015+mit+Geh.-Erho%CC%88+3%2C0+%25+u.+RR+04-06.pdf?

nicx commented 3 months ago

@mandarons the equivalent filename on my disk is "2015_07_Geh.-Abr. Juli 2015 mit Geh.-Erhö 3,0 % u. RR 04-06.pdf"

Thanks for fixing that! Meanwhile I will try to "detox" all my filenames. ;)

nicx commented 2 months ago

@mandarons any news to this bug?

nicx commented 2 weeks ago

@mandarons still no news? ;)

nicx commented 6 days ago

i dint get my file names all fixed. There are still special cases in file naming which are leading to this Error 500. @mandarons Would be really great if this could be fixed in a near future :)

mandarons commented 4 days ago

Yes, I will eventually get to it. Reproducing this issue on my end has been challenging.