search

cjnaz / rclonesync-V2

A Bidirectional Cloud Sync Utility using rclone
MIT License
355 stars 39 forks source link

file error when there is no internet #56

Open luislozoya opened 4 years ago

luislozoya commented 4 years ago

When coping from google drive to the computer if the internet if off it will retry 3 times, then it will report a fail, but after that when i checked the folder of the copied files i notice one with 0 size then when running first sync again it skips the file and even worse as it does a sync after that it overwrites the correct file in google drive with the 0 size file

suggestion fix is to: -remove the failed file

cjnaz commented 4 years ago

Ouch. I agree, it shouldn't do that. Would you please post a verbose log for when the 0 size file is created? This will help a lot in isolating the problem and fix.

luislozoya commented 4 years ago

hi here is the log

2020/08/04 16:00:58 INFO : Transferred: 70.163M / 417.356 MBytes, 17%, 27.217 kBytes/s, ETA 3h37m42s Transferred: 0 / 1, 0% Elapsed time: 43m59.8s Transferring:

2020/08/04 16:01:58 INFO : Transferred: 70.163M / 417.356 MBytes, 17%, 26.611 kBytes/s, ETA 3h42m39s Transferred: 0 / 1, 0% Elapsed time: 44m59.8s Transferring:

2020/08/04 16:02:58 INFO : Transferred: 70.163M / 417.356 MBytes, 17%, 26.033 kBytes/s, ETA 3h47m36s Transferred: 0 / 1, 0% Elapsed time: 45m59.8s Transferring:

2020/08/04 16:03:58 INFO : Transferred: 70.163M / 417.356 MBytes, 17%, 25.479 kBytes/s, ETA 3h52m33s Transferred: 0 / 1, 0% Elapsed time: 46m59.8s Transferring:

2020/08/04 16:04:58 INFO : Transferred: 70.163M / 417.356 MBytes, 17%, 24.948 kBytes/s, ETA 3h57m30s Transferred: 0 / 1, 0% Elapsed time: 47m59.8s Transferring:

2020/08/04 16:05:58 INFO : Transferred: 70.163M / 417.356 MBytes, 17%, 24.439 kBytes/s, ETA 4h2m27s Transferred: 0 / 1, 0% Elapsed time: 48m59.8s Transferring:

2020/08/04 16:06:00 ERROR : TT T01-08.mkv: Failed to copy: multpart copy: failed to open source: open file failed: Get https://www.googleapis.com/drive/v3/files/1BEbDzqatVHKjWEIorIGYzA3GxTUq3bjW?alt=media: dial tcp: lookup www.googleapis.com on 10.255.255.2:53: write udp 192.168.0.106:42540->10.255.255.2:53: write: operation not permitted 2020/08/04 16:06:00 ERROR : Attempt 1/3 failed with 1 errors and: multpart copy: failed to open source: open file failed: Get https://www.googleapis.com/drive/v3/files/1BEbDzqatVHKjWEIorIGYzA3GxTUq3bjW?alt=media: dial tcp: lookup www.googleapis.com on 10.255.255.2:53: write udp 192.168.0.106:42540->10.255.255.2:53: write: operation not permitted 2020/08/04 16:06:58 INFO : Transferred: 70.163M / 70.163 MBytes, 100%, 24.420 kBytes/s, ETA 0s Elapsed time: 49m2.1s

2020/08/04 16:07:39 ERROR : Attempt 2/3 failed with 1 errors and: couldn't list directory: Get https://www.googleapis.com/drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%2715KW9nKs3kbWe1J23RxGsmr5cr0OU3FqZ%27+in+parents%29+and+%28name%3D%27TT+T01-08.mkv%27%29+and+mimeType%21%3D%27application%2Fvnd.google-apps.folder%27&supportsAllDrives=true: dial tcp: lookup www.googleapis.com on 10.255.255.2:53: write udp 192.168.0.106:60725->10.255.255.2:53: write: operation not permitted 2020/08/04 16:07:58 INFO : Transferred: 70.163M / 70.163 MBytes, 100%, 24.420 kBytes/s, ETA 0s Elapsed time: 49m2.1s

2020/08/04 16:08:58 INFO : Transferred: 70.163M / 70.163 MBytes, 100%, 24.420 kBytes/s, ETA 0s Elapsed time: 49m2.1s

2020/08/04 16:09:58 INFO : Transferred: 70.163M / 70.163 MBytes, 100%, 24.420 kBytes/s, ETA 0s Elapsed time: 49m2.1s

2020/08/04 16:10:25 ERROR : Attempt 3/3 failed with 1 errors and: couldn't list directory: Get https://www.googleapis.com/drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%2715KW9nKs3kbWe1J23RxGsmr5cr0OU3FqZ%27+in+parents%29+and+%28name%3D%27TT+T01-08.mkv%27%29+and+mimeType%21%3D%27application%2Fvnd.google-apps.folder%27&supportsAllDrives=true: dial tcp: lookup www.googleapis.com on 10.255.255.2:53: write udp 192.168.0.106:48458->10.255.255.2:53: write: operation not permitted 2020/08/04 16:10:25 Failed to copyto: couldn't list directory: Get https://www.googleapis.com/drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%2715KW9nKs3kbWe1J23RxGsmr5cr0OU3FqZ%27+in+parents%29+and+%28name%3D%27TT+T01-08.mkv%27%29+and+mimeType%21%3D%27application%2Fvnd.google-apps.folder%27&supportsAllDrives=true: dial tcp: lookup www.googleapis.com on 10.255.255.2:53: write udp 192.168.0.106:48458->10.255.255.2:53: write: operation not permitted 2020/08/04 16:14:33 Source doesn't exist or is a directory and destination is a file 2020/08/04 16:18:45 Source doesn't exist or is a directory and destination is a file 2020/08/04 16:23:13 Source doesn't exist or is a directory and destination is a file

cjnaz commented 4 years ago

I've posted a question in the rclone forum RE temporary file support during copies/syncs. I understand that some remotes don't support server-side renames, so the temp file method can't work in all cases. I'm considering a minor revamp of rclonesync to use filtering for bulk copy operations, which make it messier for rclonesync to initially push a file with a temp name and then rename it after successful transfer. (Thinking out loud)

cjnaz commented 4 years ago

As noted in rclone issue #4513, this is a known issue with rclone. rclonesync is not the best place to fix this up. I'll leave this issue open for now. Perhaps it'll get fixed on rclone sometime soon.

cjnaz commented 4 years ago

Check out V3.0, just posted. This version aborts nicely when a run is done while there is no internet. Running again should work nicely. The core problem of this thread is a corrupted file on local/ftp/sftp remotes if the internet dies or the rclonesync or rclone process dies in the middle of a file copy. This is a problem for rclone to solve, as there is no way for rclonesync to gracefully and reliably deal with it. I'll leave this issue open.

luislozoya commented 4 years ago

Check out V3.0, just posted. This version aborts nicely when a run is done while there is no internet. Running again should work nicely. The core problem of this thread is a corrupted file on local/ftp/sftp remotes if the internet dies or the rclonesync or rclone process dies in the middle of a file copy. This is a problem for rclone to solve, as there is no way for rclonesync to gracefully and reliably deal with it. I'll leave this issue open.

cool thanks ill check it out.