search

hschauhan / gosync

An Opensource Google Drive client written in Python
https://hschauhan.github.io/gosync/
GNU General Public License v2.0
281 stars 57 forks source link

Files over 4GB are not correctly synced #38

Open Fabian42 opened 4 years ago

Fabian42 commented 4 years ago

I have a folder with two copies of a 4.4GB (4669979776B) file. One of those copies got downloaded by GoSync as a 1000MB (1048576010B=1024²×1000B+10B) file, one as a 2.5GB (2726297626B=???) file. The copy in the cloud was not damaged.

hschauhan commented 4 years ago

This is because the download sometimes fails. The idea is to retry. Alain is working on adding exponential backoff algorithm for retries. Should be fixed soon.

Fabian42 commented 4 years ago

The download failed after exactly 1000MB+10B? That seems to be too exact of a number to be a concidence.

arentoine commented 4 years ago

Hello,

a fix was added to GoSync to handle large file download.

Can you update GoSync with the current version and share your comments ?

Regards

Fabian42 commented 4 years ago

Not worth its own report, but: "Debuging"→"Debugging"

arentoine commented 4 years ago

Hello,

does it work ? :-)

If not, do you have log ?

Thanks

hschauhan commented 4 years ago

Not worth its own report, but: "Debuging"→"Debugging"

Ouch!

Fabian42 commented 4 years ago

I was testing it and doing other stuff at the same time, so I didn't notice that I had to click "resume sync". Now I did and it instantly ends again:

2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Trying to acquire lock.
2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Lock acquired.
2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Validating sync settings
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - validated
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - Staring the sync now
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - Start - Syncing remote directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - End - Syncing remote directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################

2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - Start - Syncing local directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - ### SyncLocalDirectory: - Sync Started
2020-05-26 20:44:05,702 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - Sync done

10/10 error message right there.

arentoine commented 4 years ago

Hello,

issue is in the upload section. That definitely need some fixing.

Did you test large file download ? From GoogleDrive to Local ?

Thanks for you comments.

Fabian42 commented 4 years ago

I just starting syncing. Since large files didn't work before, that would have been the only thing to sync for it. But it didn't even start syncing properly.

On Tue, 26 May 2020 at 20:50, arentoine notifications@github.com wrote:

Hello,

issue is in the upload section. That definitely need some fixing.

Did you test large file download ? From GoogleDrive to Local ?

Thanks for you comments.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hschauhan/gosync/issues/38#issuecomment-634209869, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACP66YXVSBD5E6AFDWTDIY3RTQFOFANCNFSM4NGLNEEQ .

arentoine commented 4 years ago

Do you know if you have a "half downloaded file" sitting in your local drive ?

Fabian42 commented 4 years ago

I do, the big ones.

On Tue, 26 May 2020 at 20:57, arentoine notifications@github.com wrote:

Do you know if you have a "half downloaded file" sitting in your local drive ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hschauhan/gosync/issues/38#issuecomment-634213375, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACP66YWGCIUKYH7YM42BQUTRTQGHVANCNFSM4NGLNEEQ .

arentoine commented 4 years ago

Can you move the incomplete files in a safe place out of the sync directory. Then if you could update your tree with the most uptodate version. Then tell us if this will help you ?

Regards

Fabian42 commented 4 years ago

It seems to have actually uploaded the small, broken version of one of the big files again, duplicating it on Drive. It did not do that for any other one.

I did what you said, moved the broken files somewhere else and tried syncing again. This is what happened last:

2020-05-27 15:22:56,384 - GoSync - DEBUG - Checking: file.mp4

2020-05-27 15:22:56,385 - GoSync - DEBUG - SyncRemoteDirectory: Checking file (file.mp4)
2020-05-27 15:22:56,392 - GoSync - DEBUG - DownloadFileByObject: Download Started - File (/home/fabian/Google Drive/fabianroeling@googlemail.com/path/to/file.mp4)
2020-05-27 15:22:56,392 - GoSync - DEBUG - DownloadFileByObject: Download Started - File size (1301474114)
2020-05-27 15:22:56,517 - GoSync - DEBUG - Observer: /home/fabian/Google Drive/fabianroeling@googlemail.com/path/to/file.mp4 created

2020-05-27 15:22:56,522 - GoSync - DEBUG - UploadObservedFile: File /home/fabian/Google Drive/fabianroeling@googlemail.com/path/to/file.mp4 is created but sync is running. Possibly created by sync. Skipping.
2020-05-27 15:26:24,527 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (file.mp4)
2020-05-27 15:26:24,528 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (to)
2020-05-27 15:26:24,528 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (path)
2020-05-27 15:26:24,529 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-27 15:26:24,529 - GoSync - INFO - SyncThread - run - Sync done
2020-05-27 15:26:54,570 - GoSync - DEBUG - SyncThread - run - Waiting for Sync to be enabled
2020-05-27 15:27:14,716 - GoSync - INFO - SyncThread - run - Internet is down. Clearing running.

Now it's stuck at "Network is down" and nothing happens anymore.

arentoine commented 4 years ago

I will prepare a special python module to gather more information about the issue.

I'll get back to you in a couple of days ...

Greetings

Fabian42 commented 4 years ago

Just a quick reminder that I only test this because it's fun and helps you, I don't really need the program. If nobody else has my issues, you don't have to spend all that time.

hschauhan commented 4 years ago

Hi Fabian,

Your testing is quite important to this project. It has uncovered lot of problems and made the project quite robust as compared to previous versions. Please keep testing if you can.

Regards Himanshu

On Thu, 28 May 2020 at 1:43 PM, Fabian Röling notifications@github.com wrote:

Just a quick reminder that I only test this because it's fun and helps you, I don't really need the program. If nobody else has my issues, you don't have to spend all that time.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/hschauhan/gosync/issues/38#issuecomment-635189552, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXIY53IDXSB3HKYKQ4IKLRTYMJBANCNFSM4NGLNEEQ .

arentoine commented 4 years ago

Exactly !

Keep having fun testing as much as we do resolving bugs.

And may be, someone will benefit from that !

Thanks again!

hschauhan commented 4 years ago

I found the problem, a silly one. Will commit tomorrow.

On Wed, 27 May 2020 at 12:20 AM, arentoine notifications@github.com wrote:

Hello,

issue is in the upload section. That definitely need some fixing.

Did you test large file download ? From GoogleDrive to Local ?

Thanks for you comments.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/hschauhan/gosync/issues/38#issuecomment-634209869, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXIY4AIZBTE4ZNZT7SNN3RTQFOFANCNFSM4NGLNEEQ .

hschauhan commented 4 years ago

I have committed the changes and local sync is working fine.

Fabian42 commented 4 years ago 
2020-05-30 12:12:36,500 - GoSync - DEBUG - Checking: file.iso

2020-05-30 12:12:36,501 - GoSync - DEBUG - SyncRemoteDirectory: Checking file (file.iso)
2020-05-30 12:12:36,506 - GoSync - DEBUG - DownloadFileByObject: Download Started - File (/home/fabian/Google Drive/fabianroeling@googlemail.com/path/to/file.iso)
2020-05-30 12:12:36,507 - GoSync - DEBUG - DownloadFileByObject: Download Started - File size (4699979776)
2020-05-30 12:12:36,576 - GoSync - DEBUG - Observer: /home/fabian/Google Drive/fabianroeling@googlemail.com/path/to/file.iso created

2020-05-30 12:12:36,586 - GoSync - DEBUG - UploadObservedFile: File /home/fabian/Google Drive/fabianroeling@googlemail.com/path/to/file.iso is created but sync is running. Possibly created by sync. Skipping.
2020-05-30 12:14:57,565 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (file.iso)
2020-05-30 12:14:57,565 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (to)
2020-05-30 12:14:57,566 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (path)
2020-05-30 12:14:57,566 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-30 12:14:57,566 - GoSync - INFO - SyncThread - run - Sync done
2020-05-30 12:15:27,600 - GoSync - DEBUG - SyncThread - run - Waiting for Sync to be enabled
2020-05-30 12:15:47,744 - GoSync - INFO - SyncThread - run - Internet is down. Clearing running.
2020-05-30 14:41:00,783 - GoSync - INFO - calculateUsage - GoSync is shutting down!

I just tried the "exit" menu option for the first time and after confirming, both the confirmation dialogue and the program froze and needed to be forcibly shut down.