Closed Fabian42 closed 4 years ago
hschauhan
commented
4 years ago Is this 0.5 released version or you are running it directly from source code checkout of github? Also, the error "httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com" suggests that temporarily the network was down. Can you check again?
Fabian42
commented
4 years ago Yes, as I said, the network very likely was down for a short time. But that shouldn't completely destroy GoSync forever.
I installed GoSync from the readme, which included first installing it with pip, but then in the setup I got directed to a wiki page that told me to re-download and compile it. I don't know why that would be necessary, but it worked and the program successfully started downloading ~10GB of my files, before failing in the way described above.
hschauhan
commented
4 years ago You see the same issue after restarting?
hschauhan
commented
4 years ago I found the problem. Expect the fix in an hour or two. Thanks for reporting the issue!
hschauhan
commented
4 years ago I have committed the fix. Please try with latest code.
Fabian42
commented
4 years ago OK, this failure to initialise seems to be fixed, but not the fact that it hangs forever after an issue.
I got this error:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/home/fabian/.local/lib/python3.8/site-packages/watchdog/observers/api.py", line 196, in run
self.dispatch_events(self.event_queue, self.timeout)
File "/home/fabian/.local/lib/python3.8/site-packages/watchdog/observers/api.py", line 369, in dispatch_events
handler.dispatch(event)
File "/home/fabian/.local/lib/python3.8/site-packages/watchdog/events.py", line 452, in dispatch
super(PatternMatchingEventHandler, self).dispatch(event)
File "/home/fabian/.local/lib/python3.8/site-packages/watchdog/events.py", line 331, in dispatch
{
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 1056, in on_created
self.sync_handler.UploadObservedFile(evt.src_path)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 440, in UploadObservedFile
self.UploadFile(file_path)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 426, in UploadFile
self.CreateRegularFile(file_path, f['id'], newfile)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 396, in CreateRegularFile
upfile = self.drive.files().create(body=file_metadata,
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 871, in execute
_, body = self.next_chunk(http=http, num_retries=num_retries)
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 1054, in next_chunk
return self._process_response(resp, content)
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 1085, in _process_response
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v3/files?fields=id&alt=json&uploadType=resumable returned "Bad Request">And then it was stuck at "starting in 00:01" again. Killing the program and starting it again "fixed" it.
hschauhan
commented
4 years ago Hi, Can you please provide GoSync.log? It is created in your home directory.
hschauhan
commented
4 years ago Is there a way I can reproduce this?
Fabian42
commented
4 years ago You can probably reproduce this by disconnecting from the internet, blacklisting some stuff in a firewall or similar. There are also some VMs that can simulate bad internet connections, maybe that would work as well. I'll get the log once it happens again, currently it's running and the log is spammed with downloaded file names (which I do not want to share).
Fabian42
commented
4 years ago The problem occurred, I killed the program, emptied the log, started it again, waited until the problem happened again, killed the program, copied the log and did the censoring and shortening:
2020-05-18 21:49:44,122 - GoSync - DEBUG - Initialize - Started Initialize
2020-05-18 21:49:44,122 - GoSync - INFO - Initialize - Completed Credentials Verification
2020-05-18 21:49:44,122 - GoSync - INFO - Initialize - Saving credentials
2020-05-18 21:49:44,122 - GoSync - INFO - Initialize - starting oberserver
2020-05-18 21:49:44,122 - GoSync - INFO - Initialize - Going for authentication
2020-05-18 21:49:44,122 - GoSync - INFO - Authenticate - Loading pickle file
2020-05-18 21:49:44,123 - GoSync - INFO - Authenticate - expired. Refreshing
2020-05-18 21:49:44,235 - GoSync - INFO - Authenticate - Saving pickle file
2020-05-18 21:49:45,021 - GoSync - INFO - Initialize - Completed Drive Quota Execution
2020-05-18 21:49:45,022 - GoSync - DEBUG - Initialize - Completed Account Information Load
2020-05-18 21:49:45,022 - GoSync - DEBUG - Initialize - Completed mirror_directory validation
2020-05-18 21:49:45,023 - GoSync - DEBUG - Initialize - Completed Config File Load
2020-05-18 21:49:45,030 - GoSync - DEBUG - Initialize - Completed GoogleDriveTree File
2020-05-18 21:49:45,030 - GoSync - DEBUG - Initialize - Completed Initialize
2020-05-18 21:49:50,342 - GoSync - INFO - ###############################################
2020-05-18 21:49:50,344 - GoSync - INFO - Start - Syncing remote directory
2020-05-18 21:49:50,344 - GoSync - INFO - ###############################################
2020-05-18 21:49:50,344 - GoSync - INFO - ### SyncRemoteDirectory: - Sync Started - Remote Directory () ... Recursive = True
2020-05-18 21:49:50,754 - GoSync - DEBUG - SyncRemoteDirectory: Checking directory ([censored])
2020-05-18 21:49:50,754 - GoSync - DEBUG - SyncRemoteDirectory: Syncing directory ([censored])
2020-05-18 21:49:50,755 - GoSync - INFO - ### SyncRemoteDirectory: - Sync Started - Remote Directory ([censored]) ... Recursive = True
[… a ton of "SyncRemoteDirectory: Checking directory", "SyncRemoteDirectory: Syncing directory", "SyncRemoteDirectory: Checking file", "DownloadFileByObject: Skipping File" and "SyncRemoteDirectory: - Sync Completed - Remote Directory" here …]
[ some repetitions of this: ]
2020-05-18 21:52:10,672 - GoSync - DEBUG - SyncRemoteDirectory: Checking file ([censored])
2020-05-18 21:52:10,679 - GoSync - DEBUG - DownloadFileByObject: Download Started - File (/home/fabian/Google Drive/fabianroeling@googlemail.com/[censored])
2020-05-18 21:52:10,921 - GoSync - DEBUG - Observer: /home/fabian/Google Drive/fabianroeling@googlemail.com/[censored] created
2020-05-18 21:52:15,859 - GoSync - INFO - DownloadFileByObject: Download Completed - File (/home/fabian/Google Drive/fabianroeling@googlemail.com/[censored])
[ and then: ]
2020-05-18 21:52:25,328 - GoSync - DEBUG - SyncRemoteDirectory: Checking file ([censored])
2020-05-18 21:52:25,328 - GoSync - DEBUG - DownloadFileByObject: Download Started - File (/home/fabian/Google Drive/fabianroeling@googlemail.com/[censored])
2020-05-18 21:53:49,480 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory ([censored])
2020-05-18 21:53:49,480 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory ([censored])
2020-05-18 21:53:49,480 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory ([censored])
2020-05-18 21:53:49,481 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory ([censored])
2020-05-18 21:53:49,489 - GoSync - DEBUG - file: /home/fabian/Google Drive/fabianroeling@googlemail.com/[censored] drivepath is [censored]
[ a few repetitions of this: ]
2020-05-18 21:53:49,495 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on (root)
2020-05-18 21:53:49,967 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on (root)
2020-05-18 21:53:49,967 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on ([censored])
2020-05-18 21:53:50,986 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on ([censored])
2020-05-18 21:53:50,987 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on ([censored])
2020-05-18 21:53:51,601 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on ([censored])
2020-05-18 21:53:53,240 - GoSync - DEBUG - Found file [censored] on remote (dpath: [censored])
2020-05-18 21:53:53,241 - GoSync - DEBUG - Checking if they are same...
2020-05-18 21:53:53,274 - GoSync - DEBUG - yes
2020-05-18 21:53:53,278 - GoSync - DEBUG - Observer: /home/fabian/Google Drive/fabianroeling@googlemail.com/[censored] created
2020-05-18 21:53:53,278 - GoSync - DEBUG - file: /home/fabian/Google Drive/fabianroeling@googlemail.com/[censored] drivepath is [censored]
[ and then: ]
2020-05-18 21:54:00,653 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on (root)
2020-05-18 21:54:01,022 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on (root)
2020-05-18 21:54:01,023 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on ([censored])
2020-05-18 21:54:01,637 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on ([censored])
2020-05-18 21:54:01,637 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on ([censored])
2020-05-18 21:54:02,046 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on ([censored])
2020-05-18 21:54:03,076 - GoSync - DEBUG - Found file [censored] on remote (dpath: [censored])
2020-05-18 21:54:03,076 - GoSync - DEBUG - Checking if they are same...
2020-05-18 21:54:03,076 - GoSync - DEBUG - no
2020-05-18 21:54:03,076 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on (root)
2020-05-18 21:54:03,479 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on (root)
2020-05-18 21:54:03,479 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on ([censored])
2020-05-18 21:54:03,889 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on ([censored])
2020-05-18 21:54:03,889 - GoSync - DEBUG - GetFolderOnDrive: Checking Folder ([censored]) on ([censored])
2020-05-18 21:54:04,299 - GoSync - INFO - GetFolderOnDrive: Found Folder ([censored]) on ([censored])
2020-05-18 21:54:04,299 - GoSync - DEBUG - Create file /home/fabian/Google Drive/fabianroeling@googlemail.com/[censored]
I can give a bit more information on all those "[censored]"s, if you need it, but you probably already know roughly what is supposed to be there.
Fabian42
commented
4 years ago I just (sort of) witnessed it giving out an error message, counting down its timer as usual and then continuing. So I guess sometimes it does indeed fix itself. I looked at the VM every few minutes and saw those different states. My internet connection has been extremely unstable today, so it's a good day for testing this. :D The good news is that the program does no damage to my Google Drive files. I've created a backup before starting, just in case, but it did not actually do what I feared: Downloading some of it, getting stuck, after restarting assume that the files that were missing locally were deleted locally, delete on cloud. I think some other program that I tried before did that, but I'm not sure, it was long ago.
Fabian42
commented
4 years ago Actually, it's stuck at "sync started" now.
hschauhan
commented
4 years ago Thanks for the logs! Looks like the download failed and the chain of problems started. I think this is duplicate of issue #23. I am not closing this right now. Let me try to fix the problem. Once you confirm that you don't see this problem, then I will close this issue as well.
hschauhan
commented
4 years ago Hi, It was indeed issue #23. I have fixed and it seems to work fine at my end. I request you to please update the source, retry and provide comment.
Fabian42
commented
4 years ago Nope, not fixed, still hangs at "00:01".
Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/fabian/.local/lib/python3.8/site-packages/httplib2/__init__.py", line 1557, in _conn_request
conn.connect()
File "/home/fabian/.local/lib/python3.8/site-packages/httplib2/__init__.py", line 1305, in connect
address_info = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)
File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 724, in MakeFileListQuery
response = self.drive.files().list(q=query,
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 892, in execute
resp, content = _retry_request(
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 204, in _retry_request
raise exception
File "/home/fabian/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 177, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/home/fabian/.local/lib/python3.8/site-packages/google_auth_httplib2.py", line 197, in request
response, content = self.http.request(
File "/home/fabian/.local/lib/python3.8/site-packages/httplib2/__init__.py", line 1982, in request
(response, content) = self._request(
File "/home/fabian/.local/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
(response, content) = self._conn_request(
File "/home/fabian/.local/lib/python3.8/site-packages/httplib2/__init__.py", line 1564, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/home/fabian/.local/lib/python3.8/site-packages/watchdog/observers/api.py", line 196, in run
self.dispatch_events(self.event_queue, self.timeout)
File "/home/fabian/.local/lib/python3.8/site-packages/watchdog/observers/api.py", line 369, in dispatch_events
handler.dispatch(event)
File "/home/fabian/.local/lib/python3.8/site-packages/watchdog/events.py", line 452, in dispatch
super(PatternMatchingEventHandler, self).dispatch(event)
File "/home/fabian/.local/lib/python3.8/site-packages/watchdog/events.py", line 331, in dispatch
{
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 1056, in on_created
self.sync_handler.UploadObservedFile(evt.src_path)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 440, in UploadObservedFile
self.UploadFile(file_path)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 405, in UploadFile
f = self.LocateFileOnDrive(drivepath)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 606, in LocateFileOnDrive
f = self.LocateFolderOnDrive(dirpath)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 647, in LocateFolderOnDrive
folder = self.GetFolderOnDrive(dir1, croot)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 664, in GetFolderOnDrive
file_list = self.MakeFileListQuery("'%s' in parents and trashed=false" % parent)
File "/home/fabian/.local/lib/python3.8/site-packages/GoSync/GoSyncModel.py", line 738, in MakeFileListQuery
self.SendlToLog(1,"MakeFileListQuery: failed with reason %s\n" % error.resp.reason)
UnboundLocalError: local variable 'error' referenced before assignmentDid you sync the code again? self.SendlToLog(1,"MakeFileListQuery: failed with reason %s\n" % error.resp.reason) is not there at 731 line number. Please update the code.
Fabian42
commented
4 years ago I downloaded it and used "sudo pip3 install --upgrade ." there. Is that not the update command?
hschauhan
commented
4 years ago The latest code hasn't been published in pip yet. I would suggest that you download the latest tar from github. Untar it. It will create a "gosync" directory. Go inside "gosync" and run "python GoSync/GoSync.py"
With this you can test the latest committed code without waiting for PIP publication.
hschauhan
commented
4 years ago Let me know if you need any help regarding this.
Fabian42
commented
4 years ago 
That's not really better at all.
hschauhan
commented
4 years ago Please update your source code. You would probably like it now. Such dialog boxes are removed. Notification message box is used instead. The pause and resume of the sync based on network conditions happens automatically without user intervention.
Also, if a file download was interrupted because of server issue, the retries are added.
Thanks for testing it so patiently!
Fabian42
commented
4 years ago It seems to switch between "Sync starts in […]" and "Checking […]" now, but it also spams notifications:
It might have downloaded everything by now, except for files over 4GB, but it's hard to say whether everything is there while those big files are missing. I'll create a new report for that.
If that time between syncs is considered normal, it should definitely be configurable. Or maybe it already is, I don't know, because --help does nothing.
hschauhan
commented
4 years ago There definitely a problem with connection getting severed and download failing. Alain is working on the issue. The download doesn’t always fail. The next sync should have downloaded the file. I have tested with file size of 2GB.
Regarding sync timing. Are you pointing towards 30 minute interval between two sync? That will be configurable. Another tab is coming up. That will have option to configure this, change the local sync directory, and enable/disable auto start of sync as soon as application is launched.
Notification are going to stay.
hschauhan
commented
4 years ago Can I close this issue? I see that you have opened another one for download errors.
Fabian42
commented
4 years ago Well, what's still left from this is that it spams notifications. InSync somehow manages to notice immediately when there are changes on Google Drive, without needing to loop through all folders every time. With my 21217 files in 811 folders, that also takes quite a while. I don't know what exactly InSync does better here, but it would be worth considering for GoSync as well.
hschauhan
commented
4 years ago Hi Fabian,
Now there is a checkbox in settings page. If you uncheck it, the notifications will not be sent. Please try the new code. Thanks
hschauhan
commented
4 years ago The last standing problem "spamming the notification" has been fixed via "settings" tab. User can choose if updates should be delivered via system notification or not.
I tried to do the initial sync over night (>90GB). The internet connection that I used for that was somewhat unstable, so it cut off for a short time at some point and GoSync printed this error to the console:
The UI got stuck at "sync starting in 00:01" (or similar, I can't see it anymore) and didn't react to "sync now" or other attempts to start it again. No files were downloaded anymore. I killed the program and tried to restart it, but now I only ever get this:
There is no error message in the console. I have no idea if it's even possible to recover from this, probably only by resetting everything and starting over.
(This issue is not urgent for me personally, since I'm already using InSync. I only tested GoSync because I like testing programs and because this could be a program that I might want to recommend in the future, once it actually works reliably, because then it would be the only actually working free two-way sync for Google Drive and Linux. I also don't have any issues with my system, because I installed GoSync in a VM.)