search

brandonongnz / google-api-python-client

Automatically exported from code.google.com/p/google-api-python-client
Other
0 stars 0 forks source link

Resumable media upload fails once the access_token expires. #231

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
First of all, I don't really know for sure this is a bug, for I'm too beginner 
with both the API and Python. But I tried several different ways and got the 
same error, so I thought it might really be a bug.

When uploading a large file (one that takes more than 1~2 hours to upload 
completely, if I use resumable upload, after about 1h, the upload is 
interrupted with 401 error from server. After some digging in the API 
documentation and source code, I found out that it automatically refreshs the 
access token using the refresh token when it gets this error. Thing is that, 
although it does change the access token, the server keeps responding with 401, 
no matter how many times the access token is refreshed.

I'm attaching the code I used here and copying the terminal output... the 
output lines for the http.request.credentials.access_token are for debugging 
purposes.

ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Uploaded 2.28%
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Uploaded 2.29%
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Uploaded 2.29%
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Uploaded 2.30%
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Error (401)... retrying.
ya29.AHES6ZQqp3_qbWsTk4yVDdHnlwc_7GvPZiFIReDnhIIiHao
Error (401)... retrying.
ya29.AHES6ZSqx90ZOUKqDEP4AAfWCVgXZYT2vJAiLwKDRu87JOs
Error (401)... retrying.
ya29.AHES6ZTp0RZ6U5K5UdDom0gq3XHnyVS-2sVU9hILOrG4o3Y
Error (401)... retrying.
ya29.AHES6ZSR-IOiwJ_p_Dm-OnCanVIVhCZLs7H_pYLMGIap8W0
Error (401)... retrying.
ya29.AHES6ZRnmM-YIZj4S8gvYBgC1M8oYy4Hv5VlcwRqgnZCOCE
Error (401)... retrying.
ya29.AHES6ZSF7Q7C3WQYuPAWrxvqbTRsipaVKhv_TfrD_gef1DE
Error (401)... retrying.
ya29.AHES6ZTsGzwIIprpPhCrqmoS3UkPsRzst5YHqL-zXJmz6Ak
Error (401)... retrying.
ya29.AHES6ZSS_1ZBiQJvZG_7t5uW3alsy1piGe4-u2YDnwycVrI
Error (401)... retrying.
ya29.AHES6ZTLFbBS8mSFWQ9zK8cgbX8RPeLghPxkfiKY54hBB-0
Error (401)... retrying.
ya29.AHES6ZQBeMWY50z6fWXvaCcd5_AJr_AYOuL2aiNKpK-mmyU
Error (401)... retrying.
ya29.AHES6ZTs2mYYSEyOqI_Ms4itKDx36t39Oc5RNZHkV4Dq49c
Retries limit exceeded! Aborting.

My version infos:
Debian Lenny, with kernel 2.6.26-2-686.
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) 
[GCC 4.3.2] on linux2

I don't really know how to find the version of the api client, but I got it 
from pip install about a week ago (both the google-api-python-client and ssl).

Original issue reported on code.google.com by grfro...@gmail.com on 13 Jan 2013 at 5:51

Attachments:

GoogleCodeExporter commented 8 years ago 
Apparently, the problem isn't with the api, but with long resumable uploads. I 
tried to upload two small files with the same code, but with 1h between them, 
and the api does renew the access token and successfully uploads the second 
file.

I'm attaching this new script and copying the output:

ya29.AHES6ZRUssiLfuhqCP9Cu7C7LuhRV2rYzPldU27wiMJZWb8
Uploaded 66.89%
ya29.AHES6ZRUssiLfuhqCP9Cu7C7LuhRV2rYzPldU27wiMJZWb8
Upload 1 Complete!
ya29.AHES6ZRUssiLfuhqCP9Cu7C7LuhRV2rYzPldU27wiMJZWb8
Uploaded 57.62%
ya29.AHES6ZQd3o1ciwXpNFImH3CK0-dJAtQba_oeIO9DDbIq154
Upload 2 Complete!

Original comment by grfro...@gmail.com on 13 Jan 2013 at 9:40

Attachments:

GoogleCodeExporter commented 8 years ago 
This is an issue with the backend and not with the API or with your code. As 
you deduced, if the upload goes too long the access_token expires and at that 
point the resumable upload can't be continued. There is work on progress to fix 
this issue right now, I will update this bug once the issue is fixed on the 
server side.

Original comment by jcgregorio@google.com on 13 Jan 2013 at 11:00

GoogleCodeExporter commented 8 years ago 

I've the same problem (401 error) with the java implementation!

Original comment by m...@michaelholst.com on 3 Feb 2013 at 10:51

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
The same problem for ruby!

Original comment by ablya...@gmail.com on 4 Feb 2013 at 6:43

GoogleCodeExporter commented 8 years ago 
I think I got same issue but different error. When uploading 1GB file after 1 
hour i got this exception (using java library)
WARNING: exception thrown while executing request
java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at sun.security.ssl.InputRecord.readFully(Unknown Source)
        at sun.security.ssl.InputRecord.read(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
        at sun.security.ssl.AppInputStream.read(Unknown Source)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
        at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
        at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
        at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
        at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
        at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
        at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
        at com.google.api.client.http.apache.ApacheHttpRequest.execute(ApacheHttpRequest.java:59)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1009)
        at com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:368)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:415)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:340)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:458)
        at com.jmtech.googledrive.helper.GoogleDriveHelper$1.execute(GoogleDriveHelper.java:301)
        at com.jmtech.googledrive.helper.GoogleDriveHelper$GoogleExecutor.getResult(GoogleDriveHelper.java:624)
        at com.jmtech.googledrive.helper.GoogleDriveHelper.uploadFile(GoogleDriveHelper.java:289)
        at com.jmtech.googledrive.helper.CsvUploadHelper$WorkerThread.run(CsvUploadHelper.java:225)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Original comment by putra...@jmtech.co.jp on 14 Mar 2013 at 4:55

GoogleCodeExporter commented 8 years ago 
Having the same issue with js implementation of resumable uploads. After 1-2 
hours I get a 401 status. I'm refreshing my Bearer token but it fails. Also, 
when I make a PUT request to get the upload status I'm also getting a 401 
status which suggests that maybe the file was deleted. My chunk size is about 
2MB, would it be a good idea to increase this in order to reduce the number of 
requests?

Thanks

Original comment by gustavo...@feathermexico.com on 25 Mar 2013 at 5:59

GoogleCodeExporter commented 8 years ago 
I'm using Drive REST API from C++ code, having the same issues. My resumable 
upload sessions die if they run for over 1 hour with no ability to resume, 
because upload session seems to be implicitly bound to bearer token that 
expires. Maybe some workaround solution exists?

Original comment by brittl...@gmail.com on 26 Mar 2013 at 8:53

GoogleCodeExporter commented 8 years ago 
This is still a known issue on the server side, so it will affect all client 
libraries. It is being worked on and I will update this bug when a fix is in 
place.

Original comment by jcgregorio@google.com on 26 Mar 2013 at 1:23

GoogleCodeExporter commented 8 years ago 
Hi jcgregorio,

Seriously, I really can't understand how a company like Google did not resolve 
such big issue in a few days...now it's been 3 months and the problem is still 
there.
Don't you have a few engineers to work on it and solve the issue ?
Google products used to be in permanent beta stage while we are paying for 
production software !!
It's a shame and this is the last time I give a chance to your services, next 
time I will advice my clients to use services that just work (ie Dropbox), we 
don't have time to waste debugging your piece of code.

Original comment by l...@eenov.com on 5 May 2013 at 1:18

GoogleCodeExporter commented 8 years ago 
@jcgregorio
Shouldn't the bug status be changed from "new" since it is being worked on?

Original comment by tais.han...@gmail.com on 29 May 2013 at 12:53

GoogleCodeExporter commented 8 years ago 
I'm trying to back up hd 1080p video files over 1gb each. Google Please help!

Original comment by rahulvar...@gmail.com on 29 May 2013 at 3:28

GoogleCodeExporter commented 8 years ago 
I am getting the same issue, there should be a workaround for the same? Can't 
we refresh the credentials and resume the upload from the same location?

Original comment by saurabh....@gmail.com on 30 May 2013 at 1:29

GoogleCodeExporter commented 8 years ago 
@saurabh
Refreshing auth token doesn't help, as resumable upload implicitly binds to old 
authorizartion token. I.e. you can do any other requests with the new token, 
but can't resume your upload session.

Original comment by brittl...@gmail.com on 30 May 2013 at 2:00

GoogleCodeExporter commented 8 years ago 
I tried to avoid the acess token expiry during upload by refreshing the token 
every 30 minutes and resuming the upload with the refreshed token. But this 
does not help. Again, after about 1 hour (i.e. a bit more) the upload was 
stopped with 401 error.

The "official" Google Drive windows tool does not seem to have this problem. I 
successfully "synced" a 800MB video file to my Drive (which took about 2 
hours). 

@jcgregorio
Can you please share the workaround implemented in the Windows tool with us?

Original comment by frg...@googlemail.com on 30 May 2013 at 6:34

GoogleCodeExporter commented 8 years ago 
If you're backing up these videos for personal use, I suggest you use the 
browser (and hope for the connection to stay still)... the gdrive web interface 
doesn't have this problem. I did this wih my big files.

Another option that I didn't try, but that might do it is to upload one single 
big chunk.... you wouldn't need a credentials refresh, so I believe it might 
work. It has the same drawback as browser upload: you don't get resumable 
upload. @frg, I believe the gdrive tool isn't doing resumable uploads.... can 
you confirm?

I really don't believe the fix is coming anytime soon.

Original comment by grfro...@gmail.com on 30 May 2013 at 6:43

GoogleCodeExporter commented 8 years ago 
Now i am using java api to upload files to Google Drive.They said the size 
limits for a user (free) upto 5gb.
Plz answer the following questions.

1) for this free account, what is the maximum size of a file (any format) to 
upload into gdrive?.
I got a google url and use this to get a authorization code.Using this single 
authorization code, i want to upload many files with big sizes in multiple of 
MB's. 
2) When will this authorization code expire?
3) How Can i reuse this code via java code directly(not web application),when 
its expires? 

Please reply to my mail 

npediasenthil@gmail.com,senthilmurugangsm@gmail.com

Original comment by npediase...@gmail.com on 3 Jun 2013 at 12:54

GoogleCodeExporter commented 8 years ago 
@grfo: It really seems the gdrive win tool does not support resumable uploads. 
I interrupted a 2-hours upload after ONE hour with a reboot. After the reboot, 
the upload did need another 2 hours.

I further tried to upload one single big chunk (~800 MB) with a modified 
version of your script. It terminates with a "Memory Error" due to the fact 
that the complete file is read into memory before upload. So this is also not a 
solution :-(

Any idea how to achieve at least a "chunked" upload? 

I agree, a fix for the resumable upload is not coming soon.

Original comment by frg...@googlemail.com on 3 Jun 2013 at 8:45

GoogleCodeExporter commented 8 years ago

Original comment by jcgregorio@google.com on 5 Aug 2013 at 2:09

GoogleCodeExporter commented 8 years ago 
any update on this issue?

Original comment by publicfr...@gmail.com on 7 Sep 2013 at 8:32

GoogleCodeExporter commented 8 years ago 
This should now be fixed, please try it out.

Original comment by jcgregorio@google.com on 9 Sep 2013 at 1:42

GoogleCodeExporter commented 8 years ago 
I ran into the same issue last week. 
Will try it again today and post the results.

Original comment by fexco...@gmail.com on 9 Sep 2013 at 4:56

GoogleCodeExporter commented 8 years ago 
See also: http://code.google.com/p/gdata-issues/issues/detail?id=5124

Original comment by yann...@gmail.com on 9 Sep 2013 at 10:57

GoogleCodeExporter commented 8 years ago 
Still not working for me. Still stops after about 1 hour with 401.

Original comment by grfro...@gmail.com on 23 Sep 2013 at 7:54

GoogleCodeExporter commented 8 years ago 
hi, what's the current status with this issue ? Is the work still ongoing to 
get the fix for this issue?

Original comment by lavik...@gmail.com on 28 Jan 2014 at 11:43

GoogleCodeExporter commented 8 years ago 
This is till a problem files above 7-8 meg will not sync.  I just get a lost 
connection and then maybe one or two files will sync next time around

Original comment by 2richmon...@googlemail.com on 6 Apr 2014 at 2:29

GoogleCodeExporter commented 8 years ago 
OK so what's up with this issue?? I see it's been well over a year since it was 
created.

It looks like I ran into the same issue using a Java client 
(https://github.com/LeviticusMB/Stream2GDrive/).

Not being able to upload large files kind of defeats the purpose of buying 1 TB 
of storage quota ... :(

Original comment by mar...@blom.org on 27 Apr 2014 at 5:44

GoogleCodeExporter commented 8 years ago 
Any updates on this issue?

Original comment by prakashv...@gmail.com on 2 Jul 2014 at 3:10

GoogleCodeExporter commented 8 years ago 
Same problem with the Synology Cloud Sync program which use this *** API

It seems google doesn't want to correct it maybe to prevent uploading big files 
? Or to force use the web interface.

Original comment by david.gi...@gmail.com on 24 Jul 2014 at 6:26

GoogleCodeExporter commented 8 years ago 
It's been 19 months since this was first reported. Worse still, Google Drive 
now advertises that it will support file sizes up to 5TB. That's right, while 
most of us are struggling to upload 5GB, Google proudly boasts 5000GB.  This is 
deceptive advertising.

WHEN WILL THIS BE FIXED ?!!!!??!?!?!?!

Original comment by grantphe...@gmail.com on 1 Aug 2014 at 7:09

GoogleCodeExporter commented 8 years ago 
also interested in this or been able to append to a file

Original comment by ch...@gilligan.id.au on 11 Aug 2014 at 7:11

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
It's been 20 months since this was first reported. Worse still, Google Drive 
now advertises that it will support file sizes up to 5TB. That's right, while 
most of us are struggling to upload 5GB, Google proudly boasts 5000GB.  This is 
deceptive advertising.

WHEN WILL THIS BE FIXED ?!!!!??!?!?!?!

Original comment by ro...@nsp.net.ua on 25 Sep 2014 at 12:21

GoogleCodeExporter commented 8 years ago 
Any chance to get it fixed?

Original comment by a.vent...@gluak.com on 20 Oct 2014 at 8:45

GoogleCodeExporter commented 8 years ago 
Is there any other way to upload large files via APIs?
I mean, by using some other APIs version?

Original comment by a.vent...@gluak.com on 20 Oct 2014 at 8:52

GoogleCodeExporter commented 8 years ago 
a quick update: the internal side of this is once again in motion. (it had hit 
a roadblock for other reasons, and that's now gone.)

i'll update again once the fix is live. i don't have an ETA right now.

Original comment by craigcitro@google.com on 22 Oct 2014 at 9:42

GoogleCodeExporter commented 8 years ago 
Is this an internal (backend) issue or an API client issue?

Original comment by o...@linkgard.com on 22 Oct 2014 at 9:43

GoogleCodeExporter commented 8 years ago 
it's nothing we can fix in the client -- it's a problem on the backend (and as 
such, affects clients in any language).

Original comment by craigcitro@google.com on 22 Oct 2014 at 9:46

GoogleCodeExporter commented 8 years ago 
Thank you for the information.

Original comment by o...@linkgard.com on 22 Oct 2014 at 10:15

GoogleCodeExporter commented 8 years ago 
and how long to wait for fixes this bug?

Original comment by ro...@nsp.net.ua on 22 Oct 2014 at 11:17

GoogleCodeExporter commented 8 years ago 
Ok, thanks, just keep us posted.

Original comment by a.vent...@gluak.com on 23 Oct 2014 at 10:03

GoogleCodeExporter commented 8 years ago 
So just to clarify. Once google resolves the problem on there end, ALL of the 
current api apps will work with no change to there code? Or will we have to 
update our code once google pushes a fix?

PS Any updates on progress, or an ETA?

Original comment by Rick.Dug...@gmail.com on 7 Nov 2014 at 10:56

GoogleCodeExporter commented 8 years ago 
They will not solve it. They are not profitable. Better to have a 
malfunctioning client with lots of files, and that does not support symbolic 
links, etc.

Original comment by ro...@nsp.net.ua on 9 Nov 2014 at 9:48

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
The same problem was in YouTube API, it is fixed now
https://code.google.com/p/gdata-issues/issues/detail?id=5124

Please, star and spam this new ticked
http://code.google.com/p/gdata-issues/issues/detail?id=6757
It is the same issue, they have to fix it.

Microsoft OneDrive does not support resuming, they are far behind.

Original comment by psyfre...@gmail.com on 12 Nov 2014 at 5:10

GoogleCodeExporter commented 8 years ago 
Isn't that bug a duplicate of this one? The root cause is the same and they are 
aware and working on a fix.

Original comment by cstc...@gmail.com on 12 Nov 2014 at 6:10

GoogleCodeExporter commented 8 years ago 
To those above that ask for alternative ways, the old API works just fine... It 
has been deprecated, but for the time being it offers a robust solution. 

Here the old API description: 
https://developers.google.com/gdata/docs/resumable_upload

An acceptable strategy consists in using the new API for everything except for 
the large files, which can be handled relying on the old API. (And get rid of 
the old API stuffs as soon as this issue is resolved).

Original comment by loic.mer...@gmail.com on 13 Nov 2014 at 9:46

GoogleCodeExporter commented 8 years ago 
hi all -- the fix has been completely deployed, and this should now be fixed on 
the backend.

Original comment by craigcitro@google.com on 4 Dec 2014 at 6:52

GoogleCodeExporter commented 8 years ago

Original comment by craigcitro@google.com on 4 Dec 2014 at 6:53