search

chrsmithdemos / google-api-java-client

Automatically exported from code.google.com/p/google-api-java-client
0 stars 0 forks source link

Non-chunked resumable uploads #587

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
External references, such as a standards document, or specification?
https://devsite.googleplex.com/storage/docs/json_api/v1/how-tos/upload#resumable

Java environments (e.g. Java 6, Android 2.3, App Engine, or All)?
Java 6, non-App Engine

Please describe the feature requested.
Requests using the resumable upload feature be sent as one streaming HTTP 
request (i.e., not having to be entirely resident in application memory) rather 
than chunks, as chunking incurs a significant performance penalty (4 to 5x 
longer) over just sending bytes until a connection error or early error result 
is returned.

Specifically, that a buffered stream wrap (if necessary) the underlying 
InputStream such that is has enough buffer to go back as far as required if the 
upload fails. This should be calculable based on some minimum internal commit 
chunk size plus (speed of the connection * maximum time before you know 
something went wrong with the connection, e.g., tcp timeout). Then, you are 
guaranteed in case of TCP termination or early HTTP error result that you can 
perform the resume an interrupted upload steps (query for how many bytes remain 
to be sent) and then send a subsequent single streaming HTTP session with the 
remaining bytes and the appropriate Content-Range header. Because there's no 
regular HTTP result code as for a chunk, you have to use that same calculus to 
determine when to invalidate earlier parts of the stream (record current 
location + Reset + Skip + Mark + Skip to current location) to avoid bloating 
application memory.

Original issue reported on code.google.com by nherr...@google.com on 9 Aug 2012 at 9:24

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 2 Nov 2012 at 5:26

GoogleCodeExporter commented 9 years ago

Original comment by rmis...@google.com on 11 Dec 2012 at 1:37

GoogleCodeExporter commented 9 years ago 
Streaming is not possible on Google App Engine.  Therefore, we definitely do 
not want this for platforms like Google App Engine.

Original comment by yan...@google.com on 19 Jan 2013 at 1:55

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 7 Feb 2013 at 12:05

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 10 Jun 2013 at 1:26

GoogleCodeExporter commented 9 years ago

Original comment by pele...@google.com on 29 Jul 2013 at 6:09

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 27 Sep 2013 at 12:04