"An item with the same key has already been added" error in ResumableUploader

[GoogleCodeExporter]

Error "An item with the same key has already been added" occurs in ..

Google.GData.Client.ResumableUpload.ResumableUploader.UploadStream(String 
httpMethod, Uri sessionUri, Authenticator authentication, Stream payload, 
String mediaType, AsyncData data)

.. when interrupting and restoring network connectivity while running the 
YouTubeUploader sample application at ..

http://code.google.com/p/google-gdata/source/browse/#svn%2Ftrunk%2Fclients%2Fcs%
2Fsamples%2FYouTubeUploader

The error occurs after the sample application pauses for a retry and then 
resumes to attempt to retry. The error is traced at line 357 of 
YouTubeUploader.cs.

What steps will reproduce the problem?
1. Execute the YouTubeUploader sample app.
2. View the readme to know the .csv file format.
3. Reference a .csv file that points to a sufficiently large video file for 
testing.
4. Begin uploading.
5. While uploading, disconnect from the network.
6. As soon as the uploader detects a problem and indicates that it will retry 
after a pause, restore network connectivity.

What is the expected output? What do you see instead?

The attempt should either fail due to network connectivity again or else should 
succeed and continue to progress towards 100% uploaded. Instead, the error is 
"An item with the same key has already been added" and the upload cannot resume.

Original issue reported on code.google.com by MrStimp...@gmail.com on 1 Aug 2011 at 9:01

[GoogleCodeExporter]

Complete stack trace (no debug symbols / line numbers, sorry):

System.ArgumentException: An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Google.GData.Client.ResumableUpload.ResumableUploader.UploadStream(String httpMethod, Uri sessionUri, Authenticator authentication, Stream payload, String mediaType, AsyncData data)
   at Google.GData.Client.ResumableUpload.ResumableUploader.Resume(Authenticator authentication, Uri resumeUri, String httpmethod, Stream payload, String contentType, AsyncData data)
   at Google.GData.Client.ResumableUpload.ResumableUploader.AsyncResumeWorker(AsyncResumableUploadData data, AsyncOperation asyncOp, SendOrPostCallback completionMethodDelegate)

Original comment by MrStimp...@gmail.com on 1 Aug 2011 at 9:05

[GoogleCodeExporter]

Can you capture the HTTP request sent when resume is attempted (and fails)?

Original comment by ccherub...@google.com on 1 Aug 2011 at 5:46

[GoogleCodeExporter]

There is no HTTP request when resume is attempted and fails. The error occurs 
before it reaches the HTTP stack.

This is the HTTP request that occurs before the outage occurs.

POST 
http://uploads.gdata.youtube.com/resumable/feeds/api/users/HomebodyBuddy/uploads
?upload_id=AEnB2Uqck_s0dmWGd-fMZ3enUwAK5yvHr58rlMIImDwn3P0QeCib8ZBTuk4K1jRw8-45K
DOQB6lhzBZ73el6nKLlAEkrXKPZjw HTTP/1.1
X-GData-Key: 
key=AI39si5HV9zaLn4okq_gTqY4vARdZBf2_8D3bTkiK2FWgqaVVH9tITjBa6nyaAdg19Y4in-hnqcF
TXu7i3d-2RIKSCvIMBARAg
Authorization: GoogleLogin 
auth=DQAAALgAAACrOOBkad3IIa-CflEQGQRc3NEXoZ4xeUr2ZPk7XotDc8MjeT5pYyAgA8mTKLT2ynl
9EWu157nqBB0vK7xRmo8UXLqths2DUA3_NT38o5gZT4E5eU7Xhty2M4nsiz4g_WlLfXqFaRvSN5cznxM
JefQx-9PKfgjzd9j94nic0735jrLB32HOdd4J_9ej_qWD7LD-u9IKeJDPsWBrpCYO76TWI9ZNAObMMHF
WR4J7BBIDf1AEJ4yXRH9gew_fdBHEtqo
Content-Type: video/mp4
Content-Range: bytes 0-25690111/63624398
Host: uploads.gdata.youtube.com
Content-Length: 25690112

Original comment by MrStimp...@gmail.com on 2 Aug 2011 at 2:00

[GoogleCodeExporter]

Hi,
I have exactly the same problem.
Did you find any workaround?

Original comment by martin.r...@googlemail.com on 2 Sep 2011 at 8:25

[GoogleCodeExporter]

No, I have been waiting for this problem to be fixed and presume 
ResumableUploader to be an unusable resource for resumable uploading. I have 
been trying to get by with "retryable uploads"--basically starting over after a 
network error occurs--with disappointing success.

Original comment by MrStimp...@gmail.com on 2 Sep 2011 at 10:49

[GoogleCodeExporter]

Sorry about the delay, we have been busy cutting version 1.9 of the library 
which will be published on Tuesday.

This issue is part of the next release, work on it will start next week.

Original comment by ccherub...@google.com on 2 Sep 2011 at 10:53

[GoogleCodeExporter]

This issue does still exist in Version 1.9.

I now use the source I'm getting from svn and changed 
ResumableUploader.UploadStream to check the Dictionary before calling Add:
if (!lastChunks.ContainsKey(sessionUri)) lastChunks.Add(sessionUri, 0);

After this change there is another issue. If you try to resume an upload after 
the connection was cut and restored, the Write operation on the request stream 
throws this error:

Exception Type: System.IO.IOException
Message: Unable to write data to the transport connection: An established 
connection was aborted by the software in your host machine.
Source: System
Stacktrace:    at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 
offset, Int32 size)
   at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at Google.GData.Client.ResumableUpload.ResumableUploader.CopyData(Stream input, HttpWebRequest request, Int32 partIndex, AsyncData data, Uri requestId) in D:\Downloads\code\google\google-gdata\clients\cs\src\core\resumableupload.cs:line 575
   at Google.GData.Client.ResumableUpload.ResumableUploader.UploadStreamPart(Int32 partIndex, String httpMethod, Uri sessionUri, Authenticator authentication, Stream payload, String mediaType, AsyncData data) in D:\Downloads\code\google\google-gdata\clients\cs\src\core\resumableupload.cs:line 500
   at Google.GData.Client.ResumableUpload.ResumableUploader.UploadStream(String httpMethod, Uri sessionUri, Authenticator authentication, Stream payload, String mediaType, AsyncData data) in D:\Downloads\code\google\google-gdata\clients\cs\src\core\resumableupload.cs:line 457
   at Google.GData.Client.ResumableUpload.ResumableUploader.Resume(Authenticator authentication, Uri resumeUri, String httpmethod, Stream payload, String contentType, AsyncData data) in D:\Downloads\code\google\google-gdata\clients\cs\src\core\resumableupload.cs:line 376
   at Google.GData.Client.ResumableUpload.ResumableUploader.AsyncResumeWorker(AsyncResumableUploadData data, AsyncOperation asyncOp, SendOrPostCallback completionMethodDelegate) in D:\Downloads\code\google\google-gdata\clients\cs\src\core\resumableupload.cs:line 389

Inner Exception:
Exception Type: System.Net.Sockets.SocketException
Message: An established connection was aborted by the software in your host 
machine
Source: System
Stacktrace:    at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, 
Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)

Original comment by zwirne...@st-sportservice.com on 23 Sep 2011 at 2:29

[GoogleCodeExporter]

Hi, 

I committed the small suggested change to ResumableUploader.UploadStream in 
rev. 1135, but as you already noticed, this doesn't fix the whole issue.

It looks like the implementation of the resume operation is flawed and it will 
require a bigger effort to fix it.

Original comment by ccherub...@google.com on 29 Sep 2011 at 11:51

[GoogleCodeExporter]

Issue 576 has been merged into this issue.

Original comment by ccherub...@google.com on 4 Mar 2012 at 5:43

[GoogleCodeExporter]

Greetings,
Our application gets about 50 "Unable to write data to the transport 
connection..." exceptions per week" from its users out of roughly 2500 uploads. 
We use oAuth2/resumable. Our uploads range from 10-150mb. I would love to see 
this fixed, please let me know if it would help to see portions of our code.

Cheers!

Original comment by dsie...@gmail.com on 10 Jul 2012 at 3:01