400 Bad Request failures (due to BOM in the XML POST data?) when trying to use YouTubeUploader

[GoogleCodeExporter]

What steps will reproduce the problem?

I'd imagine this is something specific to me, although I'm not seeing how.

Google acct: james.manning@gmail.com
YouTube username: jamesmanning32

The request is below and attached, although one part that's not shown is 
that the UTF-8 BOM (0xEF, 0xBB, 0xBF) is in the content stream after the 2 
newlines (separating headers from body) and the <?xml - presumably this is 
causing the "Content is not allowed in prolog" SAX exception (from Java, I 
guess) from the response.

You should be able to hex edit the attached file to see the BOM if desired.

Request
*******
POST 
http://uploads.gdata.youtube.com/resumable/feeds/api/users/jamesmanning32/u
ploads HTTP/1.1
X-GData-Key: key=[removed my developer key]
Authorization: GoogleLogin auth=[removed]
Slug: Feb 21 - Caroline.MP4
X-Upload-Content-Type: video/mp4
X-Upload-Content-Length: 56458168
GData-Version: 2.0
Host: uploads.gdata.youtube.com
Content-Type: application/atom+xml; charset=UTF-8
Content-Length: 651
Expect: 100-continue
Connection: Keep-Alive

<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" 
xmlns:gd="http://schemas.google.com/g/2005">
  <media:group xmlns:media="http://search.yahoo.com/mrss/">
    <media:title>Feb 21 - Caroline</media:title>
    <media:description>Recorded on February 21, 2010 using a Flip Video 
camcorder.</media:description>
    <media:keywords>kids</media:keywords>
    <media:category />
  </media:group>
  <title type="text">Feb 21 - Caroline</title>
  <link 
href="http://uploads.gdata.youtube.com/resumable/feeds/api/users/jamesmanni
ng32/uploads" rel="http://schemas.google.com/g/2005#resumable-create-media" 
/>
</entry>
*******

Response
*******
HTTP/1.1 400 Bad Request
Server: Upload Server Built on Jan 25 2010 11:36:53 (1264448213)
X-GData-User-Country: US
Content-Type: application/vnd.google.gdata.error+xml
Date: Wed, 24 Feb 2010 18:30:36 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Content-Length: 195
X-XSS-Protection: 0

<errors xmlns='http://schemas.google.com/g/2005'>
<error>
<domain>GData</domain>
<code>ParseException</code>
<internalReason>Content is not allowed in prolog.</internalReason>
</error>
</errors>
*******

What is the expected output? What do you see instead?
see above

What version of the product are you using? On what operating system?
happens with both YouTubeUploader.exe (1.0.0) from sample SDK and 
YouTubeUploader source fetched via svn (revision 988 from Feb 15)

Please provide any additional information below.

===
Side note: it looks like the sample should be updated to use whatever 
mechanism it can/should to get the actual error from the 400's response 
body - currently it just gives the WebException's (via e.Error) ToString 
and it doesn't look at the response (which correctly has a content type of 
application/vnd.google.gdata.error+xml) at all - that's where the actual 
valuable error message is, after all :)
===

Thanks!

Original issue reported on code.google.com by james.ma...@gmail.com on 24 Feb 2010 at 7:00

Attachments:

• 22_Request.txt

[GoogleCodeExporter]

FWIW, I was able to confirm that's the problem - I used Fiddler 
(http://www.fiddler2.com/) to modify the requests to remove the BOM (although I 
was a 
little worried about the content-length then being wrong) and the requests 
started 
working fine (although painfully slowly, since it was then using Fiddler as a 
proxy 
for everything :)

Here's the fiddler rule I made if you'd like to confirm - add it to the 
OnBeforeRequest method.

    var utf8: System.Text.Encoding = System.Text.Encoding.UTF8;
    var bom: String = utf8.GetString(utf8.GetPreamble());
    oSession.utilReplaceInRequest(bom + "<?xml", "<?xml");

Original comment by james.ma...@gmail.com on 24 Feb 2010 at 8:56

[GoogleCodeExporter]

BTW, the bug doesn't appear to be in the YouTubeUploader itself, but well below 
it in 
the stack - maybe it's even in the BCL (i'm running it on 3.5sp1) as a behavior 
change.  
Googling around, it seems like there's some debate as to whether the BOM should 
be 
allowed to be there or not.

One option is to change the server to accept/ignore the BOM, too :)

Original comment by james.ma...@gmail.com on 24 Feb 2010 at 10:11

[GoogleCodeExporter]

Dug in a little more, and it looks like the problem is in the SaveToXml(Stream) 
method of Google.GData.Client.AtomBase - it constructs an XmlTextWriter to wrap 
the 
stream but passes in an encoding of Encoding.UTF8.  

XmlTextWriter writer = new XmlTextWriter(stream, Encoding.UTF8)

Unfortunately, that version of UTF8 includes writing out the BOM (after all, if 
the 
output stream is a file, you may want the BOM, even though it's optional for 
UTF-8 of 
course).

Instead it needs to pass in a version of the encoding that has the write-BOM 
off - 
you can construct one with: new UTF8Encoding(false)

I'm going to try rebuilding Google.GData.Client.dll with this change to confirm 
it 
fixes it.

Some related blogs posts:

http://www.west-wind.com/WebLog/posts/317089.aspx
http://www.west-wind.com/WebLog/posts/10540.aspx

Original comment by james.ma...@gmail.com on 24 Feb 2010 at 10:37

[GoogleCodeExporter]

Yes, that did the trick - made the one line change, built 
google.gdata.client.dll and 
dropped it in to Google YouTube SDK for .NET\Samples and now upload works great.

Patch attached, although it's just changing the one param in that one line, as 
mentioned above.

Original comment by james.ma...@gmail.com on 25 Feb 2010 at 2:12

Attachments:

• atombase_xmltextwriter.patch

[GoogleCodeExporter]

It might be that this fixes it, but it's not clear why.

The way the XML is persisted is that way for 5 years, and it never caused a 
problem.
In fact, having a BOM in front is not, contrary to your bloglinks, incorrect. 

Furthermore, this does not happen for me, or anyone else that i know of who is 
using
the Uploader, or the SDK in general (if what you wrote would be correct, all 
posts
would fail, against all services, not just the uploader).

Weird. Will require some more data from your side. Runtime is 3.5 SP1? What 
Windows?

Original comment by fman...@gmail.com on 1 Mar 2010 at 7:43

• Changed state: Reviewed

[GoogleCodeExporter]

3.5SP1, 4.0 RC is also on the box.  OS is Win7 32-bit.

I couldn't figure out if the BOM at the front is ok or not, but it certainly 
appeared 
to cause the upload API to fail with the 400.  Maybe a new server build caused 
the 
break (started caring about the BOM, didn't before) and I just got unlucky in 
hitting 
it.  

Knowing whether the BOM is in the success case for others would be great, of 
course.

Original comment by james.ma...@gmail.com on 1 Mar 2010 at 7:50

[GoogleCodeExporter]

I got information that confirmed that there was a problem in the servers. That 
bug
was discovered and fixed. So it looks like your tests happened during that 
time. Can
you verify that, using the original app, this problem is gone for you?

Frank Mantek
Google

Original comment by fman...@gmail.com on 3 Mar 2010 at 2:41

[GoogleCodeExporter]

Confirmed - went back to the original dll and upload is working fine.

Thanks!

Original comment by james.ma...@gmail.com on 3 Mar 2010 at 2:47

[GoogleCodeExporter]

closed, service fixed the issue

Original comment by fman...@gmail.com on 3 Mar 2010 at 2:56

• Changed state: Obsolete