search

Dynalon / Rainy

Simple Tomboy/Tomdroid cloud/syncing server. Written in C# with AngularJS web frontend. Supports SQLite & Postgres Backends.
http://dynalon.github.io/Rainy
GNU Affero General Public License v3.0
88 stars 15 forks source link

Rainy's HTTP-responses are lacking "Content-Length"-header #20

Closed arne1987 closed 11 years ago

arne1987 commented 11 years ago

The responses by the Rainy-server don't have a "Content-Length"-field in the HTTP-header.

While this is not a problem for some clients (e.g. the Tomdroid client), it is a problem for others. E.g. it is NOT possible to sync the Windows version of Tomboy (using .NET, not Mono) with Rainy. Whenever I try to sync that Tomboy version with my own Rainy server OR with the official demonstration server (e.g. https://notesync.org/anna/jMmkjo/), I will get a Tomboy error message: "Server not responding". This is the detailed error message in the Tomboy logfile (for Tomboy v1.14.0): vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv 28.04.2013 16:48:20 [ERROR]: Failed to get auth URL from https://notesync.org/anna/jMmkjo. Exception was: System.Net.WebException: Der Remoteserver hat einen Fehler zurückgegeben: (411) Länge erforderlich. bei System.Net.HttpWebRequest.GetResponse() bei Tomboy.WebSync.Api.OAuth.MakeWebRequest(RequestMethod method, String url, List1 parameters, String postData) in C:\Projects\tomboy-gnome\Tomboy\Addins\WebSyncService\Api\OAuth.cs:Zeile 266. bei Tomboy.WebSync.Api.OAuth.WebRequest(RequestMethod method, String url, String postData) in C:\Projects\tomboy-gnome\Tomboy\Addins\WebSyncService\Api\OAuth.cs:Zeile 212. bei Tomboy.WebSync.Api.OAuth.Post(String uri, IDictionary2 queryParameters, String postValue) in C:\Projects\tomboy-gnome\Tomboy\Addins\WebSyncService\Api\OAuth.cs:Zeile 126. bei Tomboy.WebSync.Api.OAuth.GetAuthorizationUrl() in C:\Projects\tomboy-gnome\Tomboy\Addins\WebSyncService\Api\OAuth.cs:Zeile 55. bei Tomboy.WebSync.WebSyncPreferencesWidget.OnAuthButtonClicked(Object sender, EventArgs args) in C:\Projects\tomboy-gnome\Tomboy\Addins\WebSyncService\WebSyncPreferencesWidget.cs:Zeile 165. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The English version of the above error message is: "The remote server returned an error: (411) length required"

This is most likely related to the fact, that Rainy's headers are missing the "Content-Length" field.

Fix: Add header.

Dynalon commented 11 years ago

Thanks for your very got bug report, I will look into it.

What I can tell now, the public test server uses an nginx reverse proxy in between, which will transfer the data via the Transfer-Encoding: chunked mechanism. And for chunked transfers, there are no Content-Length entries, that is normal. But the chunked encoding might be the reason for the trouble.

Dynalon commented 11 years ago

Commit 8b626c344690c5e0a8616f1d1733269b4f889d04 should hopefully fix the issue, found the solution here: http://stackoverflow.com/questions/15928202/servicestack-self-hosted-service-uses-chunked-encoding-is-unbuffered

Current master is in a non-releasable state, for testing I uploaded an experimental release here: https://dl.dropboxusercontent.com/u/14469783/rainy-0.2.4-experimental.zip If you can, please try this release and let me know if this fixes the windows sync issue. Note that the public server was not updated as I want to keep it only for stable releases.

arne1987 commented 11 years ago

Thanks for your quick response!

I downloaded and tried your experimental release. Unfortunately, it is still not possible to sync when using Tomboy under Windows with .NET. The log file shows the same error as before. I can see that the "Content-Length" header is available, but the .NET runtime still complains about missing length information! :-( Synchronization does work when using Mono, but Mono has some other issues on Windows (and most people don't use Mono on Windows).

Do you know about any other public services for Tomboy note syncing? (Ubuntu One note syncing is not available anymore - that's why I moved to Rainy). If I had a working sync service I could use wireshark to further analyse the problem and identify differences between a working service and Rainy...

Dynalon commented 11 years ago

I suspect the missing "Content-Type" header in the response might be the cause. I've rolled another build containing commit d3e5ddd339d79664d97202929022f61f89f52891 that you can download here: https://dl.dropboxusercontent.com/u/14469783/rainy-0.2.5-experimental.zip (md5: 76e53cf0a1682916b498cab13c591171). This build always send explicit Content-Type: application/json for every response.

Please try again. Also, a wireshark log might be helpful to me, you could attach it as a gist/pastbin. The only alternative I know of Rainy is snowy, which is unmaintained for a few years and uses python/django/mysql and is somewhat hard to setup on windows (a reason why I started Rainy).

ghost commented 11 years ago

Hello, I have the same problem with tomboy under windows. The sync with tomdroid and rainy works well. I have tested the experimental build 0.2.5 under Debian Linux. Tomboy still shows this issue in the logfile:

01.05.2013 21:43:30 [ERROR]: Failed to get auth URL from http://notes.regu.org:8080/user1/testpass Exception was: System.Net.WebException: Der Remoteserver hat einen Fehler zurückgegeben: (411) Länge erforderlich.bei System.Net.HttpWebRequest.GetResponse()

Do you need more infos from the logfile or something else?

danmoz commented 11 years ago

Hi, just tested rainy-0.2.5-experimental on an Ubuntu 12.04.2 server using Tomboy 1.14 for Windows as a client. Same problem unfortunately :-(

...at Tomboy.Tomboy.Main(String[] args) in C:\Projects\tomboy-gnome\Tomboy\Tomboy.cs:line 158 5/05/2013 5:23:16 PM [ERROR]: Failed to get auth URL from https://offworld.net.au:8080/xxx/xxx. Exception was: System.Net.WebException: The remote server returned an error: (411) Length Required.

Thanks for looking at this Dynalon!

trepidity commented 11 years ago

Do you think you could get a wire shark capture of the http traffic?

danmoz commented 11 years ago

Sure, TCP Stream from wireshark:

GET /test/test/api/1.0 HTTP/1.1
Host: offworld.net.au:8080
Connection: Keep-Alive

HTTP/1.1 200 OK
Content-Type: application/json
Server: Mono-HTTPAPI/1.0
Date: Sun, 05 May 2013 20:25:18 GMT
Content-Length: 358
Keep-Alive: timeout=15,max=100
{"user-ref":{"api-ref":"http://offworld.net.au:8080/api/1.0/test","href":"http://offworld.net.au:8080/test"},"oauth_request_token_url":"http://offworld.net.au:8080/oauth/request_token","oauth_authorize_url":"http://offworld.net.au:8080/oauth/authorize/test/test/","oauth_access_token_url":"http://offworld.net.au:8080/oauth/access_token","api-version":"1.0"}

POST /oauth/request_token HTTP/1.1
Authorization: OAuth realm="Snowy",oauth_callback="http%3A%2F%2Flocalhost%3A8000%2Ftomboy-web-sync%2F",oauth_consumer_key="anyone",oauth_nonce="1305557",oauth_signature="4dG%2bz2eAEWNnNahLl7u5IAyyTVE%3d",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1367785517",oauth_version="1.0"
Host: offworld.net.au:8080

HTTP/1.1 411 Length Required
Content-Type: text/html; charset=utf-8
Server: Mono-HTTPAPI/1.0
Date: Sun, 05 May 2013 20:25:18 GMT
Content-Length: 24
Connection: close
<h1>Length Required</h1>

Looks like the Tomboy client is indeed not sending a content-length header on the POST request.

I found a bug report that says the Mono HttpListener requires a content-length header on all POST requests: https://bugzilla.novell.com/show_bug.cgi?id=323215#c1

danmoz commented 11 years ago

Logged Tomboy bug: https://bugzilla.gnome.org/show_bug.cgi?id=699727

Dynalon commented 11 years ago

@danmoz the original bugreporter (@arne1987) had the issue only when using the Microsoft .NET runtime for Tomboy on Windows. Can you try using mono to run tomboy and see if this error still occurs to confirm that behaviour?

Maybe try the same with Rainy. If it is in fact the case that the HttpListener requires the Content-Length field but the MS .NET doesnt, this might also be a bug in mono.

danmoz commented 11 years ago

Yes my testing was also using .NET Tomboy client, same as @arne1987

Unfortunately I lack the expertise to compile Tomboy / Rainy with Mono on Windows :-/

Dynalon commented 11 years ago

see my comment on the gnome bugzilla: https://bugzilla.gnome.org/show_bug.cgi?id=699727#c1

Current MS .NET implemenation does not send a content-length header for empty messages, while mono does. That is why this problem only exists in the windows port of Tomboy. I can't fix that in Rainy, Tomboy's client logic has to be adjusted for Windows.

trepidity commented 11 years ago

@danmoz if I make Timo's suggested change would you be willing to test it?

danmoz commented 11 years ago

@trepidity Absolutely! Thanks

trepidity commented 11 years ago

I think Tammy Miller is going to try and tackle this for us. Lets give her a chance to roll a release for you.

Jared

On May 6, 2013, at 3:06 PM, danmoz notifications@github.com wrote:

@trepidity Absolutely! Thanks

trepidity commented 11 years ago

Timo,

Not sure what is going on here. Adding content-length = 0 didn't help. The error was still being thrown about content-length, but then I realized that Rainy is complaining about the method POST not being implemented. So I put a hack in for Tomboy so that during this part, when the postData.length = 0 to use Get instead of Post.

This worked.

Now I"m getting this.. when using a self hosted Rainy.

} 13/05/11_23:09:54.588 [DEBUG] Rainy.WebService.RequestLogFilterAttribute->RequestFilter - Received request headers:

13/05/11_23:09:54.590 [DEBUG] Rainy.WebService.RequestLogFilterAttribute->RequestFilter - Authorization: OAuth realm="Snowy",oauth_consumer_key="anyone",oauth_nonce="9876070",oauth_signature="9O%2fIG2goCE1BEqPNgqyB8W%2fcaWM%3d",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1368313794",oauth_token="797bbb46-95fe-433c-a7fc-bcab85965fec",oauth_version="1.0" 13/05/11_23:09:54.593 [DEBUG] Rainy.WebService.RequestLogFilterAttribute->RequestFilter - Host: 127.0.0.1:8080 13/05/11_23:09:54.613 [DEBUG] Rainy.WebService.ApiService->Get - ApiRequest received 13/05/11_23:09:54.615 [DEBUG] Rainy.WebService.ResponseLogFilterAttribute->ResponseFilter - Unserialized response data to send (JSV): { user-ref: { api-ref: "http://127.0.0.1:8080/api/1.0/janedoe", href: "http://127.0.0.1:8080/janedoe" }, oauth_request_token_url: "http://127.0.0.1:8080/oauth/request_token", oauth_authorize_url: "http://127.0.0.1:8080/oauth/authorize/janedoe/none/", oauth_access_token_url: "http://127.0.0.1:8080/oauth/access_token", api-version: 1.0 } 13/05/11_23:09:54.630 [DEBUG] Rainy.WebService.RequestLogFilterAttribute->RequestFilter - Received request at: /api/1.0/janedoe Deserialized data (JSV): { Username: janedoe } 13/05/11_23:09:54.635 [DEBUG] Rainy.WebService.RequestLogFilterAttribute->RequestFilter - Received request headers:

13/05/11_23:09:54.638 [DEBUG] Rainy.WebService.RequestLogFilterAttribute->RequestFilter - Authorization: OAuth realm="Snowy",oauth_consumer_key="anyone",oauth_nonce="2647224",oauth_signature="W4D1yJ0Ckt2C7fpv3QseeD2HgX4%3d",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1368313794",oauth_token="797bbb46-95fe-433c-a7fc-bcab85965fec",oauth_version="1.0" 13/05/11_23:09:54.641 [DEBUG] Rainy.WebService.RequestLogFilterAttribute->RequestFilter - Host: 127.0.0.1:8080 13/05/11_23:09:54.644 [DEBUG] Rainy.WebService.OAuth.OAuthRequiredAttribute->RequestFilter - trying to acquire authorization 13/05/11_23:09:54.651 [DEBUG] Rainy.WebService.OAuth.OAuthRequiredAttribute->RequestFilter - authorization granted for user janedoe 13/05/11_23:09:54.683 [DEBUG] Rainy.WebService.UserService->Get - CAUGHT EXCEPTION: Argument cannot be null. Parameter name: DBUser: 'WHERE ("Username" = 'janedoe')' does not exist 13/05/11_23:09:54.688 [DEBUG] Rainy.ErrorHandling.ExceptionHandler->HandleException - Argument cannot be null. Parameter name: DBUser: 'WHERE ("Username" = 'janedoe')' does not exist

trepidity commented 11 years ago

I tried against notesync.org using one of the public accounts and I am now able to sync properly from a Windows instance of Tomboy.

I'm still not sure this is a Tomboy issue though....

trepidity commented 11 years ago

So I tried implementing the POST method for OAuthRequestTokenRequest, but that appears to be more work than I can see at this time. So while implementing this method while the Tomboy length problem doesn't happen, but then nothing has implemented the

You can get a copy of Tomboy for Windows with maybe a fix that will work for both of us. https://dl.dropboxusercontent.com/u/11996020/Tomboy-rainy-debug.msi

I added this code. if (String.IsNullOrEmpty (postData)) webRequest.ContentLength = postData.Length;

Although, you must implement the POST method. If I change the code to if (String.IsNullOrEmpty (postData)) { webRequest.ContentLength = postData.Length; webRequest.Method = WebRequestMethods.Http.Get; }

Then I can make it all work.

trepidity commented 11 years ago

Diff of the possible fix. http://pastebin.com/4v8St9k1

tgmiller5 commented 11 years ago

I am going to call it a day as I am unable to get it to work for today. Thank you so much for the help

Date: Sat, 11 May 2013 17:39:50 -0700 From: notifications@github.com To: Rainy@noreply.github.com Subject: Re: [Rainy] Rainy's HTTP-responses are lacking "Content-Length"-header (#20)

Diff of the possible fix. http://pastebin.com/4v8St9k1

— Reply to this email directly or view it on GitHub.

trepidity commented 11 years ago

Sure thing. I won't be much available tomorrow.

Jared

On May 11, 2013, at 8:18 PM, tgmiller5 notifications@github.com wrote:

I am going to call it a day as I am unable to get it to work for today. Thank you so much for the help

danmoz commented 11 years ago

I managed to get a copy of Visual Studio and build Tomboy 1.15 from source with .NET. I tried synchronizing and confirmed the bug occurs in the latest version.

Then I implemented the fix suggested by Timo in the Tomboy ticket (modify the IF statement on line 249 of OAuth.cs). Ran a test sync, wireshark shows that Tomboy now appends "Content-Length: 0" on its POST requests:

POST /oauth/request_token HTTP/1.1
Authorization: OAuth realm="Snowy",oauth_callback="http%3A%2F%2Flocalhost%3A8000%2Ftomboy-web-sync%2F",oauth_consumer_key="anyone",oauth_nonce="9890219",oauth_signature="hRenwf4GSW%2fBWqDIY4LAhEWcDE0%3d",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1368405484",oauth_version="1.0"
Content-Type: application/json
Host: offworld.net.au:8080
Content-Length: 0

HTTP/1.1 200 OK
Server: Mono-HTTPAPI/1.0
Date: Mon, 13 May 2013 00:38:05 GMT
Content-Length: 134
Keep-Alive: timeout=15,max=99
oauth_token=9e1da1dc-2aaf-4a7c-abd1-601204a6dc45&oauth_token_secret=7504b382-c33f-4e69-8f39-9504fabb11e9&oauth_callback_confirmed=true

POST /oauth/access_token HTTP/1.1
Authorization: OAuth realm="Snowy",oauth_callback="http%3A%2F%2Flocalhost%3A8000%2Ftomboy-web-sync%2F",oauth_consumer_key="anyone",oauth_nonce="2226684",oauth_signature="%2bAVYG3oDJ9PuGPAw5h8aaiIoASY%3d",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1368405486",oauth_token="9e1da1dc-2aaf-4a7c-abd1-601204a6dc45",oauth_verifier="fb9f70b7-fedb-429d-aadd-e1400e22ca68",oauth_version="1.0"
Content-Type: application/json
Host: offworld.net.au:8080
Content-Length: 0

HTTP/1.1 200 OK
Server: Mono-HTTPAPI/1.0
Date: Mon, 13 May 2013 00:38:06 GMT
Content-Length: 104
Keep-Alive: timeout=15,max=98

This fixes the issue ("The remote server returned an error: (411) Length Required") for me. Synchronisation now works perfectly!! Thanks for your help @trepidity and @Dynalon.

I'll see if I can figure out how to create a patch, and update the tomboy ticket.

danmoz commented 11 years ago

I just noticed that @trepidity already provided a patch in this thread. I've just applied this as best I could (adding the two lines manually, VS doesn't seem to support patch files) and this solution also works perfectly. Thanks

trepidity commented 11 years ago

Sweet!

danmoz commented 11 years ago

Sorry, I just did I bit more testing, the patch Jared posted does the authorization part successfully, but the sync process throws an error:

13/05/2013 12:26:35 PM [ERROR]: Caught exception. Message: Object reference not set to an instance of an object. 13/05/2013 12:26:35 PM [ERROR]: Stack trace for previous exception: at Tomboy.WebSync.Api.OAuth.MakeWebRequest(RequestMethod method, String url, List1 parameters, String postData) in D:\Dev\tomboy\Tomboy\Addins\WebSyncService\Api\OAuth.cs:line 260 13/05/2013 12:26:35 PM [ERROR]: Rest of stack trace for above exception: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at Tomboy.WebSync.Api.OAuth.MakeWebRequest(RequestMethod method, String url, List1 parameters, String postData) in D:\Dev\tomboy\Tomboy\Addins\WebSyncService\Api\OAuth.cs:line 268

The change Timo suggested in the Tomboy bug seems to work OK for both authorisation and sync.

danmoz commented 11 years ago

Attached a patch to the Tomboy bug.

Dynalon commented 11 years ago

closing as it is fixed in the tomboy side.

igorsantos07 commented 10 years ago

Sorry for bothering on such an old bug, but I'm trying to put Tomboy to work on a Windows Vista machine. I installed the last stable version from the website, which did not work with Rainy 0.2.3 - as in this issue, says it could not connect. I tried them to install the debug version posted in a dropbox link here. It does connect to the server and I can authenticate just fine, but it does not synchronize, stating almost instantly it could not synchronize the notes.

Here's what happens in the log error when I try to sync: http://pastebin.com/yCqs6abL