Websocket support

[JavaJens]

As #127 is a little overloaded and goes beyond the technical, I thought to open up this issue to gather implementation details and coordinate efforts for this.

Support for WebSockets is already present on the server, so the idea is to use WebSockets on Android as well (for now instead of a possible more suited protocol like MQTT)

@geileszeuch mentioned a guide on how to use websockets in native apps: http://www.elabs.se/blog/66-using-websockets-in-native-ios-and-android-apps

However I don't think this would be a good guide as it doesn't talk about Wake/Sleep and connection loss. This article list several issues one has to deal with in a mobile enviroment: http://dalelane.co.uk/blog/?p=1599 I believe the approach mentioned here http://stackoverflow.com/a/16602970 to be reasonable, if I find the time I will give it a try.

I guess for developing this one has to run an own server, as WS is disabled per default.

So please, if you have a question, concern or are already working on this: comment!

[ghost]

Is there any technical reason to use web technology ? why not any XMPP like ?

[JavaJens]

There is already support in the Server :)

[ghost]

As you said, a mobil environment is likely to break connections and thus we will be force to reimplement what already exists in session protocol like XMPP or SIP that is registration, session management, etc.

I think it would be easier to develop something on top of a session protocol even though it will require some slight modification on the server side.

[JavaJens]

Do you have anything particular in mind? I haven't checked the code, but maybe the GCM server part is using XMPP already which could be reused? I know @v-0-d mentioned MQTT in the other thread, which sounds pretty good as well.

Nevertheless, I think the current server can handle session mobility to some extend, but I haven't looked very thoroughly. From what I see the client simply has to connect (and reconnect if needed) to the server and listen to incoming messages which are queued until delivered. And as a client as a clear identification number the mapping to a WebSocket connection shouldn't be such a big deal.

[JavaJens]

I've created a local branch with very rudimentary support, more a PoC: https://github.com/JavaJens/TextSecure/tree/feature-1000 Maybe @thors you can take this as a starting point? @TheBlueMatt is working on the desktop client afaik. Maybe he can provide some insight to the registration for WebSocket devices?

[geileszeuch]

He recently open sourced bis early version of the textsecure browser extension. Hopefully this will provide some insight: https://github.com/TheBlueMatt/textsecure-chrome

[JavaJens]

Yes, I looked into that. right now multiple devices can be registered on one number.

For the Android version i've got to make little changes therefore.

Right now the device connects and all seems well, I only have to test sending/receiving messages which doesn't work right now.

Probably because I only have one device and try to work with the emulator and a modified server version.

If you want you can checkout my feature branch and test yourself :)

cheers

[Manuel-Kehl]

Thank you for your efforts! I will checkout your branch as soon as my current exams are over and maybe even contribute something.

I hope that we will soon be able to (optionally) use TextSecure without GCM (and PlayStore in general!).

[SecUpwN]

@JavaJens, thank you for opening up this issue, I can't wait for this to be implemented! ;-)

[JavaJens]

Feel free to test by branch and contribute. Sadly I'm pretty busy right now, but rudimentary support exists :)

[SecUpwN]

@JavaJens, is there a compiled WIP-version somewhere or do I have to compile it by myself?

[JavaJens]

Sadly, no!

[deutrino]

Just wanted to comment after reading https://missingm.co/2014/02/fighting-dishfire-the-state-of-mobile-cross-platform-encrypted-messaging/ that XMPP and WebSockets will probably work, sort of, but will be painful on mobile devices due to frequent IP switching, sleep/wake, etc.

Unfortunately this is not (yet) a library that anyone can use, or tunnel over like ssh, but have a look at http://mosh.mit.edu/ - it's a partial ssh replacement (terminal emulator only, no tunneling or forwarding yet) which has solved ssh's problems with IP switching and sleep/wake, while managing to get around many many firewalls and NATs. Their code itself probably can't be used, but the ideas may be worth reading over.

[seroma]

@JavaJens Great Work you are doing!!! Me and quiet a couple of friends are longing desperately to get their hands on a GCM free TextSecure version. I am so glad to see that you are making it happen. This means really the world to us, since installing Google Play is absolutely no option for us. We are all really excited and following your efforts day to day. Is there anything we can do to help and speed up development other than coding (since we don't have any coding experience)? Since the day TextSecure with data came out (WhisperPush for CM) we all are dreaming of the day we can enjoy this wonderful app with all our friends. Thank you already for all you have done. You are a hero :).

:+1: for Websocket support for TextSecure.

[SecUpwN]

@seroma, I second that. Awesome work, @JavaJens! Keep rockin'! :smile_cat:

[Wikinaut]

Just in addition https://devcenter.heroku.com/articles/websocket-security

You should strongly prefer the secure wss:// protocol over the insecure ws:// transport. Like HTTPS, WSS (WebSockets over SSL/TLS) is encrypted, thus protecting against Man-in-the-Middle attacks. A variety of attacks against WebSockets become impossible if the transport is secured.

[JavaJens]

I've updated my branch. The most recent updates feature a different package name for parallel installation, increased debug output for the Wakelocking and fixed an error for a minor issue with a Wakelock not being released.

However one of the main issues, severe battery drain, I could not find a reason for; my phone does not go into deep-sleep mode when WiFi is enabled. This has nothing to do with TextSecure though, Wakelocks are held for about 10s in 2h. As I don't have 3G I can't test the battery consumption without Wifi. I would suspect this being an issue with my CM11.

The last issue for the high battery drain is the low timeout frequency of @TheBlueMatt's server of about 50s; that is we have to send a ping every ~50s, which is far to often. In real life, this could be decreased to 30min or so.

One last thing I want to address, @Wikinaut: this build is purely for testing and uses the non secure WS for developing. I've updated the Readme to point this out more clearly. In the end certificate pinning will of course be used!

Please report any issues back here or as an issue in my fork. I don't know if you can use the "Report a Bug" feature or have to use ADB....

[moxie0]

The last issue for the high battery drain is the low timeout frequency of @TheBlueMatt's server of about 50s; that is we have to send a ping every ~50s, which is far to often. In real life, this could be decreased to 30min or so.

With RedPhone, we've found that mobile data networks often require you to send keepalives as little as every 15 seconds. So 50s is potentially too long.

[JavaJens]

Ok...that comes as a surprise.... Current GCM timeout is apparently ~15min on Wifi, which is stated here as being to high: https://productforums.google.com/forum/#!msg/nexus/fslYqYrULto/lU2D3Qe1mugJ

That would support your findings! How about the Battery life with RedPhone? As I said my phone doesn't deep sleep with Wifi on...

[moxie0]

WiFi is unpredictable, since it depends on the NAT. So you'll have to deal with that the same way GCM does. The advantage GCM has is that Google has made agreements with mobile carriers not to timeout connections on those networks. Without those, websocket connection idle timeouts on mobile data connections will be unpredictable, and they tend to be more trigger happy (and also often just silently close without sending an RST). I think some kind of adaptive solution is probably going to be necessary.

[geileszeuch]

As I don't have 3G I can't test the battery consumption without Wifi. I would suspect this being an issue with my CM11.

I am building your branch right now. I will test on WiFi and 3G and report as soon as possible (probably in around 10 hours). I am on CM10.2 so let's hope the battery drain really is just a problem of CM11.

[JavaJens]

@geileszeuch Maybe you can check the wakelocks for TS and the deep sleep with/without TS on WiFi/3G.

And report any other issues u have :)

[geileszeuch]

After 10 hours of use Android's build in battery stats show that TextSecure only had 2min keep awake and did use only 2% battery. The thing is that Android OS has a keep awake time of 9 hours and 30min, which used almost the entire battery (from 100% to 20% in 10 hours without any use of the phone). This was on 3G. Further tests with WiFi will follow.

UPDATE: Those results are not very helping, because I noticed another app was running in the background, which might have influenced this drain drastically. More to come soon.

[JavaJens]

@geileszeuch This behaviour of the OS is the same for me on WiFi... Although keeping WiFi enabled is disabled.

[agrajaghh]

Perhaps you should have a look at K-9 Mail. I'm using it with three different IMAP accounts with push enabled. And push is working there with a imap idle connection to the server. I don't have significant battery use, so it should be possible for Websocket aswell. no?

[h-2]

Just hanging out at the FSFE Germany meeting, and we really think this bug report is important. Thanks for your work in trying to make TextSecure work without GooglePlay, it is really appreciated!

[JavaJens]

I will look into k9. Especially regarding the timeout and heartbeat interval. The aosp stock client does the same, but I think without wake locks.

Maybe some of you could test the battery usage, as the wakelocks being held aren't held for long... Furthermore having a group of friends sitting around is also great as no one else is on the server....

You can register phony numbers as well, they're confirmed automatically. I'm there as +49123456666

[1200200200]

I am registered as +1200200200 on the server.

@JavaJens Is your number +49123456666 correct? I can't send anything to you. Maybe you also just unchecked Push Messages and only check it when doing some tests/developing? I'd like to exchange messages for tests and maybe create a group for everyone willing to contribute to battery/bug testing of websockets branch.

[SecUpwN]

@JavaJens and @1200200200, why are you posting your private phone number publicly?

[agrajaghh]

I don't think these numbers are real ;) They just registered with this fake number...

[JavaJens]

Yes, they are fake. As I said the server verifies every number :) @1200200200 How does sending of the message fail for you? I've sent you a message which appears to be sent by the server....

[1200200200]

Saying Error sending message under the message in a green box with a red triangle on the left. Try again please. I reregistered. last time I did uncheck Push Messages.

[TheBlueMatt]

Note that the server has gotten reset a few days ago since the database was reaching the free size limit.

On June 15, 2014 3:28:22 PM EDT, 1200200200 notifications@github.com wrote:

Saying Error sending message under the message in a green box with a red triangle on the left.

---

Reply to this email directly or view it on GitHub: https://github.com/WhisperSystems/TextSecure/issues/1000#issuecomment-46125191

[JavaJens]

I also reregistered... Can send messages to you, but don't know if they're delivered... If you still have issues, try getting an adb log or so....

[1200200200]

Got your messages and sent you two messages 25 minutes ago. Did you get anything?

[1200200200]

I am now 6 hours on battery and went from 100 to 90 percent while doing nothing and only having one other app running which I assume used maybe 5 out of these 10 percent. This is on 3g. Pretty decent I would say.

[geileszeuch]

I am now 6 hours on battery and went from 100 to 90 percent while doing nothing and only having one other app running which I assume used maybe 5 out of these 10 percent. This is on 3g. Pretty decent I would say.

I got the same experience. 6 hours on 3G with running apps mail, chatsecure and Textsecure-Websockets-branch, and my battery went from 100 to 93 percent. This is awesome and probably even better than with GAPPS installed and TextSecure over GCM. Doing WiFi test at the moment. Am done in about 4 hours.

[McLoo]

during your tests, do you have TS just idle or do you send/receive messages to?

[geileszeuch]

For now I am just testing it running in the background without doing anything else, so no message exchange yet.

[geileszeuch]

Results of my WiFi test are: 6 hours on WiFi doing nothing and the only apps running in the background being Mail, Chatsecure and TextSecure let my battery go from 100 to 95 percent. These are very good results in my eyes.

[1200200200]

Results of my WiFi test are: 6 hours on WiFi doing nothing and the only apps running in the background being Mail, Chatsecure and TextSecure let my battery go from 100 to 95 percent. These are very good results in my eyes.

I got the exact same result.

[1200200200]

during your tests, do you have TS just idle or do you send/receive messages to?

@McLoo The WebSockets branch only adds websockets for receiving messages so I guess the battery consumption for sending messages should be just the same as for the official TS app.

[grote]

@JavaJens Thanks a lot for working on WebSockets support! This is really appreciated!

At the FSFE German Team meeting, we tested your app branch with various people, but messages only arrived with a long delay and some people never received messages while other people's messages never arrived. It was hard to see any pattern.

Now at home, I was testing more and noticed the following message. Maybe it is related to the problems we were seeing?

W/SystemStateListener(16014): java.lang.IllegalArgumentException: Receiver not registered: org.thoughtcrime.securesms.service.SystemStateListener$ConnectivityListener@4301d518
W/SystemStateListener(16014):   at android.app.LoadedApk.forgetReceiverDispatcher(LoadedApk.java:674)
W/SystemStateListener(16014):   at android.app.ContextImpl.unregisterReceiver(ContextImpl.java:1511)
W/SystemStateListener(16014):   at android.content.ContextWrapper.unregisterReceiver(ContextWrapper.java:489)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SystemStateListener.unregisterForConnectivityChange(SystemStateListener.java:46)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SmsSender.unregisterForRadioChanges(SmsSender.java:180)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SmsSender.handleSentMessage(SmsSender.java:145)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SmsSender.process(SmsSender.java:64)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.service.SendReceiveService$SendReceiveWorkItem.run(SendReceiveService.java:273)
W/SystemStateListener(16014):   at org.thoughtcrime.securesms.util.WorkerThread.run(WorkerThread.java:46)

[JavaJens]

@grote

I noticed the missing/delayed messages as well. For me I didn't get any errors when sending and was always connected when other parties send delayed messages.

I'll check the error message you send. Maybe @TheBlueMatt can check the error messages on the server?

[grote]

@JavaJens Thanks for the quick reaction! :)

It is a good idea to ask @TheBlueMatt about the server error logs to get to the bottom of these problems. We also did not get any errors, but still the messages did not arrive or arrive only delayed.

Since, there might be many people who want to help with testing without app building skills, I created an F-Droid repository for the GCM-free branch. Just get F-Droid and add a new repository with the following URL:

http://grobox.de/fdroid/repo

The only downside is that this requires version code increases to get new versions to people.

This picture was made at the FSFE Germany meeting last weekend:

[JavaJens]

I just updated to the most recent upstream changes. However it seems as if the server is down/damaged, one can't connect to the WebSocket URL. Maybe @TheBlueMatt can give some information on this?

[grote]

Maybe the client should throw an error when the WebSocket URL can't be reached instead of silently failing?

Would you mind bumping the version code so users of the F-Droid repo can receive a new version?

[JavaJens]

I don't know if a message is needed, but the least that is needed is some kind of exponential backoff so that we don't loop and continue to fail.

I'll try to make that later today.

[1200200200]

However it seems as if the server is down/damaged, one can't connect to the WebSocket URL.

This would explain the higher battery usage I had in the last 10 hours.

I don't know if a message is needed, but the least that is needed is some kind of exponential backoff so that we don't loop and continue to fail.

That is a very good idea. It should prevent battery drain in such a case.

[SecUpwN]

@JavaJens, you ROCK! :smile_cat: Quick question: Will these changes be incorporated onto the GooglePlay version, too? Or do I have to re-install TextSecure through F-Droid?