Reduction of Bandwidth/Traffic Consumption

[srkunze]

Is there something we can do about it?

https://github.com/subliun/Antox/issues/6

[ollieh]

Mobile apps aren't really supposed to be constantly connected to servers, but to use the push notification system the OS provides. I can't think of a way to do this without using a centralized server. The best situation I can think of is a centralized server just for sending the push notifications, and then the mobile app sends a message to the central push notification server authorizing the TCP relay it is connected to to send everything it receives for that user to the push notification server. It's not such an issue on Android, because you can have stuff working in the background, and we were getting half-decent battery life at one point so it is possible. You can't get anywhere near the battery efficiency of push notifications though, and for iOS it's a complete showstopper. On iOS you can't have things running in the background for more than a few seconds or maybe a minute, so a messaging app on iOS can't work in the background, and must use push messages if it wants to actually be useful. I think it also means user-written messages would have to go through the push system, not just friend requests and such. I may be wrong but it doesn't look promising. Can anyone see an easier way out?

[srkunze]

@ollieh I think what you said is really important. Not sure how push notifications work. If the app can configure the push server freely, it could be changed to respective peers regularly. Could this work?

I want to emphasize the fact that mobile will be (is already?) the dominant way people use chat/voice/video applications. A solution here is crucial (at least in my opinion). Most people will never (exaggerated of course) use two systems (Skype+Tox) because of the network effect (friends etc.).

[ollieh]

From the android docs:

Send data from your server to users' Android-powered devices This could be a lightweight message telling your app there is new data to be fetched from the server or...

It seems like this approach might work. The TCP relay could hold the messages for a little, send a message to the push server to tell the user's client to query the TCP relay, and then pass the messages on when it does. That way you wouldn't necessarily need to authorize each TCP relay as the worst thing they could do would be constantly send push messages that tell you to query the network, to drain your battery. As it'd be a central server controlling it you could just have a timer on it. I wish there was a way to do this without a central server though.

I totally agree with you about mobile. If Tox is to succeed, it has to be fantastic on mobile. Everyone seems to have their sights set on beating Skype, but we can't beat Whatsapp yet. iOS is impossible at the moment, and I don't know about you, but most of my friends have iPhones, and Tox isn't very useful if your friends can't use it. I know most of us probably prefer using desktop applications, but if it's desktop only it'll never take off. If it just did mobile well it'd stand a much, much better chance, even if the desktop clients were shitty or non-existent. We need to solve push messages, and offline messages.

[tttom]

@ollieh, I second that. I'd say that the average internet user does not have a computer anymore, but almost certainly just a smartphone of some kind. Battery life and data usage cost of smartphones practically mandate the use of iOS push notifications or Google Cloud messaging. In principle one could sign up to one or more message services of choice, which would keep encrypted messages and send push notifications to their phone. Although I believe that the push notification service is free of charge, running a message service server to send the push notifications would not be entirely free. Thing is, I don't see any incentive in running such a message service unless it is for oneself or ones friends. Bittorrent works because uploaders get somehow rewarded with faster download speeds. Free e-mail providers get to peek into your messages to increase their ad revenue. But here, message service providers would just pay for the bandwidth of smartphone users. Hence, my honest question, is there anything that smartphone users can offer in return for good battery life and minimal data costs?

[srkunze]

@tttom You are right. Not sure if there could be a charity organisation which can collect some money and distribute it to node operators to cover their expenses.

However, we should give people a sensible option first on mobile before thinking about compensation.

[srkunze]

on-topic @dubslow estimated here https://github.com/irungentoo/toxcore/issues/1254#issuecomment-74318898 that one package/message could be 1-2 KiB.

Unfortunately, the tablet traffic monitors tells us about 50 KiB per message. What is wrong here? Is onion routing increasing the traffic so much?

[dubslow]

There are indeed more packets than just the message packets...

[srkunze]

25 fold more?

[Kolargol00]

I'm seeing Antox use a huge amount of data on wifi: in the range of 10MB to 50MB per hour while I'm not using the app at all. Is there any number/log I can report to help you guys diagnose this?

[fcore117]

i hope there are more TCP optimizations, could be useful especially for 2G, 3G, 4G internet users and for very low bandwith ADSL users.

[dcharles525]

This has to be fixed, or Tox on the mobile platform will fail.

[DanTheBritish]

Tox on desktop can also be under threat, for people with data caps on their broadband (it's more common than you think)

[fcore117]

LapFox: this is very common outside of city like farms and houses in woods, mobile broadband is only option for desktop. I myself even have helped some people to set up usb stick to get better signal. Most common monthly plan is 5-20GB and only very rich could get 100GB to infinite bandwidth.

I really hope this is resolvable without sacrificing p2p, servers should go to hell as they are easily banned, filtered, blocked.

[srkunze]

Is there anything the community can do here?

[Kolargol00]

Disabling UDP and installing a recent Antox build brought down data usage to much lower levels (< 1MB / day). Unfortunately, I can't say which operation changed that since I did both together. Scrap that :( I opened Antox then sent 2 messages and data usage is now more than 100MB in 4 hours.

[srkunze]

What the hell?

[lovetox]

you should give this issue the highest priority

as some said before, tox can and never will be the future of online communication, if it is unusable on smartphones.

[grinapo]

Maybe it would help to track down the issues to generate stats about the various kinds of traffic tox uses and see what uses how much, and after it could be checked why and whether it can be shrunk or not. My guess (without looking) is that it may be an inherent design problem with the protocol and would go the same as bleep: unusable on mobile due to extreme high protocol traffic overhead.

[optimumtact]

This issue has become much more important, because the upcoming versions of android will require apps to properly sleep and only wake up on messages from google push notifications.

AFAIK IOS already does this

[ameenross]

They usually parasite on a "geek" and see us as mere tools.

QFT :laughing: "Geeks" should probably just charge more! (Sorry for offtopic, btw)

[subliun]

There is no p2p solution to this problem. Antox will adopt GCM to send wakeup pings/messages sometime in the future, which should mostly remedy the bandwidth issue. I am sure improvements can be made in toxcore to improve bandwidth usage also.

[quininer]

@subliun GCM? Can you describe how it will be implementation? It requires a special server?

[GrayHatter]

@subliun irssi has a nice plugin that encrypts the messages using a shared key so that neither the app, nor google can see what's in the message. I see no reason you couldn't just push the encrypted messaged to your app.

[subliun]

@GrayHatter everything going through GCM will be encrypted of course.

@timofonic The F-Droid build will likely never contain any GCM-related functionality. Unfortunately, due to restrictions imposed by Google in Android M, Antox is forced to adopt GCM or be left crippled and unable to receive phone calls, file transfers or messages in the background. Because of this, the Google Play build of Antox will be using GCM to deliver messages and updates in the background.

[tttom]

@subliun, as sending Google Cloud Messages requires each push server to have a Google API key, how would this be implemented for Antox? Ideally all tox nodes should be able to send GCM to antox users, though I don't see it happen that everybody will set up an account with Google, and later Apple etc. Would each Antox user have to run such a push server themselves or would there be a hard coded antox.net server perhaps?

Alternatively, with modifications to toxcore, one could designate one or more relays to be contacted through. Google Play users could register with a relay that is capable of reaching them by GCM and inform their contacts of this, i.e. an extended toxid. Relays would only see encrypted messages destined for that specific Antox user, hence one doesn't necessarily need to run their own service. The ability to specify multiple relays would enable high availability with low cost hardware, and perhaps even offline messaging. The configuration could remain as simple by setting up a default in the client. Users that don't like relays can specify no relay, and watch their battery drain... The designated relay principle could also be used for iOS as well as any open source alternatives for GCM.

[subliun]

Would each Antox user have to run such a push server themselves or would there be a hard coded antox.net server perhaps?

This is accurate. My current plan is for some extension in toxcore to allow telling your friend somehow that you want your messages relayed over GCM. Then when you go offline messages will be sent to an offline messaging server/TCP relay server/to be determined, which will forward to an antox.me server set up to handle rate-limiting and abuse prevention. This server will have the GCM api key and will send the messages/wakeup updates through GCM which will be received in the background by the recipient. When Antox is in the foreground (thus it is online) messages will be sent directly as they are now.

I'm open to future extensions which allow more customisation or specification of custom forwarding servers, though I'd like to get the basics worked out first.

[fcore117]

so Antox will not work with TCP mode too to send files and so on like toxcore should?

I am user who has no Google services installed, pure slim barebone.

[subliun]

@fcore117 it will, but not in the background. Please read https://commonsware.com/blog/2015/06/03/random-musing-m-developer-preview-ugly-part-one.html as it covers the issue very well.

[tttom]

@subliun, granted, multiple relays would perhaps be a bit more complex. I totally agree to try to keep things as simple as practical. Since you are already considering extending toxcore to tell friends they can relay messages over GCM, why not just tell them they can relay over a given toxid? That way the node at the specified toxid can implement the push notification system of choice. Antox could default to specify a toxid where antox.me sits, while future clients can do as they please. Am I missing something, or isn't this less complex than something GCM/antox.im specific?

[xavierle]

@subliun maybe we can implement ourbown push server and not gcm

one example https://github.com/WhisperSystems/PushServer

some others are existing

[fcore117]

would it not just be possible to make something like "idle TCP" mode that is great when client is in idle mode and when chat begins it tries to make direct connection or relayed depending of connection?

Devs what are your thoughts currently? i really hope it could be internal feature without any servers.

[LuccoJ]

Please don't use GCM. Make, and keep, this app fully free and open, with no proprietary dependencies. Doing otherwise would defeat any attempt to have a really secure (and transparent) messaging system. I'm sure if enough developers refuse to cave in to GCM to sidestep Doze and App Standby, something will be done at Google or elsewhere to fix the matter.

[xavierle]

maybe mozilla notifications (simplepush or webpush api ) may be an optional function to save bandwidth

until a better way is found

at least i would prefer a mozilla api or protocol than a Google one Le 27 déc. 2015 20:03, "LuccoJ" notifications@github.com a écrit :

Please don't use GCM. Make, and keep, this app fully free and open, with no proprietary dependencies. Doing otherwise would defeat any attempt to have a really secure (and transparent) messaging system. I'm sure if enough developers refuse to cave in to GCM to sidestep Doze and App Standby, something will be done at Google or elsewhere to fix the matter.

— Reply to this email directly or view it on GitHub https://github.com/irungentoo/toxcore/issues/1251#issuecomment-167436156 .

[LuccoJ]

@xavierle I don't know about those, but I suspect only Google (or at least something installed in the Android firmware itself) can provide push notifications that will override Doze and App Standby. Other libraries will be subjected to the same limitations that normal apps have under Android 6.0.

[GrayHatter]

The problem isn't data, that's easy. The issue is cpu cycles, and there by battery usage. We need a way to maintain a connection to [thing] that can sleep aggressively, wake up the system when it needs to, and is simple to implement and maintain.

The reason GCM fits that bill is because it has a good team behind it (Google), an easy API, and is built DEEP into the Android OS.

Also GCM is FOSS btw... https://github.com/google/gcm/blob/master/LICENSE so maybe avoid the FUD?

[LuccoJ]

@GrayHatter "This project contains client libraries and samples to help developers interface with and explore the Google Cloud Messaging APIs." is straight from the thing you linked. GCM itself (both the server and the API on Android) is most definitely NOT free software, as anyone behind F-Droid will tell you if you ask them.

It is not "built into" Android at all, as in fact, it's not part of AOSP ("Android"), but only of the Google Play Services. This is, again, something anyone from F-Droid will confirm (try asking in #fdroid on freenode if you like).

You can also check out http://forum.xda-developers.com/android/apps-games/app-microg-gmscore-floss-play-services-t3217616 which is a free partial Google Play Services replacement, which, again, shows that the original GCM is closed. Even though this provides a free replacement to the end user, applications currently still need to be built against the proprietary component. In any case, Google push servers are used for the job, which run Google's proprietary software. MicroG also has a channel on freenode, #microg, where the author or someone can no doubt explain what is proprietary about the Google Play Services including GCM.

Note that even the developer of Antox, above, says by necessity that "The F-Droid build [of Antox] will likely never contain any GCM-related functionality".

[xavierle]

like every google service advantage is easy to use. but gcm is clearly not free.

and i like im project without gcm like conversations, freelab messenger and antox.

including gcm would be a mistake in my opinion.

if a push service is necessary to lower the battery consumption. maybe we can use our own push infrastructure (i sent links to some foss push servers)

or maybe rely on a third party infrastructure that we can trust enough like Mozilla for example. ( i think both the servers and api are open source from mozilla)

most of all this kind of function for a third party server shall be optional so that every user is free to desactivate it

Xavier Le 27 déc. 2015 21:41, "LuccoJ" notifications@github.com a écrit :

@GrayHatter https://github.com/GrayHatter "This project contains client libraries and samples to help developers interface with and explore the Google Cloud Messaging APIs." is straight from the thing you linked. GCM itself (both the server and the API on Android) is most definitely NOT free software, as anyone behind F-Droid will tell you if you ask them.

It is not "built into" Android at all, as in fact, it's not part of AOSP ("Android"), but only of the Google Play Services. This is, again, something anyone from F-Droid will confirm (try asking in #fdroid on freenode if you like).

— Reply to this email directly or view it on GitHub https://github.com/irungentoo/toxcore/issues/1251#issuecomment-167440228 .

[LuccoJ]

@xavierle As I said, you can use all the alternative push infrastructures you want, but that won't let them bypass Doze like GCM can do.

[GrayHatter]

Unless someone can give me a reason, or a direct toxcore issue, I'm very tempted to close this as either off topic, or as an Antox issue. However Antox want's to fight with Doze isn't really toxcore's concern. Did I miss something?

[LuccoJ]

The reason is in the issue's title, "Reduction of Bandwidth/Traffic Consumption", which refers to the fact that Tox itself consumes a large amount of traffic, regardless and irrespective of Doze or even Antox in particular. The amount of traffic involved has been mentioned at https://github.com/subliun/Antox/issues/6

[optimumtact]

plus any solution will almost certainly have impact on the rest of the clients

[GrayHatter]

@LuccoJ I get that, but there's no description of what would be needed in a fix. Such as what's the maximum amount of data y'all would consider acceptable from a P2P client?

[LuccoJ]

Well, I'm not the person who reported this issue so my opinion may not match theirs, but I think that at least in a mobile (or generally capped/metered) scenario, ideally the amount of data transferred should only be a function of the communication between the user and their friends, and not increase "by itself" if no user-initiated communication is taking place.

So I guess one question is whether it would be possible to have a mode that mobile clients can pick where you're not a passive part of the DHT.

Aside from that, generally speaking, mobile users usually have at least a monthly 1GB allowance, and I suppose many of them would consider it unreasonable for a "lightweight" application like IM to consume more than 10% of it, which would make around 3MB a day of idle usage an upper limit of reasonableness.

[aaannndddyyy]

I'm not a dev but a normal user. What I expect from a lightweight messaging system is not to use more bandwidth than WhatsApp does. So for me a reasonable upper boundary not to be surpassed would be twice the usage of that.

What I don't understand with regards to those push notifications is why you would need a server at all. What is it a server can do what a normal client cannot do? Except by doing messages from different senders and send them in bulk to the recipient. The p2p concept, and thus serverlessness is imho a key-part of tox and should not be sacrificed. Even offline messages can be done serverless.

[aaannndddyyy]

We already have TCP relays. If they have a good uptime, they could easily maintain an idle TCP connection to mobile clients, such that the latter don't have much of battery drain. The mobile clients should opt out of most activities, e.g. participation in the dht or relaying messages in groupchats - something that is necessary if you want group chats to scale but with which you do not want to burden mobile users.

[xavierle]

maybe i expressed me wrong

when i was saying servers this was to differenciate mobile light client and any stationary stronger machine.

i agree with what said. mobile client shall be able to disable some functions (dht and relaying messages because of battery and bandwith)

stationary clients shall keep those functions acting as a p2p mesh of servers not a unique central server. in my idea every stationary client could act as a push server not a unique central one.

to avoid necessity for push

maybe the mobile client shall check for messages time to time (define by user) and being idle the other moment. and when antox is being used then it is in more active connection with other peers. Le 28 déc. 2015 06:28, "aaannndddyyy" notifications@github.com a écrit :

We already have TCP relays. If they have a good uptime, they could easily maintain an idle TCP connection to mobile clients, such that the latter don't have much of battery drain. They should opt out of most activities, e.g. participation in the dht or relaying messages in groupchats - something that is necessary if you want group chats to scale but with which you do not want to burden mobile users.

— Reply to this email directly or view it on GitHub https://github.com/irungentoo/toxcore/issues/1251#issuecomment-167487309 .

[LuccoJ]

@aaannndddyyy The issue being discussed wasn't "push" notifications in general, but the particular fact that, under Android 6.0, and in certain circumstances, GCM push notifications (only centralized GCM ones, not just any push-style notification) can be the only way to wake up a device while it's "dozing", which is a new Android concept.

There is a whitelist for applications that must not "doze", which the user can enable, but it's not wholly clear to me which abilities get re-enabled for apps that are added to the whitelist. https://developer.android.com/training/monitoring-device-state/doze-standby.html#support_for_other_use_cases has more information.

But GrayHatter is right that this (the "push" / GCM / Android part) should probably be discussed as an Antox-specific issue, not here at Toxcore.

[srkunze]

@GrayHatter @subliun I am uncertain where the actual problem lies and thus where to a solution should be applied and how it should look like. This was the initial reason why I first opened it on the Antox repo, then were redirected to toxcore by @Astonex.

Anyway, I am all in for closing this issue (e.g. splitting up into more tangible issues if necessary).

[LuccoJ]

@srkunze As long as it's not written off as a non-issue, though... because the problem is most definitely there for mobile clients, and it was called a toxcore issue that the Antox developer could do nothing about at https://github.com/subliun/Antox/issues/159#issuecomment-139843234

Without knowing many implementation details, I tend to think that the right solution would be for toxcore to provide a "lightweight" mode of operation, possibly one where it tends to only connect to TCP relays if at all possible, and takes part in the DHT as little as possible. Constant internet usage is simply not acceptable (often for compelling battery reasons even more than data cap reasons) on mobile clients, and mobile clients are very important for any IM service today.

[xavierle]

I agree with lucco

there shall be a light client for mobile device (with bandwith and battery limitations) and a full p2p client function for machines with no power or bandwith limitations

Le 28 déc. 2015 17:47, "LuccoJ" notifications@github.com a écrit :

@srkunze https://github.com/srkunze As long as it's not written off as a non-issue, though... because the problem is most definitely there for mobile clients, and it was called a toxcore issue that the Antox developer could do nothing about at subliun/Antox#159 (comment) https://github.com/subliun/Antox/issues/159#issuecomment-139843234

Without knowing many implementation details, I tend to think that the right solution would be for toxcore to provide a "lightweight" mode of operation, possibly one where it tends to only connect to TCP relays if at all possible, and takes part in the DHT as little as possible. Constant internet usage is simply not acceptable (often for compelling battery reasons even more than data cap reasons) on mobile clients, and mobile clients are very important for any IM service today.

— Reply to this email directly or view it on GitHub https://github.com/irungentoo/toxcore/issues/1251#issuecomment-167601717 .