Overestimating data usage compared to Android native metering (and telephone provider)

geekycow commented 8 years ago

Hi

(Android 5.1 - Moto G - kernel 3.4.42-g89906d6 - 221.201.2.falcon_umts.Retail.en.GB - Call Meter 3G v3.11.11)

Android's build in Data Usage monitor (Settings -> Data Usage) reports monthly data usage that matches the providers data usage but Call Meter 3G v3.11.11 overestimates it for the same period.

Examples-

May 13th -Jun 12th Mobile provider: 274.33Mb Native Android data use monitor: 273Mb Call Meter 3G: 291.99Mb

Apr 13th - 12th May Mobile provider: 439.71Mb Native Android data use monitor: 440Mb Call Meter 3G: 448Mb

Given that there isn't a configurable billing mode for data usage and that I imagine the data information is coming from pretty much the same source (the kernel), shouldn't Call Meter's estimation match pretty close to Android's native meter? Surely if there was going to be a discrepancy it would be Call Meter under recording data?

Any ideas why this might be? I'd like to use a spreadsheet to try to pin down where this discrepancy might be coming from but can't due to #113

Warm Regards Julie x

mhilcher commented 8 years ago

Hi, I have a similar issue, but in my case there's an extreme difference compared to Android and my provider.

Data used regarding to:

Android: 595 MB
My provider: 585 MB
CM3G: 4464 MB

The CM3G counter seems to count WiFi data, too. Or the calculation fails.

Martin

mhilcher commented 8 years ago

Forgot: My device is a Samsung Galaxy S6 Edge running on Android 6.0.1. Call Meter 3G v3.11.11

eoxaal commented 8 years ago

Samsung GS7 Edge. CMG3 working perfectly until a week back or so. Now grossly overestimating data usage. Since midnight start of July reports 354mb data used, while real use is 23mb (hardly used the phone). I have not touched CMG3 settings for a long time. Can the fact that my operator recently activated Voice Over LTE have a bearing on this?

eoxaal commented 8 years ago

Another contributing factor to my comment above could be thar Samsung installed an OTA firmware update on June 14.

felixb commented 8 years ago

The current app just takes traffic logs from the android API which takes the information from the linux kernel. At the moment, there is no understanding of excluded traffic.

VoIP/VoLTE is very possibly a problem, because it should be excluded but it is data traffic in the end.
OTA updates might be a problem, but it depends.

I'll try to review the android API for data traffic to check if there is a (new) way to filter the traffic counters.

eoxaal commented 8 years ago

Ok. Right now Callmeter is telling me that I've used 10,2 Gb so far this month, while actual use is about 600 mb ;-)

Erik Oxaal mailto:erik@oxaal.com Tlf: +47 91514641 On 10 Jul 2016 3:37 p.m., "Felix Bechstein" notifications@github.com wrote:

The current app just takes traffic logs from the android API which takes the information from the linux kernel. At the moment, there is no understanding of excluded traffic.

VoIP/VoLTE is very possibly a problem, because it should be excluded but it is data traffic in the end.

OTA updates might be a problem, but it depends.

I'll try to review the android API for data traffic to check if there is a (new) way to filter the traffic counters.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/felixb/callmeter/issues/114#issuecomment-231589426, or mute the thread https://github.com/notifications/unsubscribe/AAjyl_CT1mfDxVSdYUuCwpRPo1-IHUs5ks5qUPWogaJpZM4I2hvm .

felixb commented 8 years ago

I believe, I can't do anything about it. The current version uses TrafficStats.getMobileRxBytes() et. al. to collect the metrics. I first thought using the new NetworkStatsManager can fix your issues. But reading further it says:

NOTE: Calling querySummaryForDevice(int, String, long, long) or accessing stats for apps other than the calling app requires the permission PACKAGE_USAGE_STATS, which is a system-level permission and will not be granted to third-party apps. However, declaring the permission implies intention to use the API and the user of the device can grant permission through the Settings application.

eoxaal commented 8 years ago

Strange. So here's a challenge for you ;-) I also have another data counter app installed, and it manages to count the data correctly. So I think you you should be able to do just as well. :-)

http://android.roysolberg.com/datacounter/

BR, Erik

Erik Oxaal mailto:erik@oxaal.com Tlf: +47 91514641 On 10 Jul 2016 18:27, "Felix Bechstein" notifications@github.com wrote:

I believe, I can't do anything about it. The current version uses TrafficStats.getMobileRxBytes() https://developer.android.com/reference/android/net/TrafficStats.html#getMobileRxBytes() et. al. to collect the metrics. I first thought using the new NetworkStatsManager https://developer.android.com/reference/android/app/usage/NetworkStatsManager.html can fix your issues. But reading further it says:

NOTE: Calling querySummaryForDevice(int, String, long, long) or accessing stats for apps other than the calling app requires the permission PACKAGE_USAGE_STATS, which is a system-level permission and will not be granted to third-party apps. However, declaring the permission implies intention to use the API and the user of the device can grant permission through the Settings application.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/felixb/callmeter/issues/114#issuecomment-231597543, or mute the thread https://github.com/notifications/unsubscribe/AAjyl7pW6hTU7e922_ha5vlNOCytwrqrks5qUR1RgaJpZM4I2hvm .

deMattin commented 7 years ago

I can confirm, that VoLTE Traffic is counted as data traffic with Callmeter 3G on my phone. I tested it for 2 weeks now with disabled VoLTE and Callmeter 3g ist counting data traffic right with that phone setting.

There has to be a possibility to track only data traffic without VoLTE by usig another function because:

internal phone counter counts with ignoring VoLTE traffic
"datacounter" as mentioned from @eoxaal counts with ignored VoLTE traffic
Callmeter 3G counts VoLTE traffic as separate entries in log with entires, that appear to be the cumulated traffic of VoLTE again and again some times a day (some xx or xxx MB on each count). If you delete these enties in log manually, counted data traffic is right again.

My Phone: Samsung S7 with Android 7.0

@felixb

I first thought using the new NetworkStatsManager can fix your issues. But reading further it says:

NOTE: Calling querySummaryForDevice(int, String, long, long) or accessing stats for apps other than the calling app requires the permission PACKAGE_USAGE_STATS, which is a system-level permission and will not be granted to third-party apps. However, declaring the permission implies intention to use the API and the user of the device can grant permission through the Settings application.

What does that mean? Granting permission is possible by user, if the app intends to use the api? Should be no problem, I will grant, if I will be asked to do so ... ;)

greets, Martin

PS: @eoxaal: Please delete the "mute thread link" in you previous post, if you don't want to give everybody the opportunity to deactivate your thread notfiations.

neobesseya commented 7 years ago

I have the same problem since I updated my Samsung Galaxy to Android 6.0.1.

I investigated a little further, and found that whenever I switch mobile data on, your app adds the total accumulated amount that is recorded by the system. So it appears that android system returns to your app an accumulated amount at switch-on. I hope there is a flag that allows you to identify this and not to add this to your internal accumulated amount. That would then fix the bug.

Unfortunately I am no expert at all on android...

cheers, Neo

deMattin commented 7 years ago

@neobesseya Do you have "Voice over LTE" active? If yes, try to deactivate it and have a look, if your traffic counting is correct then. As I have described above, I see your described accumulated counting only for the voice part of the "Voice over LTE" traffic (you don't need to have calls for this traffic). This traffic isn't part of the data plan and so shouldn't be counted at all. So there is a not accumulated counted/displayed part of traffic in callmeter, which is representing the real, complete and correct traffic (2G/3G/LTE) and additional an unwanted and accumulated displayed traffic, if VoLTE is active.

It's realy a pitty, because I like callmeter very much and defined a complicated setup for my data and call plan. For now I deactivated VoLTE to have a correct counting of local and roamed traffic/calls and hope @felixb will find a solution for this in future - perhaps by finding a "marker" for this traffic and ignoring this accumulated traffic in callmeter.

Manual deleting the additional traffic in callmeter data log (which corrects counting) is also possible but very hard, has do be done frequently and is not reliable at beginning of the counting period, because at the beginning of an accounting period this traffic is of smaller amount and so harder to identify.

neobesseya commented 7 years ago

Hi deMattin, no, I don't have voice over LTE. Only data connections use the LTE protocol, if available. The accumulated amount is added even when i switch on data with a low-speed GPRS connection.

And the problem started with Android v. 6, never a problem with v. 4 and 5. There must have been a change in the interface.

neobesseya commented 7 years ago

OK, I have now the free app "data usage monitor" by Lufesu Inc. installed, which counts my data reliably. So my fix is to use callmeter 3G for calls and sms only, and another app for data. Btw, this app uses the permission PACKAGE_USAGE_STATS, so this seems to be permitted for 3rd-party apps (without asking anything from the user when being installed)!

I understand that felix has not the time to investigate into the interface problems with some flavors of android (like mine v.6.0.1), or implement the NetworkStatsManager, and I am perfectly happy with my two-app solution. I suppose that helps also the other users suffering from this problem.

cheers, Neo.

deMattin commented 6 years ago

Issue is fixed with actual Samsung Update on my Galaxy S7edge! In the past, I tried it with every update Samsung made und after the last one, call meter is working fine again also with VoLTE activated! @eoxaal please have a try!

Edit: I don't understand this issue - it's frustrating. It has worked perfectly for more than 2 weeks after the Samsung firmware update and now the problem is the same as before.

lu-p-us commented 3 years ago

I investigated a little further, and found that whenever I switch mobile data on, your app adds the total accumulated amount that is recorded by the system. So it appears that android system returns to your app an accumulated amount at switch-on. I hope there is a flag that allows you to identify this and not to add this to your internal accumulated amount. That would then fix the bug.

Started to have the same issue recently. Can confirm that turning on mobile data results in "false" data entries.

Could have started with the upgrade from Android 9 to 10 (both LineageOS) or one of the monthly updates for 10. Not sure about that.

Edit 2021-03-20 Note for future visitors: Did a small one-week-experiment. Keeping mobile data ON all the time and only turning wifi ON/OFF results in less false data entries in Call Meter 3G, but does not completely prevent them, at least for me. Because of that, I resorted to

So my fix is to use callmeter 3G for calls and sms only, and another app for data.

Chefche commented 1 year ago

Hello: I think, this software is too valuable and the widgets too beautiful to let it die. Thus, I dug a bit deeper into this problem. VoLTE is definitely the root cause for the false data counting because

if I disable VoLTE, the data logs disappear that happened to occur when mobile data were disabled at all,
having a look into the souce code of TrafficStats reveals that counting the data happens in fact summarized over all interfaces (rmnet1 is for VoLTE, rmnet0 for normal data): public static long getMobileRxBytes() { long total = 0; for (String iface : getMobileIfaces()) { total += getRxBytes(iface); } return total; } Fortunately, from API level 31 there is a simple solution. Instead of getMobileRxBytes() use getRxBytes("rmnet0"), that's it. I intended to provide a user ready fix, but I failed in converting the whole project to API level 33 and especially the adjustments for Google AdMob :(

Best regards, Th.

felixb / callmeter

Overestimating data usage compared to Android native metering (and telephone provider) #114