wrong bandwidth calculation

[boxintense]

https://bucket.vinc.me/0K5kpv5137.png

At first I thought the high speed was due to local net, but now I'm convinced that it's not, because the ethernet is only capable of 1 Gbps, and it says far more than 1 Gbps.

I'm on latest dd-wrt version (tp-link archer c7 v2) so I'm pretty sure this is not the officially supported version for bwmon.

[vortex-5]

Was this a recent development as a result of updating to the latest firmware version from DDWRT? I'm running 41686 myself.

Alternatively have you had correct bandwidth measurements with previous versions?

Due to the way bandwidth measurements work for the first 10 seconds or so the bandwidth is averaging out so it's unreliable for the first 3 seconds or so when you first open the page. If it's staying at that value sustained then the bandwidth value is wrong however other routers are not encountering this issue based on user comments and my own testing.

Unfortunately if there are differences between routers I may have to add a file where people can configure a "fudge factor" so you can adjust what the tool reports to account for differences in different router manufactures.

If you want to try this out for yourself you can change this line: https://github.com/vortex-5/ddwrt-bwmon/blob/master/bwmon/www/bwmon.js#L571 with some multiplier to bring it more in line with the speeds you're actually getting. In your case you'd be adding something like: let bps = Bps10 $scope.CONVERSION_FACTOR 0.1 (the last digit is your multiplier to try and reduce that speed number).

going forward I may add another file to allow users to put this in when their routers are off as I just checked my Netgear WNDR4300 and it seems to be reporting these speeds accurately.

But as a quick test you can add an adjustment factor to that line and see if you can get your numbers in line.

You probably need to apply the same factor to: https://github.com/vortex-5/ddwrt-bwmon/blob/master/bwmon/www/bwmon.js#L582 as well.

My theory of what is most likely going on is maybe different router kernels will choose to have the iptables tool report speeds in bytes or kilobytes by default so maybe you're off by a factor of 10

[vortex-5]

If you have ssh access or telnet access to your router you can try executing:

iptables -L BWMON -vnx

My top heading looks like:

pkts      bytes target     prot opt in     out     source               destination

Meaning that the router is reporting all values in bytes if the 2nd column is something else then the tool will be reporting wrong bandwidth measurements for sure.

[boxintense]

Was this a recent development as a result of updating to the latest firmware version from DDWRT?

Not likely, at that same day, I was updating both dd-wrt firmware as well as bwmon version to latest.

I updated dd-wrt first, and then, if I remember correctly, I checked bwmon and it seemed to work fine. Afterwards, I updated bwmon to latest version, I was using 1.10.1, a very old version which worked good.

iptables -L BWMON -vnx

Result :

https://bucket.vinc.me/7fUBfpHrFo.png

Seems like it's still using bytes.

I am going to revert back to bwmon version 1.10.1 to check whether the issues is caused by bwmon or ddwrt update. Will report back here

[boxintense]

Just reverted back to 1.10.1, and it's fixed.

[vortex-5]

Looking at the releases the last time the bandwidth measurement was changed in terms of how it's counted was DDWRT Bandwidth Monitor 1.14.0 so version 1.13.0 should work fine. let me know if all versions beyond 1.14.0 have this issue I may consider looking into the possibility of reverting how the bandwidth was calculated.

Although 1.14.0 actually resolved bandwidth calculations for some others so I'm not completely sure that's the best approach.

[boxintense]

1.13.0 - confirmed working good.

Trying 1.14.0 now

[vortex-5]

The reason for the changes in 1.14.0 is that some of the users were unhappy that devices with low usage (less than 1kb / min) were not counted at all.

[boxintense]

Yeah. 1.14.0 is where it's start having problem.

[boxintense]

The reason for the changes in 1.14.0 is that some of the users were unhappy that devices with low usage (less than 1kb / min) were not counted at all.

Not really sure what's changed between 1.13.0 and 1.14.0, but this is most probably due to routers bandwidth counting reporting different value, some of them in bpc and other Bpc, as you said?

If that's the case, perhaps adding a simple bwmon.conf file should give way to user to configure things according to their router's specification.

[vortex-5]

I can look into revering how some of the reporting works to go back to a version that looks more similar to 1.13 at the end of the day I don't think the higher precision was worth it overall.

[boxintense]

I don't think the higher precision was worth it overall.

Gonna have to agree with this, network throughput precision isn't exactly beneficial.. not at a small, SOHO-grade router at least.

[vortex-5]

I've patched in a revert from 1.13 into 2.0.0 let me know if this seems to work for you. You'll have to re-install and probably need to run ./clean.sh if you don't it will report that each device used way more data than expected.

bwmon.tar.gz

[boxintense]

the new 2.0.0 patched fixed the bandwidth problem, however there was one UI problem which was fixed at 1.13, it came back at the 2.0.0 patched version (as well as 2.0.0 unpatched):

When I click KB/s, the data table stays at Kbps

[vortex-5]

Ah ok how about this then: bwmon.tar.gz

[boxintense]

Flawless. Perfectly done!

[boxintense]

Thank you so much for the assist vortex!

[vortex-5]

I'll probably end up publishing this as 2.1.0 it's hard to say why it only affects TP-Link routers and not Netgear.

[vortex-5]

When you have time I have changed some calculations that didn't really look like they make sense can you check if this still works.

bwmon.tar.gz

And if you have the patience I may have a few minor experiments to see if I can figure out if the on router calculations are any different.

[boxintense]

Sure thing! I'll check them all out. Glad to be of assistance :)

[vortex-5]

oops found a bug.

bwmon.tar.gz

You're probably not using the non lighttpd version so you won't see it.

[boxintense]

When you have time I have changed some calculations that didn't really look like they make sense can you check if this still works.

bwmon.tar.gz

And if you have the patience I may have a few minor experiments to see if I can figure out if the on router calculations are any different.

This one works.

You're probably not using the non lighttpd version so you won't see it.

Yeah :)

[boxintense]

Got an 404 error for bwreader.cgi, but doesn't seem critical as everything functions normally here.

[boxintense]

oops found a bug.

bwmon.tar.gz

You're probably not using the non lighttpd version so you won't see it.

This one works as well. Couldn't find any bug.

A quick question, if I change $scope.SCRIPT_INTERVAL = 10; to 1, is there any repercussions in doing that?

[vortex-5]

If you're running lighttpd $scope.SCRIPT_INTERVAL isn't used I don't think. Lighttpd uses a polling rate of 2 second ($scope.SERVICE_INTERVAL). The actual script interval are controlled by lighttpd-running.sh

if you run lighttpd because you can communicate with the router and execute commands you get a polling rate of 2 seconds and the router itself will write to the internal usage log every 60 seconds.

If you are not running lightpd then the router won't be able to dynamically keep track if you're currently viewing the UI or not so you get a polling rate of 10 seconds and a database write rate of 10 seconds.

You want to minimize database writes when possible it's not great for the flash device and generally speaking the router has to do some work but if you delay too much the UI won't be able to show download / upload rates or will be showing past rates that are too far back to be useful. When you have lighttpd I drop you into a more efficient version that actually runs a bandwidth log every 10min and only if you're connected via UI will it both to poll faster (not possible with out cgi script execution).

Ok so I think I'm closer to figuring out why there's an issue I have a final version to test that may allow recording low bandwidth devices while still getting the rates right probably have something to test within 12 hours keep the last version around I'll rename the next one with a "BYTES" so it's easier to keep track of it if that one doesn't work then I guess I can't have my cake and eat it too.

[vortex-5]

Ok I guess I'm on a roll since I apparently wrote the changes during breakfast.

bwmon-BYTES.tar.gz

If that goes back to not working then I guess I can only log KB at the router level.

[boxintense]

If you are not running lightpd then the router won't be able to dynamically keep track if you're currently viewing the UI or not so you get a polling rate of 10 seconds and a database write rate of 10 seconds.

You want to minimize database writes when possible it's not great for the flash device

I've been using bwmon for roughly.. 2/3 years I think, using the exact same drives from the very start, with write cycle of 10 seconds. Considering the drive still lives (and this is a $1 USB drive), I think I can live with sacrificing away 1 or 2 flash drive ($2) every year to get a more realtime bandwidth monitoring capabilities.

Say, if I want to reduce the database write to every 2s and reload page every 2s, on a non-lighttpd router, what should I change in the script?

Ok I guess I'm on a roll since I apparently wrote the changes during breakfast.

bwmon-BYTES.tar.gz

If that goes back to not working then I guess I can only log KB at the router level.

This one seems to work fine as well!

[vortex-5]

You can change the interval and then also change: https://github.com/vortex-5/ddwrt-bwmon/blob/master/bwmon/bwmon-running.sh#L6 to match the interval that you changed.

There are people out there that have decided to use the built in flash on their routers despite my advice to not do so so with less than 100mb flash you's probably wear it out with writes eventually.

[boxintense]

Whoops its router flash and not usb flash? In that case.. I'll refrain

On Dec 20, 2019, 11:23 PM, at 11:23 PM, FanFan notifications@github.com wrote:

You can change the interval and then also change: https://github.com/vortex-5/ddwrt-bwmon/blob/master/bwmon/bwmon-running.sh#L6 to match the interval that you changed.

There are people out there that have decided to use the built in flash on their routers despite my advice to not do so so with less than 100mb flash you's probably wear it out with writes eventually.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/vortex-5/ddwrt-bwmon/issues/68#issuecomment-567982701

[vortex-5]

By default everything is setup to write to USB flash some people that don't want to plug a USB key still chose to set it up to flash to router flash. Anyways I have to account for every situation here so the timings set is a compromise. If you are using a USB flash you should be able to make those changes and have it set to basically every 2 seconds write there might be a slight hit on CPU load depending on how strong the router's CPU is.

[boxintense]

Hmm I can see that bwmon is writing to /tmp/www/usage.js every minute, which I believe is the router nvram.

When the backup is executing (every 15 minutes), it will then write to /jffs/bwmon/data/usage.js

Would it be possible to skip the /tmp/www/usage.js and go direct to /jffs/bwmon/data/usage.js instead?

[vortex-5]

/tmp/www/usage.js is on the router's ram not nvram /jffs/bwmon/data/usage.js can be your USB drive or your router's flash depending on how you configure it.

There's no issue if you use a USB stick like recommended but if you map jffs to your router's flash you will have a limited lifetime.

Like I said you can make the modifications that you describe but those will probably not be moved into an official version because people do choose to map jffs to nvram even though it's not a great idea in this application.

[boxintense]

Did set script interval to 1, but the web still refresh every 2s (seems like 2s is the lowest I can go)..

[oubcat96]

I have been following along with the thread and I am not sure if the issue I am seeing qualifies as the same problem. At some point a few months ago, i ran the clean.sh (do it monthly to track my usage for my monthly data cap) and all of the data that came back on the web page was garbled with a bunch of random characters. I decided to reinstall and I thought everything was good. I noticed that after a day or so there were certain devices that weren't dropping from the view able list. The one that really stood out is my PC. I use Plex and serve up content so it should always be my biggest consumer of data. I did reinstall a few times and the result was the same. I gave up until I saw this thread going. I tried 2.1.0 and the problem came up again. It looks like some devices are not calculating right and some devices (like my PC) are not even showing up. Is there some temp data somewhere that I may need to wipe out that the reinstall or clean.sh is missing?

[vortex-5]

@oubcat96 Data is located in /jffs/bwmon/data (if you install it to the default location) and while actively running data is also located in the following areas: /tmp/traffic_pre.tmp /tmp/traffic_post.tmp /tmp/usage.js /tmp/usage_stats.js

You can clear all of these after stop.sh is executed if you remove these while the script is running most likely you'll crash bwmon and need to restart it.

[oubcat96]

Thanks. Would this cause the erratic calculations? Here is what I happen to be seeing right now. I am assuming the reason the one showing the negative number eventually drops from the list completely because the total becomes negative?

[vortex-5]

it shouldn't no. Might be something unique to your router / config unfortunately.