Multiple routers > duplicate ifIndex'es > RRD graphs wrong

[d307473]

Hello, we have mutiple routers sending netflow streams to the AS-Stats collector port. Unfortunately we have duplicate ifIndex Interface IDs among the routers. We noticed that the data in as-stats_day.txt is correct, but the rrd graphs are missing huge parts of our traffic (which seems to only affect interfaces sharing the same ifIndex).

Is there anything we can do about this? We're using latest v1.6 release. Thanks

[rlanyi]

I experience something similar since my latest updates a few days ago but I'm not sure of the reason yet. Would you please try the version at https://github.com/manuelkasper/AS-Stats/commit/5b14fb721f6f2e868a95d6fd06328ed658931520 and see if it works with that? I'll try to investigate the root cause of this.

[rlanyi]

At the same time please make sure that the values of the third column of knownlinks is unique. For example prefix it with your router name, for example

192.168.0.1 74  router1_74  Link5 (router1) 8A4EB8  8192
192.168.0.2 74  router2_74  Link6 (router2) C90DF9  8192

[d307473]

Thanks @rylani. Checked out that revision but the issue persists. Yes, tags in our knownlinks are unique... RRD files are written, just large parts of the traffic are missing in the graphs, although I can confirm that the AS-per-interface counters in asstats_day.txt are correct.

I'm currently thinking of rewriting this whole as-stats thing in Python with https://github.com/bitkeks/python-netflow-v9-softflowd and python-rrdtool ...

[rlanyi]

It is strange because I had correct value before but now I see something similar. Please note that the collector wasn't changed in any way. Which GUI do you use?

Do you also see that besides missing charts, some bandwidth values are doubled? For example I see 16G traffic on some 10G links.

[d307473]

Short follow-up: I have started rewriting as-stats from scratch in Python and so far I'm already able to receive and parse netflow data. Currently I'm working on ip2as-lookups and automated pfx2as db updates via Caida Routeviews. The new script should also work with Netflow exporters which do not send the SRC_AS/DST_AS fields in flowset. Next step will be rrd generation with python-rrdtool.

Compared to this version the new python implementation will be much smaller, more lightweight and only a single instead of multiple script. Support only for Netflow v1/v5/v8/v9 curently, so no sflow and no IPFIX, but franky I do not see a big problem here nowadays. I expect to release a first test version within next week.

[falz]

Sounds great that it's being rewritten in python, but could you clarify why not supporting sflow/ipfifx?

I envision more people using IPFIX than say, netflow v5 or even v9. IPFIX is IETF standard rfc7012 vs Netflow, which is not.

There appear to be multiple ipfix python3 projects, such as https://pypi.org/project/ipfix/. [edit: this project seems stale, but perhaps others exist] Perhaps consider building some modularity in the code, with your netflow parser as one option, but built in a way that someone could add in / swap to a different module?

[d307473]

@falz thanks for the input. yes modularity would be nice, will consider that. for now i'm relying on https://github.com/bitkeks/python-netflow-v9-softflowd which seems to be easiest, most stable and actively maintained project i've found on github. maybe we could/should contribute ipfix support to the project. but as you mentionted, unfortunately many projects on pypi are unmaintained or stale.

[falz]

@schadom I see you opened an issue over there checking in IPFIX support (https://github.com/bitkeks/python-netflow-v9-softflowd/issues/20), cheers!

[d307473]

Hello, unfortunately the issue with our current as-stats installation was caused by a misconfiguration with our netflow collector/forwader samplicator. Issue closed for now, still working on a re-write of as-stats in Python but will take some additional days/weeks.