search

Juniper / open-nti

Open Network Telemetry Collector build with open source tools
Apache License 2.0
232 stars 93 forks source link

Decouple open-nti.py from the database #81

Open dgarros opened 8 years ago

dgarros commented 8 years ago

Hi We've been talking about it for a while, I think it will be great to decouple the netconf collector (open-nti.py) from the Influxdb. The idea would be to send data to Fluentd or Telegraf (format to be defined) to leverage their plugin architecture. Both have the ability to execute external script periodically to accept plugin in any languages.

Anil has started to look into this and we realized that with the current implementation it's gonna be challenging because of how the delta computation is done today. Is this part still mandatory ? Do we know if Influxdb has improved on this point ? it would be great to get rid of this part if possible

Would love to get everyone's feedback on that. Damien

mmellin commented 8 years ago

As an fyi - We noticed influx gets pretty slow when sending lots of syslog to it via Unix logger so we built out logstash with Kafka instead of using fluentd.

Matt

On Aug 25, 2016 10:42 PM, "Damien" notifications@github.com wrote:

Hi We've been talking about it for a while, I think it will be great to decouple the netconf collector (open-nti.py) from the Influxdb. The idea would be to send data to Fluentd or Telegraf (format to be defined) to leverage their plugin architecture. Both have the ability to execute external script periodically to accept plugin in any languages.

Anil has started to look into this and we realized that with the current implementation it's gonna be challenging because of how the delta computation is done today. Is this part still mandatory ? Do we know if Influxdb has improved on this point ? it would be great to get rid of this part if possible

Would love to get everyone's feedback on that. Damien

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Juniper/open-nti/issues/81, or mute the thread https://github.com/notifications/unsubscribe-auth/ABW46dz1s6yHp0tNcQDvXW-Ivpm7kmoSks5qjny_gaJpZM4Jtwfm .

dgarros commented 8 years ago

@mmellin interesting, would love to see what you guys have built.

cookiesowns commented 7 years ago

Any luck on this? We've recently built a stack using InfluxDB, Grafana/Chronograf & Telegraf/Graphite for monitoring our servers / networking devices.

The last piece of our puzzle is integrating our JunOS devices into the network for monitoring without resorting to hacky solutions with SNMP OID's and SNMP input plugins on Telegraf..

Even for now it would be great if additional documentation on open-nti.py could be provided on how one could leverage this within an existing InfluxDB deployment

dgarros commented 7 years ago

Hi @cookiesowns

I pushed recently an alpha version of the NETCONF collector that can work in standalone mode. https://github.com/Juniper/py-netconf-collector

It's still early but you are welcome to give it a try. it's based on the current open-nti.py script

To summarize the changes I made:

the migration to an object oriented solution has a lot of benefits, it become much easier to maintain, test and add features.

I proposed a slight change of format of the parsers and we are still haven't agreed on it with the rest of the team so for now, the number of parsers is very limited. We should probably move the discussion to github to collect everyone's feedback.

Least but not least, in // I'm working on integrating this new collector within OpenNTI. the changes are part of the branch consul

Let me know if you need help to get started

Damien

cc/ @3fr61n @psagrera