Closed door7302 closed 6 months ago
Hi David
Thanks for the logs, could you please enable to sent the fluentd output to stdout?
In your docker-compose file just set the flag OUTPUT_STDOUT to true, as follows
root@node1:~/open-nti# cat docker-compose.yml
input-jti:
image: $INPUT_JTI_IMAGE_NAME:$IMAGE_TAG
container_name: $INPUT_JTI_CONTAINER_NAME
environment:
- "INFLUXDB_ADDR=opennti"
- "OUTPUT_INFLUXDB=true"
- "OUTPUT_STDOUT=true" <-------
ports:
- "$LOCAL_PORT_JTI:50000/udp"
- "$LOCAL_PORT_ANALYTICSD:50020/udp"
volumes:
- /etc/localtime:/etc/localtime
links:
- opennti
So after bringing up the container, please check the logs for that container, and you'll see in json format the measurements sent to influxdb.
I would like to check the measurement format/content that's causing the error appears.
Regards
hello
here the log. trace-opennti.txt
So, from your logs it seems there is no error at parsing (I don't see not supported sensors), so the next thing to check is influxdb.
1.- Are you using other collectors like netconf/snmp, etc? in order to check if they are properly writing into the database.
2.- Check influxdb logs from inside the opennti container.... execute
3.- Check if influxdb basic connectivity
juniper@rspjmco1-02:~/scripts/open-nti$ curl -sl -I http://localhost:8086/ping
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: bd7309f5-d434-11e7-8427-000000000000
X-Influxdb-Version: 1.2.0
Date: Tue, 28 Nov 2017 12:07:43 GMT
juniper@rspjmco1-02:~/scripts/open-nti$
Regards Efrain
Thank you for your quick feedback. Here the output of the requested URL :
/opt/open-nti$ curl -sl -I http://localhost:8086/ping HTTP/1.1 204 No Content Content-Type: application/json Request-Id: cc02f766-d37d-11e7-8068-000000000000 X-Influxdb-Version: 1.2.0 Date: Mon, 27 Nov 2017 14:18:10 GMT
And the influxdb log attached : influxdb.log
Hi
Each time the following log appear it means there was a data insertion ('write'), in this case in the juniper db.
[httpd] 172.17.0.5 - juniper [27/Nov/2017:15:15:55 +0100] "POST /write?db=juniper&p=%5BREDACTED%5D&precision=s&u=juniper HTTP/1.1" 204 0 "-" "Ruby" 7bbf2946-d37d-11e7-8021-000000000000 24849
And this log is a juniper db creation....
[httpd] 127.0.0.1 - juniper [27/Nov/2017:15:15:12 +0100] "POST /query?chunked=true&db=&epoch=ns&q=create+database+juniper HTTP/1.1" 200 67 "-" "InfluxDBShell/1.2.0" 61c48f75-d37d-11e7-8006-000000000000 3940
or telegraf db
[httpd] 127.0.0.1 - - [27/Nov/2017:15:15:09 +0100] "POST /query?db=&q=CREATE+DATABASE+%22telegraf%22 HTTP/1.1" 200 62 "-" "InfluxDBClient" 5feecc08-d37d-11e7-8001-000000000000 2171
Based on this logs it seems that the db should have some values....
Could you please access through influx cli and do basic queries like....
Regards Efrain
hereafter the outputs :
jnpr.jvision
docker docker_container_blkio docker_container_cpu docker_container_mem docker_container_net
root@574155367ce1:/# influx -database telegraf -execute 'show measurements' root@574155367ce1:/#
Thanks David
Could you please execute also
root@1cea636c464a:/# influx -database juniper -execute 'select * from "jnpr.jvision" limit 10'
name: jnpr.jvision
time device filter_name filter_timestamp type value
---- ------ ----------- ---------------- ---- -----
1511883109000000000 R1:172.16.0.1 __default_arp_policer__ 1511536072 memory_usage.HEAP 1576
1511883109000000000 R1:172.16.0.1 test 1511800028 memory_usage.HEAP 1688
1511883109000000000 R1:172.16.0.1 __default_bpdu_filter__ 1511800027 memory_usage.HEAP 2468
1511883119000000000 R1:172.16.0.1 test 1511800028 memory_usage.HEAP 1688
1511883119000000000 R1:172.16.0.1 __default_bpdu_filter__ 1511800027 memory_usage.HEAP 2468
1511883119000000000 R1:172.16.0.1 __default_arp_policer__ 1511536072 memory_usage.HEAP 1576
here the output
root@574155367ce1:/# influx -database juniper -execute 'select * from "jnpr.jvision" limit 10' name: jnpr.jvision time device egress_queue filter_counter_name filter_name filter_timestamp interface interface_parent type value
1511792113000000000 ncbor101:193.251.127.153 0 et-10/1/0 ae40 egress_queue_info.rl_drop_bytes 0 1511792113000000000 ncbor101:193.251.127.153 0 xe-10/0/0 ae111 egress_queue_info.peak_buffer_occupancy 65986 1511792113000000000 ncbor101:193.251.127.153 3 xe-8/1/5 ae15 egress_queue_info.bytes 2098794628 1511792113000000000 ncbor101:193.251.127.153 3 xe-8/1/5 ae15 egress_queue_info.avg_buffer_occupancy 0 1511792113000000000 ncbor101:193.251.127.153 3 xe-8/1/5 ae15 egress_queue_info.allocated_buffer_size 6389760 1511792113000000000 ncbor101:193.251.127.153 3 xe-8/1/4 ae14 egress_queue_info.tail_drop_packets 0 1511792113000000000 ncbor101:193.251.127.153 3 xe-8/1/4 ae14 egress_queue_info.rl_drop_packets 0 1511792113000000000 ncbor101:193.251.127.153 3 xe-8/1/4 ae14 egress_queue_info.rl_drop_bytes 0 1511792113000000000 ncbor101:193.251.127.153 3 xe-8/1/4 ae14 egress_queue_info.red_drop_packets 0 1511792113000000000 ncbor101:193.251.127.153 3 xe-8/1/4 ae14 egress_queue_info.red_drop_bytes 0
Moreover on Grafana, I've got an error when ALL interfaces is selected. But I think it's normal due to the number of Interfaces I have ? Other point, I didn't see my "ae" interfaces in the list, do you know why ?
So the good part is that your BD is NOT empty :), not sure if there was a problem there.
Now let's focus on grafana.
1.- How many interfaces do you have? (physical and aggregated)... 2.- In your previous output you can see that there is a field called ' interface_parent ' so all physical interfaces that belong to a ae, should have that field with a value. If you want to include aeX interfaces on your templating I suggest to edit the query or add a new templating variable with only aeX interfaces.
Hello
I have around 1000 interfaces (physical + logical) per router.
I have 4 routers with telemetry enabled
hello. Any news ?
cc. @3fr61n
It's weird, I have templating variable in production with +700 entries and I did not have that kind of errors..
Could you please check if your code contains this fix?
https://github.com/Juniper/open-nti/commit/bff2ea100c2a400665db717d9bb88ab669884897
Regards Efrain
Does it ring the bell ?
First I have already have the right value there :
max-row-limit = 0
Regarding the second point : Does it mean I have to modify the datasource.js directly ? If yes, where is located the file ?
Thks
TBH I don't know, but it seems there is a hardcoded limit on grafana somewhere.
Not sure which datasource.js are you talking about.
Another workaround is trying to reduce the templating query in order get less than 1000 entries.
For example..... one approach could be to use multiple templating queries that are related in a hierarchy, like
So at then end the Fourth template (that I asume is the one with 1000+ entries), will be narrowed down.
You have a discussion about it here https://github.com/grafana/grafana/issues/2214
Awesome... It works now with only one interface. So my last question will be : how to create a template for only 'ae' interfaces ? - currently I saw only physical and logical
Hello
Host receives well the streams :
xxxxxxxxx:~$ sudo tcpdump -n -i ens160 udp port 50000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 15:53:35.376488 IP 193.251.127.49.1000 > 193.252.147.55.50000: UDP, length 347 15:53:35.376993 IP 193.251.127.49.1000 > 193.252.147.55.50000: UDP, length 1003 15:53:35.389436 IP 193.251.127.49.1000 > 193.252.147.55.50000: UDP, length 1021 15:53:37.108930 IP 193.251.127.49.1000 > 193.252.147.55.50000: UDP, length 449 15:53:37.109874 IP 193.251.127.49.1000 > 193.252.147.55.50000: UDP, length 1502 15:53:37.122460 IP 193.251.127.49.1000 > 193.252.147.55.50000: UDP, length 1047
Streams are well forwarded to container open-nti :
xxxxxxxxx:~$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6cdb593ebffa juniper/open-nti-input-jti:latest "/bin/sh -c /home/..." 6 hours ago Up 6 hours 5140/tcp, 24224/tcp, 0.0.0.0:50000->50000/udp, 24284/tcp, 0.0.0.0:50020->50020/udp opennti_input_jti 5eb3f0533a60 opennti_input-internal "/source/start-inp..." 6 hours ago Up 6 hours opennti_input_internal 812964c10395 juniper/open-nti-input-syslog:latest "/bin/sh -c /home/..." 6 hours ago Up 6 hours 5140/tcp, 24220/tcp, 24224/tcp, 0.0.0.0:6000->6000/udp opennti_input_syslog 17ad5ac288ae opennti_input-snmp "/source/start-inp..." 6 hours ago Up 6 hours 0.0.0.0:162->162/udp opennti_input_snmp 01a616d37842 juniper/open-nti:latest "/sbin/my_init" 6 hours ago Up 6 hours 0.0.0.0:80->80/tcp, 0.0.0.0:3000->3000/tcp, 0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp, 0.0.0.0:8125->8125/udp opennti_con
xxxxxxxxx:~$ sudo docker exec -i -t 6cdb593ebffa /bin/bash
bash-4.3$ sudo tcpdump -n -i eth0 udp port 50000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:55:31.840339 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 264 15:55:31.840705 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 616 15:55:31.843955 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 427 15:55:31.849741 IP 172.17.0.1.57931 > 172.17.0.6.50000: UDP, length 656 15:55:31.851842 IP 172.17.0.1.57931 > 172.17.0.6.50000: UDP, length 2538 15:55:31.853600 IP 172.17.0.1.57931 > 172.17.0.6.50000: UDP, length 352 15:55:31.891920 IP 172.17.0.1.52406 > 172.17.0.6.50000: UDP, length 810 15:55:31.895362 IP 172.17.0.1.52406 > 172.17.0.6.50000: UDP, length 3398 15:55:31.910804 IP 172.17.0.1.52406 > 172.17.0.6.50000: UDP, length 2982 15:55:31.969255 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 2970 15:55:31.972879 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 3350 15:55:31.975477 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 2905 15:55:31.978896 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 407 15:55:32.008557 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 1073 15:55:32.010519 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 2506 15:55:32.014715 IP 172.17.0.1.38599 > 172.17.0.6.50000: UDP, length 476 15:55:32.674373 IP 172.17.0.1.44608 > 172.17.0.6.50000: UDP, length 547 15:55:32.675256 IP 172.17.0.1.44608 > 172.17.0.6.50000: UDP, length 1880 15:55:32.689090 IP 172.17.0.1.44608 > 172.17.0.6.50000: UDP, length 3291 15:55:32.979041 IP 172.17.0.1.44608 > 172.17.0.6.50000: UDP, length 671 15:55:32.980645 IP 172.17.0.1.44608 > 172.17.0.6.50000: UDP, bad length 1476 > 1472 15:55:32.996452 IP 172.17.0.1.44608 > 172.17.0.6.50000: UDP, length 2688 15:55:33.321255 IP 172.17.0.1.57931 > 172.17.0.6.50000: UDP, length 366 15:55:33.321827 IP 172.17.0.1.57931 > 172.17.0.6.50000: UDP, length 1079 15:55:33.323468 IP 172.17.0.1.57931 > 172.17.0.6.50000: UDP, length 379
But unfortunafly DB is empty.
Logs of DB seems fine :
2017-11-26 09:38:58 +0100 [info]: reading config file path="/tmp/fluent.conf" 2017-11-26 09:38:58 +0100 [info]: starting fluentd-0.12.29 2017-11-26 09:38:59 +0100 [info]: gem 'fluent-plugin-juniper-telemetry' version '0.3.0' 2017-11-26 09:38:59 +0100 [info]: gem 'fluentd' version '0.12.29' 2017-11-26 09:38:59 +0100 [info]: adding match pattern="jnpr." type="copy" 2017-11-26 09:38:59 +0100 [info]: adding match pattern="debug." type="stdout" 2017-11-26 09:38:59 +0100 [info]: adding match pattern="fluent.**" type="stdout" 2017-11-26 09:38:59 +0100 [info]: adding source type="forward" 2017-11-26 09:38:59 +0100 [info]: adding source type="udp" 2017-11-26 09:38:59 +0100 [info]: adding source type="udp" 2017-11-26 09:38:59 +0100 [info]: adding source type="monitor_agent" 2017-11-26 09:38:59 +0100 [info]: adding source type="debug_agent" 2017-11-26 09:38:59 +0100 [info]: using configuration file:
<match jnpr.**> type copyAny help is welcome. Thks David