search

Juniper / open-nti

Open Network Telemetry Collector build with open source tools
Apache License 2.0
233 stars 93 forks source link

gRPC stream collector dashboard troubleshooting #217

Closed crmcguire11 closed 6 years ago

crmcguire11 commented 6 years ago

I'm having some trouble getting gRPC w/SSL to work on QFX10002. We're running 18.1R1. I haven't been able to find any documentation regarding SSL on the open-nti side. Is it supported in open-nti yet? I'd prefer to use unencrypted gRPC, but QFX10002 on 18.1R1 doesn't seem to allow this.

root@QFX10002_Top# set system services extension-service request-response grpc ? Possible completions:

jimmyjs1607 commented 6 years ago

Hi, clear text is a hidden configuration knob in Junos, try adding it manually.

crmcguire11 commented 6 years ago

Sweet. Thank you. Silly Juniper...hiding things like that. I now have sensors. Any ideas on why I wouldn't be seeing any data in Grafana? I'm running a brand new deployment on a brand new Ubuntu vm. I've only edited the telegraf.tmpl file, which seems to be working since I now have sensors on the QFX. I'm definitely getting traffic, too. Just no plotted data.

root@telemetry:~/open-nti/plugins/input-oc# tcpdump -ni ens160 port 50051 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 11:02:15.585259 IP 10.101.8.19.50051 > 10.101.16.20.41980: Flags [.], seq 2936297359:2936303151, ack 2055286757, win 33304, options [nop,nop,TS val 2204188371 ecr 39109801], length 5792 11:02:15.585377 IP 10.101.16.20.41980 > 10.101.8.19.50051: Flags [.], ack 5792, win 2255, options [nop,nop,TS val 39110332 ecr 2204188371], length 0

crmcguire11 commented 6 years ago

Definitely getting data...

grep write influxdb.log .... [httpd] 172.17.0.5 - juniper [04/May/2018:11:30:50 -0400] "POST /write?consistency=any&db=juniper HTTP/1.1" 204 0 "-" "-" 1fe93332-4fb0-11e8-a27d-000000000000 47632 [httpd] 172.17.0.5 - juniper [04/May/2018:11:30:55 -0400] "POST /write?consistency=any&db=juniper HTTP/1.1" 204 0 "-" "-" 22e427b5-4fb0-11e8-a27e-000000000000 62418

crmcguire11 commented 6 years ago

Well, I was using older resources to configure this. I have now successfully been able to collect data using the agent. However, I am still having issues collecting streaming data. I'm using gRPC rather than JTI, of course. I appreciate the help.

jimmyjs1607 commented 6 years ago

The issue is not clear, you don't see any data being graphed but you see data in the database? Last time I used gRPC there was no preconfigured dashboard/Query so I had to create mine.

3fr61n commented 6 years ago

Hi guys

The really important part is that there is data on the database.

Regarding the dashboards it depend on many other factors.

Not sure which dashboard are you using because there is one for native sensor (jti) (non-grpc), so those queries won’t work on your scenario

My suggestion is to her familiar to build queries/charts for the sensor you’re interested. I need to check with a colleague if there is any example available for grpc dashboard so you can habe a reference

Regards

Sent from my iPhone

On 5 May 2018, at 02:29, jimmyjs1607 notifications@github.com wrote:

The issue is not clear, you don't see any data being graphed but you see data in the database? Last time I used gRPC there was no preconfigured dashboard/Query so I had to create mine.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

crmcguire11 commented 6 years ago

I think my problem is not realizing I need to manually build a dashboard for grpc queries. I'll have to try that out tomorrow.

On Sat, May 5, 2018, 12:26 AM Efrain notifications@github.com wrote:

Hi guys

The really important part is that there is data on the database.

Regarding the dashboards it depend on many other factors.

Not sure which dashboard are you using because there is one for native sensor (jti) (non-grpc), so those queries won’t work on your scenario

My suggestion is to her familiar to build queries/charts for the sensor you’re interested. I need to check with a colleague if there is any example available for grpc dashboard so you can habe a reference

Regards

Sent from my iPhone

On 5 May 2018, at 02:29, jimmyjs1607 notifications@github.com wrote:

The issue is not clear, you don't see any data being graphed but you see data in the database? Last time I used gRPC there was no preconfigured dashboard/Query so I had to create mine.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Juniper/open-nti/issues/217#issuecomment-386780719, or mute the thread https://github.com/notifications/unsubscribe-auth/Ae5vfBdu4UBLzCsCVG0l1Cv1ohRQwHyNks5tvTgNgaJpZM4TxcEX .

jimmyjs1607 commented 6 years ago

I made a couple some time ago for RSVP reservation stat and LSP stats, maybe

those help.

jimmyjs1607 commented 6 years ago

Data Streaming Collector Dashboard-1518039809008.json.txt RSVP-Test-1518039822191.json.txt

Change the extension to .json :)

crmcguire11 commented 6 years ago

Hmmm...Your data streaming collector dashboard looks exactly the same as my auto-generated one. I compared them with Notepad++. They are legit exactly the same. I imported yours anyway. Still not working. Sorta stumped here. I completely deleted and rebuilt the docker container, modified telegraf.tmpl, and still have nothing.

sensors&ephemeral.txt

root@QFX10002_Top> show configuration system services extension-service request-response { grpc { clear-text { port 50051; } skip-authentication; } } notification { allow-clients { address 0.0.0.0/0; } }

crmcguire11 commented 6 years ago

I see....The autogenerated dashboard is looking for the jnpr.analyticsd table in InfluxDB, which doesn't seem to exist when streaming grpc. It's also looking for jnpr.jvision, which is the default table for native sensors.

Here's a snippet from telegraf.tmpl.

###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

# Read OpenConfig Telemetry from listed sensors

[[inputs.jti_openconfig_telemetry]]

  servers = ["10.101.8.19:50051"]

  ## Frequency to get data in millisecond
  sample_frequency = "1000ms"

  ## Sensors to subscribe for
  ## A identifier for each sensor can be provided in path by separating with space
  ## Else sensor path will be used as identifier
  #sensors = [
  # "/junos/system/linecard/interface/",
  # "/junos/system/cpu/memory/",
  # "/junos/system/linecard/packet/usage/",
  # "/junos/system/linecard/fabric/",
  # "/lacp",
  # "/lldp",
  # "/bgp"
  #]

  sensors = [
   "1000ms customReporting /interfaces /lldp",
   "2000ms collection /components",
   "/interfaces","/lacp","/junos/services/label-switched-path/",
   "/junos/system/linecard/interface/",
   "/junos/system/cpu/memory/",
   "/junos/system/linecard/packet/usage/",
  ]
jimmyjs1607 commented 6 years ago

Look for a dashboard named RSVP test, I have some queries there. When you're using gRPC the table takes the name of the sensor or you can put a name to it, for example: "Interface /junos/system/linecard/interface/",

Then you can log in to the influx db page and run a query like "select from Interface" or "select from ""/junos/services/label-switched-path/""

crmcguire11 commented 6 years ago

Copy. Thank you again.