search

charlie-haley / omada_exporter

Prometheus Exporter for TP-Link Omada Controller SDN.
MIT License
87 stars 14 forks source link

Metrics Errors #55

Closed charlie-haley closed 1 year ago

charlie-haley commented 1 year ago

Discussed in https://github.com/charlie-haley/omada_exporter/discussions/54

Originally posted by **jcouzy** November 22, 2022 Hi, when I run the Omaha_exporter in docker container is get errors when trying to access the metrics on http://hostname:9202. I run the container like this: docker run -d --name=omada-exporter -p 9202:9202 -e OMADA_HOST='https://omada.infra.local' -e OMADA_USER='prometheus' -e OMADA_PASS='*************' -e OMADA_INSECURE=true chhaley/omada_exporter When I try to access the metrics, I receive these errors: An error has occurred while serving metrics: 144 error(s) occurred: * collected metric "omada_port_power_watts" { label: label: label: label: label: label: label: label: label: gauge: } was collected before with the same name and label values And on and on. What am I doing wrong?
charlie-haley commented 1 year ago

I'll move the conversation over here @jcouzy, I updated my OC200 to 5.6.4 to see if I could replicate it, and I can't seem to replicate the issue you're having...

Could you send a full container log over, if possible?

jcouzy commented 1 year ago

These are the errors I keep receiving; so basically all the metrics are there, but seem not to be collected.

An error has occurred while serving metrics:

144 error(s) occurred:

dvonessen commented 1 year ago

Hi I do also have this issue. Same errors on stdout. I am using Omada 5.7 from mbentley. Version of the controller is the latest.

dvonessen commented 1 year ago

Short update, I checked all version from v0.7.1 to 0.6.0. The working version for me is 0.6.0.

jcouzy commented 1 year ago

I can confirm that version 0.6.0 works for me as well, error free.

charlie-haley commented 1 year ago

v0.7.0 is when some changes were made to use collectors, so it makes sense the error disappears when reverting.

Strange I'm not seeing this locally, I'll try and do a bit of digging to see why it's registering the collectors multiple times

charlie-haley commented 1 year ago

Are you only seeing this on omada_port_* metrics?

Could you give me a bit more details about your setup to help me diagnose this better as I can't recreate the error at all on my end.

I'm suspecting the Omada API is returning duplicates somewhere but I can't quite pinpoint where

dvonessen commented 1 year ago

I added the output of the /metrics path received by curl.

An error has occurred while serving metrics:

24 error(s) occurred:
* collected metric "omada_port_power_watts" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"1" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:8.3 > } was collected before with the same name and label values
* collected metric "omada_port_link_status" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"1" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "omada_port_link_speed_mbps" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"1" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:1000 > } was collected before with the same name and label values
* collected metric "omada_port_power_watts" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"2" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:8 > } was collected before with the same name and label values
* collected metric "omada_port_link_status" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"2" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "omada_port_link_speed_mbps" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"2" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:1000 > } was collected before with the same name and label values
* collected metric "omada_port_power_watts" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"3" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"30" > gauge:<value:8.3 > } was collected before with the same name and label values
* collected metric "omada_port_link_status" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"3" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"30" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "omada_port_link_speed_mbps" { label:<name:"client" value:"" > label:<name:"profile" value:"Wifi" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"3" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"30" > gauge:<value:1000 > } was collected before with the same name and label values
* collected metric "omada_port_power_watts" { label:<name:"client" value:"omv1" > label:<name:"profile" value:"All" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"4" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"30" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "omada_port_link_status" { label:<name:"client" value:"omv1" > label:<name:"profile" value:"All" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"4" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"30" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "omada_port_link_speed_mbps" { label:<name:"client" value:"omv1" > label:<name:"profile" value:"All" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"4" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"30" > gauge:<value:1000 > } was collected before with the same name and label values
* collected metric "omada_port_power_watts" { label:<name:"client" value:"" > label:<name:"profile" value:"Disable" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"5" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "omada_port_link_status" { label:<name:"client" value:"" > label:<name:"profile" value:"Disable" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"5" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "omada_port_link_speed_mbps" { label:<name:"client" value:"" > label:<name:"profile" value:"Disable" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"5" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:10 > } was collected before with the same name and label values
* collected metric "omada_port_power_watts" { label:<name:"client" value:"" > label:<name:"profile" value:"Disable" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"6" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "omada_port_link_status" { label:<name:"client" value:"" > label:<name:"profile" value:"Disable" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"6" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "omada_port_link_speed_mbps" { label:<name:"client" value:"" > label:<name:"profile" value:"Disable" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"6" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"" > gauge:<value:10 > } was collected before with the same name and label values
* collected metric "omada_port_power_watts" { label:<name:"client" value:"fritzbox" > label:<name:"profile" value:"IoT" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"7" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"40" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "omada_port_link_status" { label:<name:"client" value:"fritzbox" > label:<name:"profile" value:"IoT" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"7" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"40" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "omada_port_link_speed_mbps" { label:<name:"client" value:"fritzbox" > label:<name:"profile" value:"IoT" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"7" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"40" > gauge:<value:100 > } was collected before with the same name and label values
* collected metric "omada_port_power_watts" { label:<name:"client" value:"ecb5fa085793" > label:<name:"profile" value:"IoT" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"8" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"40" > gauge:<value:0 > } was collected before with the same name and label values
* collected metric "omada_port_link_status" { label:<name:"client" value:"ecb5fa085793" > label:<name:"profile" value:"IoT" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"8" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"40" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "omada_port_link_speed_mbps" { label:<name:"client" value:"ecb5fa085793" > label:<name:"profile" value:"IoT" > label:<name:"site" value:"Default" > label:<name:"site_id" value:"63223aaa02eb811e7fe191d5" > label:<name:"switch_id" value:"63223aaa02eb811e7fe19220" > label:<name:"switch_mac" value:"28-87-BA-D9-FF-C5" > label:<name:"switch_port" value:"8" > label:<name:"vendor" value:"" > label:<name:"vlan_id" value:"40" > gauge:<value:100 > } was collected before with the same name and label values

As you can see this is only on the omada_port_* metric.

My prometheus configuration:

- job_name: docker-containers
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:

Scraping is done in an 15s intervall. But the curl actually gives the error too.

I have in total 5 devices. 3x Wireless EAP670(EU) v1.0 1x TL-SG2008 v4.0 Version 4.0.6 1x TL-SG2008P v3.0 version 3.0.4

I am only using the default site, so one site in total

jcouzy commented 1 year ago

My prometheus config:

So the default prometheus config parameters

My devices: 1x OC200 1x TL-R605 1x TL-SG2016P 3x TL-2008P 1x TL-2008 2x TL-EAP620HD 1x TL-EAP610-Outdoor 1x EAP650-Wall

charlie-haley commented 1 year ago

Okay, I've just made some changes to logging so you'll be able to dump the raw API responses, that will allow me to mock your setup locally and see if I can recreate this issue.

If you could run version 0.8.0 of the exporter with the --log-level "debug" flag and then send over the output of the logs, that should help massively!

Just be aware - this will be the raw output straight from Omada, so you might want to double check there's no sensitive data in the log! (My output seemed fine, but you might want to check anyway)

dvonessen commented 1 year ago

Hi, I did it. Used the 0.8.0 tag and enabled debug log. I think there is nothing "secret" in the log. But if you find anything, let me know, so I can change it afterwards. :-) omada-exporter.json.zip

charlie-haley commented 1 year ago

For some reason, the Omada API was returning duplicate ports, so for your 8 port switch, it was returning 16 ports but all of them were duplicated... Thanks for the logs, they were a massive help pinpointing the issue!

I've added a simple but hacky fix as part of v0.8.1 which should hopefully fix your issue.

Let me know if you're still having trouble and I can re-open this issue

dvonessen commented 1 year ago

Thanks for fixing this problem. I see metrics and no errors.

Thanks for this tools.

jcouzy commented 1 year ago

The fix works for me as well, thanks for fixing the issue.