search

greenstatic / bigbluebutton-exporter

Prometheus exporter for BigBlueButton
https://bigbluebutton-exporter.greenstatic.dev
MIT License
158 stars 162 forks source link

bbb-exporter not connecting to api #88

Closed Aaomt closed 3 years ago

Aaomt commented 3 years ago

bbb-exporter | 2021-02-27 21:17:24,269 [ERROR]: Failed to perform API call bbb-exporter | 2021-02-27 21:17:24,269 [ERROR]: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /bigbluebutton/api/getMeetings?checksum=7e49b37f7d859834c541faa34e411f0a57cba74b (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5719327460>: Failed to establish a new connection: [Errno 111] Connection refused'))

bbb-2.3 Alpha-7 and scalelite and bbb-exporter 0.7.0-preview2 I am use I checked the secrets.env file many times https://example.com/metrics there is data but "bbb_api_up 0.0" - bbb_recordings_published 180.0 -

what am i doing wrong? please help

greenstatic commented 3 years ago

Are you connecting the exporter to your BBB instance and not to scalelite? I'm assuming you entered the correct domain and you left the example.com due to privacy reasons?

Aaomt commented 3 years ago

Are you connecting the exporter to your BBB instance and not to scalelite? I'm assuming you entered the correct domain and you left the example.com due to privacy reasons?

Example.com own domain instead exporter is on my bbb server

greenstatic commented 3 years ago

Make sure you entered the API key correctly and the URL with a trailing slash. https://bigbluebutton-exporter.greenstatic.dev/installation/bigbluebutton_exporter/#3-create-secrets-file

Your recordings metrics are visable because they are scraped most probably from the disk not via the API (see the RECORDINGS_METRICS_READ_FROM_DISK env variable from the docker-compose file).

Aaomt commented 3 years ago

Make sure you entered the API key correctly and the URL with a trailing slash. https://bigbluebutton-exporter.greenstatic.dev/installation/bigbluebutton_exporter/#3-create-secrets-file

Your recordings metrics are visable because they are scraped most probably from the disk not via the API (see the RECORDINGS_METRICS_READ_FROM_DISK env variable from the docker-compose file).

API_BASE_URL=https://ab.example.com/bigbluebutton/api/
API_SECRET=<lhjhkdldjjjdudjkksp09jdu78udks67sjd899dkd7>

I did as in the example but didn't happen

greenstatic commented 3 years ago

Try without the < > for the API_SECRET and issue a docker-compose restart in the dir where the compose file is located.

Aaomt commented 3 years ago

Try without the < > for the API_SECRET and issue a docker-compose restart in the dir where the compose file is located.

didn't happen, currently bbb_api_up 0.0

greenstatic commented 3 years ago

Enable debug mode and post the error message: https://bigbluebutton-exporter.greenstatic.dev/debugging/#debugging-faulty-metrics

Aaomt commented 3 years ago

Enable debug mode and post the error message: https://bigbluebutton-exporter.greenstatic.dev/debugging/#debugging-faulty-metrics

2021-03-01 13:52:24,285 [INFO]: Collecting metrics from BigBlueButton API 2021-03-01 13:52:24,286 [DEBUG]: Requesting via API meetings data 2021-03-01 13:52:24,286 [DEBUG]: TLS CA verification is enabled for API call 2021-03-01 13:52:24,289 [DEBUG]: Starting new HTTPS connection (1): ab.example.com:443 2021-03-01 13:52:24,291 [ERROR]: Failed to perform API call 2021-03-01 13:52:24,291 [ERROR]: HTTPSConnectionPool(host='ab.example.com', port=443): Max retries exceeded with url: /bigbluebutton/api/getMeetings?checksum=7e49b37f7d859834c541faa34e411f0a57cba74b (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f669bb4e220>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-03-01 13:52:24,291 [DEBUG]: Requesting via API recordings unpublished data 2021-03-01 13:52:24,292 [DEBUG]: TLS CA verification is enabled for API call 2021-03-01 13:52:24,295 [DEBUG]: Starting new HTTPS connection (1): ab.example.com:443 2021-03-01 13:52:24,296 [ERROR]: Failed to perform API call 2021-03-01 13:52:24,296 [ERROR]: HTTPSConnectionPool(host='ab.example.com', port=443): Max retries exceeded with url: /bigbluebutton/api/getRecordings?checksum=5402cd455d39b4e005ec5367cdf320f7058b6f13&state=unpublished (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f669bb36a30>: Failed to establish a new connection: [Errno 111] Connection refused')) 2021-03-01 13:52:24,297 [DEBUG]: Querying disk for recordings processing data 2021-03-01 13:52:24,297 [DEBUG]: Querying disk for recordings published data 2021-03-01 13:52:24,297 [DEBUG]: Querying disk for recordings deleted data 2021-03-01 13:52:24,298 [DEBUG]: Querying disk for recordings unprocessed data 2021-03-01 13:52:24,299 [DEBUG]: Calculating room participants histogram 2021-03-01 13:52:24,300 [DEBUG]: Calculating room listeners histogram 2021-03-01 13:52:24,300 [DEBUG]: Calculating room voice participants histogram 2021-03-01 13:52:24,300 [DEBUG]: Calculating room video participants histogram 2021-03-01 13:52:24,300 [DEBUG]: Calculating count of unique non-breakout meetings 2021-03-01 13:52:24,301 [DEBUG]: Calculating count of unique breakout rooms 2021-03-01 13:52:24,301 [INFO]: Finished collecting metrics from BigBlueButton API

greenstatic commented 3 years ago

This seems to me like a network issue. How are you running your exporter (all in one, docker-compose, systemd)? You are certain that the URL is accessible from your server correctly - no DNS issues, etc.?

Aaomt commented 3 years ago

This seems to me like a network issue. How are you running your exporter (all in one, docker-compose, systemd)? You are certain that the URL is accessible from your server correctly - no DNS issues, etc.?

running your exporter docker-compose no DNS issues https://ab.example.com/metrics I can access the url from my computer

greenstatic commented 3 years ago

I meant if you can access https://ab.example.com/bigbluebutton/api/ from the server running the exporter, i.e.

curl -v https://ab.example.com/bigbluebutton/api/

(Trailing slash is important)

Aaomt commented 3 years ago

I meant if you can access https://ab.example.com/bigbluebutton/api/ from the server running the exporter, i.e.

curl -v https://ab.example.com/bigbluebutton/api/

(Trailing slash is important)

root@BBB:~# curl -v https://ab.example.com/bigbluebutton/api/

greenstatic commented 3 years ago

I'm convinced this is an issue with the environment variables.

If you run bbb-conf --secret on your BBB host, make sure you copy and paste the Secret value (without any trailing spaces) in the secrets.env file. Example:

$ bbb-conf --secret
URL: https://ab.example.com/bigbluebutton/
Secret: foobar123

The secrets.env file should be formatted like so:

API_BASE_URL=https://ab.example.com/bigbluebutton/api/
API_SECRET=foobar123

Then run:

# cd to wherever you have the docker-compose file
$ docker-compose down -v
$ docker-compose up -d

If you are still unable to make it work, drop me an email and we can schedule a quick call.

Aaomt commented 3 years ago

I'm convinced this is an issue with the environment variables.

If you run bbb-conf --secret on your BBB host, make sure you copy and paste the Secret value (without any trailing spaces) in the secrets.env file. Example:

$ bbb-conf --secret
URL: https://ab.example.com/bigbluebutton/
Secret: foobar123

The secrets.env file should be formatted like so:

API_BASE_URL=https://ab.example.com/bigbluebutton/api/
API_SECRET=foobar123

Then run:

# cd to wherever you have the docker-compose file
$ docker-compose down -v
$ docker-compose up -d

If you are still unable to make it work, drop me an email and we can schedule a quick call.

didn't happen :(

greenstatic commented 3 years ago

During a live call, we debugged the issue. The issue was DNS resolution because the server's hostname matched the DNS of the BBB API. Because the container uses the host's DNS it resolved to 127.0.0.1, but since the container is in it's own networking stack, the exporter cannot connect to the API via localhost.

There are two solutions:

  1. Use the extra_hosts parameter in the docker-compose file: https://github.com/greenstatic/bigbluebutton-exporter/blob/6a44d9a89028d95019231cfdaa4480b2b616e862/extras/docker-compose.exporter.yaml#L12-L16
  2. Edit the docker-compose file to use the hosts networking stack 
    network_mode: host
environment:
BIND_IP: "127.0.0.1"

don't forget to remove the ports field, since when using the network_mode: host, they do not work.

Aaomt commented 3 years ago

During a live call, we debugged the issue. The issue was DNS resolution because the server's hostname matched the DNS of the BBB API. Because the container uses the host's DNS it resolved to 127.0.0.1, but since the container is in it's own networking stack, the exporter cannot connect to the API via localhost.

There are two solutions:

  1. Use the extra_hosts parameter in the docker-compose file: https://github.com/greenstatic/bigbluebutton-exporter/blob/6a44d9a89028d95019231cfdaa4480b2b616e862/extras/docker-compose.exporter.yaml#L12-L16
  2. Edit the docker-compose file to use the hosts networking stack
network_mode: host
environment:
  BIND_IP: "127.0.0.1"

don't forget to remove the ports field, since when using the network_mode: host, they do not work.

it happened either way thank you gregor