BIGBLUEBUTTON_ENDPOINT and SAFE_HOST wrongly set

abockhold commented 3 years ago

Hello, I've got an installation of BigBlueButton 2.3.0-alpha5 with Greenlight 2.7.0 which I've set up on a public server using the bbb-install.sh script. The installation were running fine. Thanks for this very convenient script!

Today i've used bbb-install.sh again to update this installation to BBB 2.3.0-alpha8 and Greenlight 2.8.1.

The script itself ran smoothly but afterwards I got an error 500 when trying to connect to Greenlight. I could trace this down to the setting of BIGBLUEBUTTON_ENDPOINT and SAFE_HOST in .env: both were set to the IP address of the server and not the domain name which I supplied using "-s my.domain.com". After changing this myself everything is up and running again.

Is this related to my environment or could there be a bug in bbb-install.sh?

Best regards Andreas

abockhold commented 3 years ago

An addition: a freshly installed environment does not show this problem, BIGBLUEBUTTON_ENDPOINT and SAFE_HOST were set correctly corresponding to the specified "-s my.domain.com".

ffdixon commented 3 years ago

Are you using a saved copy of bbb-install.sh or running the latest version?

abockhold commented 3 years ago

No, I downloaded it: wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | bash -s -- -v bionic-230-dev -s my.domain.com -e info@my.domain.com -c my.turnserver.com:myturnsecret -w -g

abockhold commented 3 years ago

Another addition: there seems to be a "bigger" problem with BigBlueButton and Greenlight installed via script.

As I said before I not only updated one instance of BBB/Greenlight with the script but I freshly installed another one. After running it and logging in for the first time it stopped doing so after running bbb-install.sh on top of it again. Now after logging in I get the error message: "Server Error Invalid BigBlueButton Endpoint and Secret"

So to validate my problem I installed yet another new instance of BigBlueButton with Greenlight via the script. This one has the "Invalid BigBlueButton Endpoint and Secret" problem from the start. I can login with my created user but Greenlight does not connect to BigBlueButton. I've checked the output of bbb-conf --secret with the content of ~/greenlight/.env - it matches. bbb-conf --check and bbb-conf --status show "everything ok". Greenlight seems to be ok, too:

> root@meeting2:~/greenlight# docker run --rm --env-file .env bigbluebutton/greenlight:v2 bundle exec rake conf:check
> 
> Checking environment: Passed
> Checking Connection: Passed
> Checking Secret: Passed

In ~/greenlight/log/production.log I see: 2021-03-22 16:31:29 +0000 - ERROR: [46f68bf6-3fee-4b65-a5d1-f1266e4d8512] BigBlueButtonException: Connection error. Your URL is probably incorrect: "https://meeting2.mydomain.de/bigbluebutton/api". Error: Failed to open TCP connection to meeting2mydomain.de:443 (Connection refused - connect(2) for "meeting2.mydomain.de" port 443)

My customization via /etc/bigbluebutton/bbb-conf/apply-config.sh:

#!/bin/bash

# Pull in the helper functions for configuring BigBlueButton
source /etc/bigbluebutton/bbb-conf/apply-lib.sh

# Enable firewall rules defined in helper functions
enableUFWRules

# Set UI messages for users
crudini --set /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties '' defaultWelcomeMessage 'Willkommen in <b>%%CONFNAME%%</b>!<br><br>Als kurze Einf\u00fchrung zu BigBlueButton schaue dir diese <a href="event:http://www.bigbluebutton.org/html5"><u>Tutorials</u></a> an.<br>Um an der Audiokonferenz teilzunehmen, klicke auf den Telefonh\u00f6rer. Um Hintergrundger\u00e4usche zu vermeiden benutze bitte ein Headset.'
crudini --set /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties '' defaultWelcomeMessageFooter 'Hier l\u00e4uft <a href="http://docs.bigbluebutton.org/" target="_blank"><u>BigBlueButton</u></a>.'

# Disable recordings globally
crudini --set /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties '' disableRecordingDefault true

# Disable recordings in breakout rooms
crudini --set /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties '' breakoutRoomsRecord false

# Set Greenlight configuration
cd ~/greenlight

# Restart Greenlight
docker-compose down && docker-compose up -d

Besides the script seems to configure the redirection to Greenlight somehow and somewhere automatically? I used to add the redirection to /etc/nginx/sites-available/bigbluebutton as described here https://docs.bigbluebutton.org/greenlight/gl-customize.html#4-configure-nginx-to-route-to-greenlight But when I do this now NGINX won't start. And if I add nothing anywhere I nevertheless get redirected from / to /b?!

lonesomewalker commented 3 years ago

Same as: https://github.com/bigbluebutton/bigbluebutton/issues/11673 Can verify that, installation is broken.

abockhold commented 3 years ago

I just did some more tests:

I had the idea: maybe it's connected to IPv6? No, it's probably not. I've just setup new servers without it. Does not run nevertheless with 2.3.

Besides I tried the script side by side for 2.2 with Greenlight on Ubuntu 16.04 and for 2.3-beta with Greenlight on Ubuntu 18.04 - 2.2 is running, 2.3 is not, Greenlight reproducibly not able to connect to BBB because of wrong endpoint or secret in the 2.3 setup. But comparing output of bbb-conf --secret and .env does not show any discrepancies.

HTTPS is set with my installation automatically - output of bbb-conf --check shows bigbluebutton.web.serverURL: https://meeting1.mydomain.de

abockhold commented 3 years ago

Yet another observation:

I've got two servers, both installed just now with a freshly downloaded copy of bbb-install.sh. meeting1 - Ubuntu 18.04 - BBB 2.3-beta-1/Greenlight 2.8.2.1 meeting2 - Ubuntu 16.04 - BBB 2.2.35/Greenlight 2.8.2.1

Now I configure the Greenlight instances to access the BBB instance at the other server respectively: Greenlight at meeting1 -> BBB at meeting1: does not work Greenlight at meeting1 -> BBB at meeting2: does not work Greenlight at meeting2 -> BBB at meeting1: does work (!) Greenlight at meeting2 -> BBB at meeting2: does work

abockhold commented 3 years ago

Hm, a last thought I had just now: completely remove Greenlight and reinstall it manually. After following the steps to uninstall and reinstall here https://docs.bigbluebutton.org/greenlight/gl-install.html the command docker run --rm --env-file .env bigbluebutton/greenlight:v2 bundle exec rake conf:check shows:

Checking environment: Passed
Checking Connection: Failed
Could not get a valid response from BigBlueButton server - <Map><timestamp>1616779156439</timestamp><status>404</status><error>Not Found</error><message>No message available</message><path>/bigbluebutton/1/bigbluebutton/api/</path></Map>

I'm out of ideas what to do or to check? Some advice? I could supply a remote login to whomever might have an idea...

lonesomewalker commented 3 years ago

To fix your issue, it is pretty simple: set the correct endpoint. The one from BBB has to match the one in the .env of Greenlight.

But: the issue WHY this happens is still open. Maybe when it gets a little more quiet this weekend i have some time to debug the installation proccess.

abockhold commented 3 years ago

No, unfortunately this is not my problem. To verify and make absolutely sure I copy and pasted the lines

BIGBLUEBUTTON_ENDPOINT=https://meeting1.mydomain.de/bigbluebutton/
BIGBLUEBUTTON_SECRET=HZk7QfDpnLyFOIxKfaU18WuhT1btonEBDzCM5D5lAc

from .env on server meeting1 (there it does not work, in Greenlight I get "wrong endpoint or secret") to .env on server meeting2 (there it does work, I can use Greenlight and BBB). Therefore I'm 100% sure that (unfortunately) it's not just the endpoint configuration which is not working...

lonesomewalker commented 3 years ago

Believe me, typically it is... You have to set the shared secret manually by:

bbb-conf --setsecret heregoesyourdesiredsecret

After this, open your .env in the Greenlight folder and check from top to bottom that there is the right BBB server and the corresponding shared secret

Then:

cd ~/greenlight/ && docker-compose down && docker-compose up -d

But, i have checked YOUR servers (since you didn't redact the URL properly...) and they are not responding publicly -> the check will fail. Either get them up, publicly available and fix the corresponding shared secret and proper server url, otherwise this will fail. Maybe your ufw settings are faulty? (ufw status)

abockhold commented 3 years ago

Thanks for the heads up! Edit: I did not get that bbb-conf --setsecret heregoesyourdesiredsecret is explicitly needed. I will have a look into it and report the result back here! Yes, as I redeploy the servers frequently for testing purposes they are currently offline again.

abockhold commented 3 years ago

Well, now I've set the secret manually with bbb-conf --setsecret mysecrethere as you advised and restarted BBB with bbb-conf --restart but it does not work, same error as before.

And this makes sense in some way: as I stated before I tried to connect to the BBB on meeting1 from the Greenlight on meeting2 - this worked ok, meaning that the secret has to be set correctly.

The real problem is probably this, although I currently don't know how to fix it. From /var/log/nginx/error.log:

2021/03/27 08:47:00 [error] 30916#30916: *100 open() "/var/www/bigbluebutton-default/client/conf/config.xml" failed (2: No such file or directory), client: 127.0.0.1, server: meeting1.mydomain.de, request: "GET /client/conf/config.xml HTTP/1.1", host: "meeting1.mydomain.de"
2021/03/27 08:47:02 [error] 30916#30916: *108 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/adapter.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4102/html5client/compatibility/adapter.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *108 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/adapter.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4103/html5client/compatibility/adapter.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *108 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/adapter.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4104/html5client/compatibility/adapter.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *108 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/adapter.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4105/html5client/compatibility/adapter.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *108 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/adapter.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4106/html5client/compatibility/adapter.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *108 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/adapter.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4107/html5client/compatibility/adapter.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *107 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/sip.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4107/html5client/compatibility/sip.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *107 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/sip.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4106/html5client/compatibility/sip.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *110 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/kurento-utils.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4105/html5client/compatibility/kurento-utils.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *110 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/kurento-utils.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4104/html5client/compatibility/kurento-utils.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *110 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/kurento-utils.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4103/html5client/compatibility/kurento-utils.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *110 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/kurento-utils.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4106/html5client/compatibility/kurento-utils.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *110 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/kurento-utils.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4107/html5client/compatibility/kurento-utils.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *110 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/compatibility/kurento-utils.js?v=1482 HTTP/1.1", upstream: "http://127.0.0.1:4102/html5client/compatibility/kurento-utils.js?v=1482", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/2691cd608bb5d40a4a635bad8cffb96751b7df95.js?meteor_js_resource=true HTTP/1.1", upstream: "http://127.0.0.1:4102/html5client/2691cd608bb5d40a4a635bad8cffb96751b7df95.js?meteor_js_resource=true", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/2691cd608bb5d40a4a635bad8cffb96751b7df95.js?meteor_js_resource=true HTTP/1.1", upstream: "http://127.0.0.1:4103/html5client/2691cd608bb5d40a4a635bad8cffb96751b7df95.js?meteor_js_resource=true", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/2691cd608bb5d40a4a635bad8cffb96751b7df95.js?meteor_js_resource=true HTTP/1.1", upstream: "http://127.0.0.1:4104/html5client/2691cd608bb5d40a4a635bad8cffb96751b7df95.js?meteor_js_resource=true", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:02 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/2691cd608bb5d40a4a635bad8cffb96751b7df95.js?meteor_js_resource=true HTTP/1.1", upstream: "http://127.0.0.1:4105/html5client/2691cd608bb5d40a4a635bad8cffb96751b7df95.js?meteor_js_resource=true", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:09 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/resources/images/whiteboard-cursor/pan.png HTTP/1.1", upstream: "http://127.0.0.1:4105/html5client/resources/images/whiteboard-cursor/pan.png", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:09 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/resources/images/whiteboard-cursor/pan.png HTTP/1.1", upstream: "http://127.0.0.1:4107/html5client/resources/images/whiteboard-cursor/pan.png", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:09 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/resources/images/whiteboard-cursor/pan.png HTTP/1.1", upstream: "http://127.0.0.1:4102/html5client/resources/images/whiteboard-cursor/pan.png", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:09 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/resources/images/whiteboard-cursor/pan.png HTTP/1.1", upstream: "http://127.0.0.1:4104/html5client/resources/images/whiteboard-cursor/pan.png", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"
2021/03/27 08:47:09 [error] 30916#30916: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 85.212.xx.yy, server: meeting1.mydomain.de, request: "GET /html5client/resources/images/whiteboard-cursor/pan.png HTTP/1.1", upstream: "http://127.0.0.1:4106/html5client/resources/images/whiteboard-cursor/pan.png", host: "meeting1.mydomain.de", referrer: "https://meeting1.mydomain.de/html5client/join?sessionToken=ghzmrbb9gldc9xfd"

The reverse proxy tries to access some resources which are not there. This is probably a misconfiguration.

Besides I noticed that bbb-install.sh installs 2.3-alpha-8 and does not recognize 2.3-beta-1.

philou511 commented 3 years ago

Hello, I have exactly the same problem as @abockhold after a fresh install of BBB 2.3 beta3 : I get the following message when trying to access Greelight rooms or recordings : "Server Error - Invalid BigBlueButton Endpoint and Secret"

Everything seems OK ("bbb-conf --check" reports no problem, the three lines of "docker run --rm --env-file .env bigbluebutton/greenlight:v2 bundle exec rake conf:check" are OK ("Passed").

I see this in greenlight/logs/production.log : ERROR: [4403306c-2d49-40ef-9cb8-9bcbdaa230b6] [172.20.0.1] BigBlueButtonException: Connection error. Your URL is probably incorrect: "https://_bbb.server.fr_/bigbluebutton/api". Error: Failed to open TCP connection to bbb.server.fr:443 (Connection refused - connect(2) for "bbb.server.fr" port 443)

I also tried to change the secret (with bbb-conf --setsecret and manually in .env file of greenlight) with no succes...

If someone solved this I'm really interested :)

An update to BBB 2.3 beta5 with a new run of bbb-install today didn't solve the problem...

abockhold commented 3 years ago

Great to read somebody else has the same problem...

I tried some things more and my current conclusion is this: On a "freshly" installed Ubuntu 18.04 from the official media it works - on a, however customized, fresh install from a preconfigured cloud hosting provides it does not work (with the symptoms described above).

On what type of server did you try to install?

On a bare KVM hypervisor with an Ubuntu 18.04 VM it reproducible works for me, on a Hetzner Cloud Server it does not work again and again (nota bene: BBB 2.2 on 16.04 works for me on a Hetzner Cloud Server - all installed via bbb-install.sh).

philou511 commented 3 years ago

I installed Ubuntu 18.04 on a virtual machine on Hyper-V, I then tuned it a little as all our Ubuntu servers (just installed a few more apt packages), and then used bbb-install.sh I'm gonna try to install a fresh Ubuntu 18.04 (and no more), and see if BBB works. If yes, I will tune one package after another to try and see when BBB doesn't work any more...

philou511 commented 3 years ago

Good point : BBB installed OK on a fresh Ubuntu 18.04 + package upgrades. I'm now going to tune the server as we usually do to see if the problem comes back...

philou511 commented 3 years ago

No problem after tuning...

But when I installed BBB on a new fresh Ubuntu this morning, I didn't use the -a switch (for API demo) as the days before. Could the problem be there ?

I'm going to install a third VM, tune it as we usually do, and install BBB without the API demo to see if the problem is here or not.

abockhold commented 3 years ago

I've never used the API demos, so at least for me that's not the point. My install command: wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | bash -s -- -v bionic-230 -s meeting1.mydomain.de -e info@mydomain.de -c turn.mydomain.de:secret -w -g

The installed packages differ quite a bit between the ISO install and the provided cloud image. Therefore until now I couldn't find the trigger for the problem.

I'm quite interested to hear from your further findings!

philou511 commented 3 years ago

Problem still present after fresh install + tuning + bbb-install without demo API So that's our tuning which leads to the problem...

In the next days, I'll try and tune step after step then install BBB (doing snapshots of the VM after each new tuning step to come back there easily and tune one step more)... It will take a certain amount of time but I hope I'll finally find where the problem comes from...

abockhold commented 3 years ago

I'm thrilled to read from you!

I've found out that using the provided "cloud image" of Ubuntu 18.04 from Hetzner does not work whereas using their "installimage" tool and selecting "Ubuntu 18.04 minimal" works. Asking for the difference in these two installation variants their support answered me: "The difference is minimal. The cloud version is optimized for use on the cloud servers, for example there are packages for automatic volume/network configuration and qemu-guest-agent (enabling password changes from webfrontend) preinstalled."

philou511 commented 3 years ago

I share the result of my investigations. The problem is in the /etc/hosts file !

During our tuning on Ubuntu servers, we modifiy the /etc/hosts file : we add the hostname and hostname+domain at the end of the lien "127.0.0.1 localhost" It dosn't have any impact on the bbb-install script as I thought : the script works in all cases.

But once BBB is installed, either I let the /etc/hosts line like this "127 0.0.1 localhost" or that "127 0.0.1 localhost hostname" and BBB works like a charm, either I add hostname.domain at the end of this line and then BBB displays the BIGBLUEBUTTON_ENDPOINT and SAFE_HOST error.

I don't understand why though...

philou511 commented 3 years ago

If I add the hostname+domain on the lien 127.0.0.1 and also add a new line with the public IP and hostname+domain then BBB works perfectly

abockhold commented 3 years ago

Thanks for sharing your insights! I can confirm them now - I checked after I had your pointers as to where to look.

After deploying a Hetzner Cloud Server:

root@meeting1:~# cat /etc/hosts
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 meeting1.mydomain.de meeting1
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

It does not work, Greenlight shows the well known error regarding the entry point. Then I change the file and reboot:

root@meeting1:~# cat /etc/hosts
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
#127.0.1.1 meeting1.mydomain.de meeting1
127.0.0.1 localhost
78.47.21.103  meeting1

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

And now it works just fine.

From this I deduce: the locally installed Docker or the Docker containers have a problem with the public domain name pointing to the loopback IP. This is further confirmed as - as I stated somewhere above - connecting to the BBB component on the "non-working server" from a Greenlight hosted on another server works fine.

So my proposal to @ffdixon would be to add a test to bbb-install.sh to check that the public DNS name does not resolve to the loopback addresses?!

ReimarBauer commented 3 years ago

just stumbled over that problem too. it is also a bit surprising that it needs its external IP on this platform.

yanosz commented 3 years ago

+1 - ran into this issue, too.

Apperently, it is reasonable to have bbb-install checking, whether the hostname resolves too 127.0.0.0/8. When run in docker, greenlight won't be able to its host under this circumstances.

H-B-Schmidt commented 3 years ago

+1, same issue after updating to v2.3. I had to find this thread for the solution.

vlasov01 commented 3 years ago

I have a similar issue, but it is not related to Greenlight . I've documented it as a separate case.

gschwed commented 3 years ago

Many thanks to @abockhold and @philou511

I had the exactly same problem and debugged many hours to find the reason. To edit /etc/hosts was the solution. Thanks, you saved me many more hours.

AndreasUfert commented 3 years ago

Many thanks to @abockhold and @philou511

I had the exactly same problem and debugged many hours to find the reason. To edit /etc/hosts was the solution. Thanks, you saved me many more hours.

Wouldn't call this a solution. It's a workaround.

tosc-admin commented 3 years ago

Indeed, many thanks to @abockhold and @philou511!

To edit /etc/hosts solved the same issue with: (111: Connection refused)

bigbluebutton / bbb-install

BIGBLUEBUTTON_ENDPOINT and SAFE_HOST wrongly set #354