search

jitsi / docker-jitsi-meet

Jitsi Meet on Docker
https://hub.docker.com/u/jitsi/
Apache License 2.0
3.06k stars 1.36k forks source link

jibri unable to record, ERR_CONNECTION_REFUSED, An error occurred while joining the call, potential DNS issue #1274

Closed dvnicolasdh closed 2 years ago

dvnicolasdh commented 2 years ago

SETUP

Running under Ubuntu 22.04 docker-ce: Docker version 20.10.14, build a224086 docker compose: Docker Compose version v2.2.3

Following instructions from: https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-docker/ with release 7210: https://github.com/jitsi/docker-jitsi-meet/releases/tag/stable-7210

and launching the follwing command after having properly adjusted the ".env" as instructed in the devops-guide-docker:

docker compose -f docker-compose.yml -f jibri.yml up -d

Jitsi works but recording fails

Opening a browser on our JITSI server: https://MYPUBLICFQDN example: https://my.example.com

Everything seems fine, we can create rooms with video audio connections multiple participants, and the "recording" option is shown, but attempts to launch a session "recording" lead to "Recording has Stopped" audio message and popup with "Echec du demarrage de l'enregistrement". RecordingHasStopped-JIBRI

EXTRACT of my JIBRI logs:

(MYPUBLICFQDN being the fully qualified domain name of our jitsi server, ex: my.example.com )

...
docker-jitsi-meet-stable-7210-jibri-1    | Jibri 2022-04-28 02:18:36.639 INFO: [57] AbstractPageObject.visit#32: Visiting url https://MYPUBLICFQDN
...
docker-jitsi-meet-stable-7210-jibri-1    | Jibri 2022-04-28 02:18:36.838 SEVERE: [57] [session_id=fqyoeugllhxdkhma] JibriSelenium$joinCall$1.run#325: An error occurred while joining the call
org.openqa.selenium.WebDriverException: unknown error: net::ERR_CONNECTION_REFUSED
docker-jitsi-meet-stable-7210-jibri-1    | org.openqa.selenium.WebDriverException: unknown error: net::ERR_CONNECTION_REFUSED
docker-jitsi-meet-stable-7210-jibri-1    |   (Session info: chrome=96.0.4664.45)
docker-jitsi-meet-stable-7210-jibri-1    |   (Driver info: chromedriver=96.0.4664.45 (76e4c1bb2ab4671b8beba3444e61c0f17584b2fc-refs/branch-heads/4664@{#947}),platform=Linux 5.15.0-27-generic x86_64) (WARNING: The server did not provide any stacktrace information)
docker-jitsi-meet-stable-7210-jibri-1    | Command duration or timeout: 0 milliseconds
...

found that 7210-1 and 7210-2 have been released since then... tried 7210-2... same problem https://github.com/jitsi/docker-jitsi-meet/releases/tag/stable-7210-2

docker-jitsi-meet-stable-7210-2-jibri-1  | Jibri 2022-04-28 04:14:58.897 INFO: [57] AbstractPageObject.visit#32: Visiting url https://MYPUBLICFQDN
docker-jitsi-meet-stable-7210-2-jibri-1  | Jibri 2022-04-28 04:15:00.583 SEVERE: [57] [session_id=ibvgviwgfwirwzlc] JibriSelenium$joinCall$1.run#325: An error occurred while joining the call
docker-jitsi-meet-stable-7210-2-jibri-1  | org.openqa.selenium.WebDriverException: unknown error: net::ERR_CONNECTION_REFUSED
docker-jitsi-meet-stable-7210-2-jibri-1  |   (Session info: chrome=96.0.4664.45)
docker-jitsi-meet-stable-7210-2-jibri-1  |   (Driver info: chromedriver=96.0.4664.45 (76e4c1bb2ab4671b8beba3444e61c0f17584b2fc-refs/branch-heads/4664@{#947}),platform=Linux 5.15.0-27-generic x86_64) (WARNING: The server did not provide any stacktrace information)
docker-jitsi-meet-stable-7210-2-jibri-1  | Command duration or timeout: 0 milliseconds

WORK-AROUND:

MANUALLY ADDING the "internal ip" of the "web" container in the "/etc/hosts" of the "jibri" container:

1) Find out the "internal IP" of the "WEB" container: (if running version 7210)

docker inspect --format '{{range .NetworkSettings.Networks}} {{.IPAddress}}{{end}}' docker-jitsi-meet-stable-7210-web-1

(if running version 7210-2)

docker inspect --format '{{range .NetworkSettings.Networks}} {{.IPAddress}}{{end}}' docker-jitsi-meet-stable-7210-2-web-1

or get list of all running containers names and ips:

docker ps -q | xargs -n 1 docker inspect --format '{{ .Name }} {{range .NetworkSettings.Networks}} {{.IPAddress}}{{end}}' | sed 's#^/##';

Take note of the obtained IP for the jitsi web docker container. ex: 172.29.0.2

2) Open a bash session inside the "JIBRI" container:

docker compose -f docker-compose.yml -f jibri.yml exec jibri bash

3) Append the "internal IP" of the WEB container, with the MYPUBLICFQDN ex:

echo 172.29.0.2 my.example.com >> /etc/hosts

leading to /etc/hosts something like:

127.0.0.1 localhost
172.29.0.2 my.example.com

Note: use your actual internal IP and your actual FQDN instead of 172.29.0.2 and my.example.com

4) Now, from the jitsi web page, attempts to launch record do work ! RecordingIsOn

we obtain the recordings under the ~/.jitsi-meet-cfg/recording/xxxxxx/yyyyyy.mp4 Where xxxx is the recording session_id and yyy is the concatenation of the roomname-and -timestamp.

It is strange that the "FQDN" is not found from within the jibri container without this manual intervention... Can this be caused by the usage of an incompatible "docker compose" version ?

saghul commented 2 years ago

Hum, that's odd! Any chance you can try Docker Compose v1? I wonder if there is some incompatibility there.

dvnicolasdh commented 2 years ago

Hello @saghul

Tried Docker Compose v1

Tried with: docker-compose version 1.29.2, build 5becea4c

Got same problem:

jibri_1    | Jibri 2022-04-28 05:19:34.787 INFO: [57] AbstractPageObject.visit#32: Visiting url https://MYPUBLICFQDN
jibri_1    | Jibri 2022-04-28 05:19:34.789 FINE: [44] org.jitsi.xmpp.extensions.DefaultPacketExtensionProvider.parse: Could not add a provider for element busy-status from namespace http://jitsi.org/protocol/jibri
jibri_1    | Jibri 2022-04-28 05:19:34.789 FINE: [44] org.jitsi.xmpp.extensions.DefaultPacketExtensionProvider.parse: Could not add a provider for element health-status from namespace http://jitsi.org/protocol/health
jibri_1    | Jibri 2022-04-28 05:19:44.905 SEVERE: [57] [session_id=mlcpyocjkytvxdhr] JibriSelenium$joinCall$1.run#325: An error occurred while joining the call
jibri_1    | org.openqa.selenium.WebDriverException: unknown error: net::ERR_CONNECTION_REFUSED
jibri_1    |   (Session info: chrome=96.0.4664.45)
jibri_1    |   (Driver info: chromedriver=96.0.4664.45 (76e4c1bb2ab4671b8beba3444e61c0f17584b2fc-refs/branch-heads/4664@{#947}),platform=Linux 5.15.0-27-generic x86_64) (WARNING: The server did not provide any stacktrace information)
jibri_1    | Command duration or timeout: 0 milliseconds

Note: MYPUBLICFQDN and PUBLIC_IP are entered in the ".env" in the parameter PUBLIC_URL=https://MYPUBLICFQDN DOCKER_HOST_ADDRESS=MYPUBLICIP

Workaround (adding internal IP for FQDN in /etc/hosts)

Adding the "internal" IP of the WEB container as MYPUBLICFQDN in the /etc/hosts inside the JIBRI container seems to be a "manual" workaround...

saghul commented 2 years ago

Is your public IP publicly routable? Does it have a valid TLS cert?

dvnicolasdh commented 2 years ago

Yes valid lets encrypt certificate and publicly accessible from internet.

Also accessible from the DOCKER HOST, even accessible from inside the "web" jitsi container (needed to apt install curl to test)... but strangely, not accessible (without manual tweak /etc/hosts) from inside "jibri" container (apt install curl to test) !

dvnicolasdh commented 2 years ago

Thanks @saghul ,

Formulating the problem, and your questions, helped solve the issue...

Docker routing seems confused with 127.0.1.1 entry in the DOCKER_HOST /etc/hosts

I had put an entry with MYPUBLICFQDN in the /etc/hosts of the DOCKER_HOST Ubuntu 22.04

127.0.0.1   localhost
127.0.1.1  MYPUBLICFQDN

Old habit of mine that solves dynamic IP issues with other systems, but seems to cause problem here... It seems that the 127.0.1.1 entry on the DOCKER_HOST confuses the routing of the containers...

JIBRI container works when removing the 127.0.1.1 entry in the DOCKER_HOST /etc/hosts

Removing the "127.0.1.1" entry on the DOCKER_HOST allows "JIBRI" container to connect without the "work-around"....

Worked with: docker-compose version 1.29.2, build 5becea4c and also with: Docker version 20.10.14, build a224086 without the need of the /etc/hosts "manual" work-around...

Just need to make sure "oldies" don't set the 127.0.1.1 that may confuse docker internal routing...

https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_hostname_resolution

For a system with a permanent IP address, that permanent IP address should be used here instead of 127.0.1.1.

Thanks.. :)

saghul commented 2 years ago

Good to hear!

JacBra commented 2 years ago

I have the same issue, but I don't have that 127.0.1.1 in my /etc/hosts. I am using CentOS 7 as docker host OS. In https://github.com/jitsi/docker-jitsi-meet/issues/1075 a workaround was found that works, but there is no structural solution yet. Note that my public IP is publicly routable and has a valid certificate. Any ideas?

dvnicolasdh commented 2 years ago

Hello @JacBra , Is it possible that you have placed your MYPUBLICFQDN (your public url without the https:// ) on the 127.0.0.1 line of your /etc/hosts of the DOCKER_HOST ( your CentOS 7)? If it is the case, you may want to try leaving 127.0.0.1 to localhost (without the MYPUBLICFQDN). And may add an entry with your MYPUBLICIP (pointing to MYPUBLICFQDN)

/etc/hosts. (on the DOCKER_HOST)

127.0.0.1 localhost

or

/etc/hosts. (on the DOCKER_HOST)

127.0.0.1 localhost
MYPUBLICIP MYPUBLICFQDN

replacing MYPUBLICIP MYPUBLICFQDN with your actual public IP and URL(without the https://). Maybe this could allow not using the workaround from https://github.com/jitsi/docker-jitsi-meet/issues/1075 ?

caos30 commented 1 month ago

I solved in a very simple and reliable way, editing docker-compose.yml file and defining STATIC IPs :-)

First, in the section where defining network do it in this way:

networks:
    meet.jitsi:
      ipam:
         driver: default
         config:
          - subnet: 172.20.0.0/16

Then for each one of the services, also on their "network" attribute set something like this:

        networks:
            meet.jitsi:
              ipv4_address: 172.20.0.6

Taking in account that:

  1. you don't assign the same IP address to two different services
  2. you don't use the 172.20.0.1, because is ussed by the gateway (i think so)

Then and finally, edit JIBRI service definition to include a new attribute extra_hosts which will add this entry (or more you mention) to the /etc/hosts file of the container (!!!) when mounting it :-)

        depends_on:
            - jicofo
        extra_hosts:
            - "live.mydomain.com:172.20.0.2"
        networks:
            meet.jitsi:
              ipv4_address: 172.20.0.6

Note: replace live.mydomain.com by the (sub)domain name of your meet public website, and replace 172.20.0.2 by the ipv4_address value you defined for the web service .

And voilà, then jibri container will be able ALWAYS to connect with the jitsi web service on the other container.

@saghul would you think that these settings would be useful to set on the official github docker-compose.yml file?