Colibri REST API disabled on jitsi-videobridge update

[ankar84]

Description

We on Ubuntu 18.04. We were on 2.0.6293 (2021-09-10) and updated to 2.0.6433 (2021-10-07). Jitsi-videobridge was 2.1-551-g2ad6eb0b-1 and now 2.1-570-gb802be83-1

We have JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/videobridge/jvb.conf in our /etc/jitsi/videobridge/config file.

We had JVB_OPTS="--apis=rest," in our /etc/jitsi/videobridge/config file. Also (and most important) we have this in /etc/jitsi/videobridge/jvb.conf file

videobridge {
  apis {
       rest {
      enabled = true
    }
  }
  http-servers {
    public {
      port = 9090
      tls-port = 443
    }
    private {
      host = 127.0.0.1
    }
}

So, according to https://github.com/jitsi/jitsi-videobridge/blob/master/doc/statistics.md and mostly https://github.com/jitsi/jitsi-videobridge/blob/master/doc/rest.md we should be enabled Colibri REST API that way (jvb.conf file)

But after update we get JVB_OPTS="--apis=," in our /etc/jitsi/videobridge/config file! And all conference statistic gone. I mentioned it here https://github.com/jitsi/jitsi-meet/issues/10415

So, now I have 2 questions:

1. Why update postinst script deleted rest in --apis?
2. Why Colibri REST API don't work despite enabled in jvb.conf file?

Current behavior

Statistics gone during update to recent jitsi-videobridge2 packet.

rest is always enable = true in jvb.conf file but I get this results: with JVB_OPTS="--apis=rest," in our /etc/jitsi/videobridge/config file I get

curl -s --noproxy "*" http://127.0.0.1:8080/colibri/stats

``` {"inactive_endpoints":0,"inactive_conferences":0,"total_ice_succeeded_relayed":0,"octo_version":1,"total_loss_degraded_participant_seconds":0,"bit_rate_download":0.0,"local_active_endpoints":0,"muc_clients_connected":1,"total_participants":0,"total_packets_received":0,"rtt_aggregate":0.0,"packet_rate_upload":0,"p2p_conferences":0,"total_aimd_bwe_expirations":0,"total_loss_limited_participant_seconds":0,"preemptive_kfr_suppressed":0,"local_endpoints":0,"octo_send_bitrate":0,"total_dominant_speaker_changes":0,"endpoints_with_spurious_remb":0,"receive_only_endpoints":0,"total_colibri_web_socket_messages_received":0,"octo_receive_bitrate":0,"tossedPacketsEnergy":{"average":null,"max":0,"total_value":0,"total_count":0,"buckets":{"<= 7":0,"<= 15":0,"<= 23":0,"<= 31":0,"<= 39":0,"<= 47":0,"<= 55":0,"<= 63":0,"<= 71":0,"<= 79":0,"<= 87":0,"<= 95":0,"<= 103":0,"<= 111":0,"<= 119":0,"> 119":0,"p99<=":-1,"p999<=":-1}},"version":"2.1.570-gb802be83","total_ice_succeeded":0,"total_colibri_web_socket_messages_sent":0,"total_bytes_sent_octo":0,"total_data_channel_messages_received":0,"total_conference_seconds":0,"num_eps_oversending":0,"bit_rate_upload":0.0,"total_conferences_completed":0,"octo_conferences":0,"num_eps_no_msg_transport_after_delay":0,"region":"nsk","endpoints_sending_video":0,"packet_rate_download":0,"muc_clients_configured":1,"outgoing_loss":0.0,"overall_loss":0.0,"conference_sizes":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"total_packets_sent_octo":0,"conferences_by_video_senders":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"endpoints_with_high_outgoing_loss":0,"stress_level":0.0,"jitter_aggregate":0.0,"total_ice_succeeded_tcp":0,"octo_endpoints":0,"current_timestamp":"2021-11-23 11:41:57.724","total_packets_dropped_octo":0,"conferences":0,"participants":0,"average_participant_stress":0.01,"largest_conference":0,"total_packets_sent":0,"endpoints":0,"total_data_channel_messages_sent":0,"incoming_loss":0.0,"total_bytes_received_octo":0,"octo_send_packet_rate":0,"conferences_by_audio_senders":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"total_conferences_created":0,"total_ice_failed":0,"preemptive_kfr_sent":0,"threads":35,"videochannels":0,"total_packets_received_octo":0,"graceful_shutdown":false,"octo_receive_packet_rate":0,"total_bytes_received":0,"total_loss_controlled_participant_seconds":0,"total_partially_failed_conferences":0,"endpoints_sending_audio":0,"dtls_failed_endpoints":0,"total_bytes_sent":0,"mucs_configured":1,"total_failed_conferences":0,"mucs_joined":1,"relay_id":"172.18.106.170:4092"}root@s-rc-jvb-prp-01:~# ```

with JVB_OPTS="--apis=," in our /etc/jitsi/videobridge/config file I get

curl -s --noproxy "*" http://127.0.0.1:8080/colibri/stats

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404 Not Found</h2>
<table>
<tr><th>URI:</th><td>/colibri/stats</td></tr>
<tr><th>STATUS:</th><td>404</td></tr>
<tr><th>MESSAGE:</th><td>Not Found</td></tr>
<tr><th>SERVLET:</th><td>org.glassfish.jersey.servlet.ServletContainer-5b1f29fa</td></tr>
</table>
<hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.40.v20210413</a><hr/>

</body>
</html>

Here is open ports with both JVB_OPTS="--apis=," and JVB_OPTS="--apis=rest," in our /etc/jitsi/videobridge/config

netstat -tulnp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 127.0.0.1:8080          :::*                    LISTEN      24292/java

I see this https://github.com/jitsi/jitsi-videobridge/pull/1738 and this https://github.com/jitsi/jitsi-videobridge/pull/1736 that means devs plans to get rid of JVB_OPTS

Expected Behavior

REST API should be enabled after updated if it was enabled before update.

Possible Solution

I don't know.

Steps to reproduce

apt update && apt upgrade -y
reboot

Environment details

Ubuntu 18.04 Jitsi Meet 2.0.6433 (2021-10-07) jitsi-videobridge2 2.1-570-gb802be83-1

[ghborger]

I have the exact same behavior. Jitsi is running in docker containers. The Jitsi Stats worked fine for more than 12 month. Yesterday I make an update of the docker containers to the latest image. Jitsi works fine, but jitsi stats stoped working after the update.

When i call the URL i get an 404 Error:

curl -v http://127.0.0.1:8080/colibri/stats
* Expire in 0 ms for 6 (transfer 0x55caa6efafb0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55caa6efafb0)
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /colibri/stats HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 510
< Server: Jetty(9.4.44.v20210927)
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404 Not Found</h2>
<table>
<tr><th>URI:</th><td>/colibri/stats</td></tr>
<tr><th>STATUS:</th><td>404</td></tr>
<tr><th>MESSAGE:</th><td>Not Found</td></tr>
<tr><th>SERVLET:</th><td>org.glassfish.jersey.servlet.ServletContainer-78dc4696</td></tr>
</table>
<hr/><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.44.v20210927</a><hr/>

</body>
</html>
* Connection #0 to host 127.0.0.1 left intact

[f-asa]

I have the same issue, my docker image is build from jvb :stable-6726 and this update breaks those stats, I've been using those stats for metric collection for a long time also and broke overnight while updating to latest.

also does anyone know what is the actual behaviour of this setting? thank you!

  rest {
        shutdown {
            enabled = false
        }
    }  

jitsi-videobridge2/stable,now 2.1-595-g3637fda4-1

videobridge {
    ice {
        udp {
            port = 10000
        }
        tcp {
            enabled = false
            port = 4443

        }
    }
    apis {
        xmpp-client {
            configs {
                shard1 {
                    HOSTNAME = "jitsihostname.here"
                    DOMAIN = "auth.meet.jitsi"
                    USERNAME = "jvb"
                    PASSWORD = "securepassword"
                    MUC_JIDS = "jvbbrewery@internal-muc.meet.jitsi"
                    MUC_NICKNAME = "muc_nickname"
                    DISABLE_CERTIFICATE_VERIFICATION = true
                }
            }
        }
        rest {
            enabled = true
        }
    }
    rest {
        shutdown {
            enabled = false
        }
    }    

    stats {
        enabled = true
    }

    websockets {
        enabled = true
        domain = "jitsidomaine.here:443"
        tls = true
        server-id = "ip.ip.ip.ip"
    }
    http-servers {
        private { 
          host = 0.0.0.0
        }
        public {
            host = 0.0.0.0
            port = 9090
        }
    }

    octo {
        enabled = true
        bind-address = "0.0.0.0"
        public-address = "ip.ip.ip.ip"
        bind-port = "4096"
        region = "xyz"
    }
    }

ice4j {
    harvest {
        mapping {
            stun {
                addresses = [ "meet-jit-si-turnrelay.jitsi.net:443" ]
            }
        }
    }
}

[f-asa]

fixed by setting up

COLIBRI_REST_ENABLED=true SHUTDOWN_REST_ENABLED=true

[ghborger]

I found the problem in my configuration. In my docker-compose.yml in the jvb videobridge section, the environment settings

• COLIBRI_REST_ENABLED
• SHUTDOWN_REST_ENABLED were missing. After i added this settings, the colibri stats worked like before the update.

[primoitt83]

Solved here.. thx @f-asa

[MB175]

@ghborger solved it thanks, many docs are outdated on that

[yiyou-ezb]

fixed by setting up

COLIBRI_REST_ENABLED=true SHUTDOWN_REST_ENABLED=true

Hi, thanks for the solution, for a non-docker installation do you know where should I put these configurations in ? here /etc/jitsi/videobridge/config ?

[damencho]

In jvb.conf

https://github.com/jitsi/jitsi-videobridge/blob/674f841f1044f0ea3ff3e82b8ceb2364369af676/jvb/src/main/resources/reference.conf#L115

https://github.com/jitsi/jitsi-videobridge/blob/674f841f1044f0ea3ff3e82b8ceb2364369af676/jvb/src/main/resources/reference.conf#L129