Fixme: Session should be removed

[fabfreitas]

Prerequisites

• [X] I agree to fill this issue template.
• [X] I have read the Troubleshooting Guide and Support Instructions.

Issue description

I am getting - indefinitely - the message: fixme KurentoWebSocketTransport WebSocketTransport.cpp:352 keepAliveSessions() Session should be removed

I already set the Garbage Collector do 60 seconds, but the old sessions still keep.

Context

After many concourrent connections and the client is gone, the sessions still kept on server.

How to reproduce?

Using this tool webrtc-benchmark:

1. Run the app with this java arguments: -Dkms.ws.uri=wss://[IP SERVER]:8433/kurento -Dfake.kms.ws.uri=wss://[IP SERVER]:8433/kurento

   Note: You can use the ws protocol too.

2. In a brower go to localhost:8443 and start a connection as Presenter.

3. Open a new tab, go to the same url at step 2, configure the parametres to 1700 fake clients and 500 miliseconds between each connection and start the connections as Viewer.

4. In your server, open the Log and observe.

5. At the end of connections or even at the middle of process, stop (pushing the stop button on web app). First to viewers and next to Presenter.

6. Stop the KMS, Start the KMS and see the log. Soon many messages of Session shoulb be removed is going to be showed.

Expected & current behavior

The fix message about sessions that should be removed are showing forever.

Possible solution

In the WebSocketTransport.cpp at the line 348 we have this:

try {
        processor->keepAliveSession (c);<br/>
} catch (KurentoException &e) {
        if (e.getCode() == INVALID_SESSION) {
          GST_FIXME ("Session should be removed: %s", c.c_str() );
          // TODO: Remove session
        } else {
          throw e;
        }
}

Could this code resolve this TODO?

try {
        processor->keepAliveSession (c);<br/>
} catch (KurentoException &e) {
        if (e.getCode() == INVALID_SESSION) {
              closeHandler(c);
        } else {
             throw e;
        }
}

INFO about Kurento Media Server

• Kurento version: 6.15.0
• Server OS: Ubuntu 16.04.6 LTS (Xenial Xerus)
• Installation method:
  • X

INFO about your Application Server

• Programming Language: Java
• Kurento Client version: 6.15.0, Maven Repository

INFO about end-user clients

• Device(s): PC, Mac, Android, iPhone, iPad, Android Tablets
• OS(es): All compatible with the browsers above
• Browser(s): Firefox 74, Chrome 80, Safari 12.0

INFO about your environment

Kurento Server

• Number of servers: Two
• Protocol in use: wss
• Port: 8433
• Garbage Collector: 60s
• STUN or TURN: no
• Port range: Same suggested for STUN in RFC 3489
• Processors: 8
• RAM: 32GB

Cliente Application

• Java SDK 11
• Spring Boot 2.2.6.RELEASE
• kurento-client, kurento-utils-js and kurento-jsonrpc-client: v6.15.0

Run these commands

cat /etc/lsb-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

kurento-media-server --version

Kurento Media Server version: 6.15.0
Found modules:
    'core' version 6.15.0
    'elements' version 6.15.0
    'filters' version 6.15.0

dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'

ii  gstreamer1.5-libav:amd64              1.8.1-1kurento1.16.04~20200506171443.gbpb33143  amd64        libav plugin for GStreamer
ii  gstreamer1.5-nice:amd64               0.1.16-1.16.04~20200506171443.gbp1854ac         amd64        ICE library (GStreamer 1.5 plugin)
ii  gstreamer1.5-plugins-bad:amd64        1.8.1-1kurento4.16.04~20200506171443.gbp17d48d  amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.5-plugins-base:amd64       1.8.1-1kurento2.16.04~20200506171443.gbp7fb324  amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-good:amd64       1.8.1-1kurento3.16.04~20200506171443.gbp9c3835  amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-ugly:amd64       1.8.1-1kurento1.16.04~20200506171443.gbpbed8b0  amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.5-pulseaudio:amd64         1.8.1-1kurento3.16.04~20200506171443.gbp9c3835  amd64        GStreamer plugin for PulseAudio
ii  kms-core                              6.15.0-0kurento1.16.04                          amd64        Kurento Core module
rc  kms-core-6.0                          6.6.3.xenial.20170811182512.2704d13             amd64        Kurento core module
ii  kms-elements                          6.15.0-0kurento1.16.04                          amd64        Kurento Elements module
rc  kms-elements-6.0                      6.6.3.xenial.20170812234356.9e3fff4             amd64        Kurento elements module
ii  kms-filters                           6.15.0-0kurento1.16.04                          amd64        Kurento Filters module
ii  kms-jsonrpc                           6.15.0-0kurento1.16.04                          amd64        Kurento JSON-RPC library
ii  kmsjsoncpp                            1.6.3-1kurento1.16.04~20200506171443.gbp15940b  amd64        Kurento jsoncpp library
ii  kurento-media-server                  6.15.0-0kurento1.16.04                          amd64        Kurento Media Server
rc  kurento-media-server-6.0              6.7.0.xenial~20170813003033.1.6ebaa27           amd64        Kurento Media Server
ii  libgstreamer-plugins-bad1.5-0:amd64   1.8.1-1kurento4.16.04~20200506171443.gbp17d48d  amd64        GStreamer development files for libraries from the "bad" set
ii  libgstreamer-plugins-base1.5-0:amd64  1.8.1-1kurento2.16.04~20200506171443.gbp7fb324  amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.5-0:amd64               1.8.1-1kurento2.16.04~20200506171443.gbp84c84b  amd64        Core GStreamer libraries and elements
ii  libnice10:amd64                       0.1.18-0kurento1.16.04                          amd64        ICE library (shared library)
ii  libsrtp0:amd64                        1.6.0-0kurento1.16.04                           amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libsrtp0-dev                          1.6.0-0kurento1.16.04                           amd64        Secure RTP (SRTP) and UST Reference Implementations - development files
ii  libusrsctp                            0.9.2-1kurento1.16.04~20200506171443.gbpfd1d0e  amd64        sctp-refimpl library
ii  openh264                              1.5.0-0kurento1.16.04                           amd64        H.264 Video Codec provided by Cisco Systems, Inc.
ii  openh264-gst-plugins-bad-1.5:amd64    1.8.1-1kurento4.16.04~20200506171443.gbp17d48d  amd64        GStreamer plugins from openh264
ii  openwebrtc-gst-plugins                0.10.0-1kurento1.16.04~20200506171443.gbp6a0d22 amd64        OpenWebRTC specific GStreamer plugins

[github-actions[bot]]

Hello @fslayer! :wave: we're sorry you found a bug... so first of all, thank you very much for reporting it.

To know about progress, check in Triage. All issues are considered Backlog Candidates until work priorities align and the issue is selected for development. It will then become part of our official Backlog.

[j1elo]

6. Stop the KMS, Start the KMS and see the log. Soon many messages of Session shoulb be removed is going to be showed.

Just to confirm: in step 6, you actually mean to completely stop the media server, i.e. finish the kurento-media-server process, so KMS is not running at all, and then re-launch it?

If that is the case, then I believe the "TODO" comment was left there because what needs to be done is more complicated: it's all the clients the ones that need to do cleanup on their sides, as in this scenario, KMS has been relaunched so it doesn't have in memory any data of any past session.

If I understood the step 6 correctly, your scenario is this:

• Lots of clients are connected to KMS and are operating on a Session ID, e.g. AABBCC
• Clients stop their transmissions when you stop all at step 5, but that doesn't mean they lose their session ID for further possible operations with the media server. So the session still is alive.
• The media server is stopped. It releases all resources: file handlers, sockets, memory, etc.
• All clients detect a disconnection from the media server; their reconnection mechanisms are triggered and they start attempting to reconnect.
• The media server is started. It acquires initial resources: JSON-RPC sockets (ports 8888 / 8433), etc.
• Clients achieve reconnection with the media server. They tell the media server their session ID AABBCC, which at this point is unknown to the media server, thus this message appears.

In this scenario, which I believe is the one you describe, the meaning of the message is: "TODO: at this point, clients should be notified that their session ID is no longer valid and they should dispose it".

Your proposed solution, on the other hand, would forcefully close the connection with the client. So, the client would either fail with a "connection rejected from server", or more possibly, think that a connectivity interruption has happened, and simply try to reconnect again.

This is all my understanding, in theory. To verify if it matches in practice, you could carefully observe the behavior of clients to verify if things are happening like that. It also would be interesting if you test the code changes that you proposed; it's just a matter of editing the code in kurento-media-server and then building your own .deb package, which is almost trivial by using the kurento-buildpackage Docker image as explained in Create Deb packages.

[fabfreitas]

@j1elo, one more issue that I observe in 6.15.0 is the fact if I restart the service, via systemclt restart, the IPV4 and IPV6 fails to bound the ports. When I run $systemctl stop kurento... the service take a while to stop and, only after this, I start the service again.

So, in step 6 I stop the server, see the running processes, make sure that KMS isn't active, start again and see the log to make sure that the port 8433 is listening.

About the reconnections:

I am testing multiple connections on 1 session ( 1 Presenter, 1700 viewers). With about 1650 connections, the server start to response timeout error, as the example below:

2020-11-18 13:13:39,958 ERROR [pool-4-thread-1685] UserSession: [Error on creating client 1685] [KurentoClient]  Timeout of 10000 milliseconds waiting from response to request {"id":52859,"method":"invoke","params":{"object":"75898415-c531-4c9a-b9e6-0d40af30d2ff_kurento.MediaPipeline/a61ff26c-1856-44ee-9789-fb674b5c0a1a_kurento.WebRtcEndpoint","operation":"getMediaPipeline","sessionId":"e29845a8-a1c2-497d-85c8-5844e0559d13"},"jsonrpc":"2.0"}

I guess this timeout is setted in some source-code, because I tried pass new values throught the kurento-client, but doesn't work. I tried pass via properties too.

-Dkurento.client.connectionTimeout=60000
-DjsonRpcServerWebSocket.timeout=25000

When this timeout occours, I stop the Viewers and the Presenter on the app, but the service on KMS server stays "stucked" and I have to stop and start it again the release resources. When it goes back on air, the messages about sessions that should be removed starts to fill of my log.

[tomasyany]

Why was this issue closed? The referenced PR only adds info / documentation, but doesn't solve the underlying problem.

The question is: should Kurento kill idle sessions, or should clients do it, isn't it?