search

jitsi / jitsi-videobridge

Jitsi Videobridge is a WebRTC compatible video router or SFU that lets build highly scalable video conferencing infrastructure (i.e., up to hundreds of conferences per server).
https://jitsi.org/jitsi-videobridge
Apache License 2.0
2.91k stars 992 forks source link

Log overflow with VideoQualityLayerLookup.transform#44: Unable to find encoding matching packet! #1703

Open kiliankoller opened 3 years ago

kiliankoller commented 3 years ago

Description

With a fresh install of jitsi-meet on an Ubuntu 20 server with a public IP address, we are seeing hundrets of these warnings in the jvb.log, just with one test conference:

JVB 2021-08-10 09:45:11.189 WARNING: [448] [confId=8de3eca41115067e epId=eb383b8a gid=127837 stats_id=Malinda-tZC conf_name=a3srshkmnfxvq@conference.sichere-videokonferenz.de] VideoQualityLayerLookup.transform#44: Unable to find encoding matching packet! packet=RtpPacket: PT=96, Ssrc=3039135604, SeqNum=62186, M=true, X=true, Ts=3711187980; sources=MediaSourceDesc 1292347528 has encodings: primary_ssrc=2808700885,secondary_ssrcs={},layers= subjective_quality=0,temporal_id=0,spatial_id=-1 subjective_quality=1,temporal_id=1,spatial_id=-1 subjective_quality=2,temporal_id=2,spatial_id=-1 primary_ssrc=2343286782,secondary_ssrcs={},layers= subjective_quality=64,temporal_id=0,spatial_id=-1 subjective_quality=65,temporal_id=1,spatial_id=-1 subjective_quality=66,temporal_id=2,spatial_id=-1 primary_ssrc=2658891082,secondary_ssrcs={},layers= subjective_quality=128,temporal_id=0,spatial_id=-1 subjective_quality=129,temporal_id=1,spatial_id=-1 subjective_quality=130,temporal_id=2,spatial_id=-1

Expected Behavior

No or little amount of warnings in the logs

Steps to reproduce

Just install jitsi-meet on an Ubuntu 20 VM (Hetzner in my case) and start a conference with Firefox and join the room with Chrome on the same machine (OS X). In our case: https://test.sichere-videokonferenz.de/ I did not change any configurations.

Environment details

root@test:/etc/jitsi/meet# dpkg -l | grep jitsi ii jitsi-meet 2.0.5963-1 all WebRTC JavaScript video conferences ii jitsi-meet-prosody 1.0.5056-1 all Prosody configuration for Jitsi Meet ii jitsi-meet-web 1.0.5056-1 all WebRTC JavaScript video conferences ii jitsi-meet-web-config 1.0.5056-1 all Configuration for web serving of Jitsi Meet ii jitsi-videobridge2 2.1-508-gb24f756c-1 all WebRTC compatible Selective Forwarding Unit (SFU)

Also see https://community.jitsi.org/t/videoparser-transform-58-unable-to-find-encoding-matching-packet/60290

kiliankoller commented 3 years ago

The error remains after this update: root@test:~# dpkg -l | grep jitsi ii jitsi-meet 2.0.6173-1 all WebRTC JavaScript video conferences ii jitsi-meet-prosody 1.0.5211-1 all Prosody configuration for Jitsi Meet ii jitsi-meet-web 1.0.5211-1 all WebRTC JavaScript video conferences ii jitsi-meet-web-config 1.0.5211-1 all Configuration for web serving of Jitsi Meet ii jitsi-videobridge2 2.1-538-g062e9f56-1 all WebRTC compatible Selective Forwarding Unit (SFU)

fluboi commented 3 years ago

Issue still present with latest stable jvb 2.1-538-g062e9f56-1 Hundreds of messages per second (on a pretty small setup) complaining "Unable to find encoding matching packet!"

kiliankoller commented 3 years ago

Still there with ii jitsi-videobridge2 2.1-551-g2ad6eb0b-1 all WebRTC compatible Selective Forwarding Unit (SFU)

p-try commented 3 years ago

We are also experiencing this on Ubuntu 20.04. Our VM is hosted by Hetzner, like OP's is, too (coincidence or clue?).

el-dge commented 3 years ago

Same log, in debian 11 upated and stable jitsi updated on (VM Public Cloud OVH) :

ii  jitsi-meet                        2.0.6293-1                     all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                1.0.5307-1                     all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-turnserver             1.0.5307-1                     all          Configures coturn to be used with Jitsi Meet
ii  jitsi-meet-web                    1.0.5307-1                     all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config             1.0.5307-1                     all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                2.1-551-g2ad6eb0b-1            all          WebRTC compatible Selective Forwarding Unit (SFU)
fluboi commented 3 years ago

Not 100% sure yet, but using last stable jvb2 version (2.1-570-gb802be83-1) seems to fix that.

kiliankoller commented 3 years ago

I can confirm, that with the following versions, the warning is gone:

aCandidMind commented 2 years ago

Happens here with a later version:

dpkg -l | grep jitsi
ii  jitsi-meet                     2.0.6865-2                            all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody             1.0.5818-1                            all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-web                 1.0.5818-1                            all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config          1.0.5818-1                            all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2             2.1-617-ga8b39c3f-1                   all          WebRTC compatible Selective Forwarding Unit (SFU)

Started only with my desktop browser, the messages appeared as soon as I joined with the iOS app which had a super-sharp video quality (note the component VideoQualityLayerLookup.transform). After a while the stream for the iOS client gets disabled for bandwidth reasons on my desktop browser. Doesn't happen when joining from another private browser window on the same desktop so I guess it's the iOS video quality. Those lines only appear the moment the iOS video actually starts, just joining from iOS without video doesn't trigger it. Using standard configs for the server.

Also received a lot of #1769 when iOS enables video:

Transceiver$rtpReceiver$1.invoke#116: Sending large locally-generated RTCP packet of size 15436, first packet of type 205.
2022-02-06 20:14:52.088 WARNING: [82] ByteBufferPool.returnBuffer#226: Received a suspiciously large buffer (size = 15436)
citronalco commented 1 year ago

Still having this. And streams get disabled "to save bandwith".

ii  jitsi-meet                       2.0.8138-1                     all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody               1.0.6854-1                     all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-turnserver            1.0.6854-1                     all          Configures coturn to be used with Jitsi Meet
ii  jitsi-meet-web                   1.0.6854-1                     all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config            1.0.6854-1                     all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2               2.2-63-g252d14bc-1             all          WebRTC compatible Selective Forwarding Unit (SFU)
ii  lua-basexx                       0.4.1-jitsi1                   all          baseXX encoding/decoding library for Lua
ii  lua-cjson:amd64                  2.1.0.10-jitsi1                amd64        JSON parser/encoder for Lua
Rosch commented 3 months ago

Happening here for me with the following version

ii  jitsi-meet                            2.0.8960-1                              all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                    1.0.7531-1                              all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-turnserver                 1.0.7531-1                              all          Configures coturn to be used with Jitsi Meet
ii  jitsi-meet-web                        1.0.7531-1                              all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config                 1.0.7531-1                              all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                    2.3-44-g8983b11f-1                      all          WebRTC compatible Selective Forwarding Unit (SFU)
ii  lua-basexx                            0.4.1-jitsi1                            all          baseXX encoding/decoding library for Lua
ii  lua-cjson:amd64                       2.1.0.10-jitsi1                         amd64        JSON parser/encoder for Lua
kiliankoller commented 3 months ago

For us, the error came back too, 23.880 of these errors yesterday only...

ii  jitsi-meet                            2.0.9584-1                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                    1.0.8043-1                        all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-tokens                     1.0.8043-1                        all          Prosody token authentication plugin for Jitsi Meet
ii  jitsi-meet-web                        1.0.8043-1                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config                 1.0.8043-1                        all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                    2.3-149-g793df5a9-1               all          WebRTC compatible Selective Forwarding Unit (SFU)
ii  lua-basexx                            0.4.1-jitsi1                      all          baseXX encoding/decoding library for Lua
ii  lua-cjson:amd64                       2.1.0.10-jitsi1                   amd64        JSON parser/encoder for Lua
Rosch commented 3 months ago

Just from midnight to now (1pm) I got 909898 messages and there had just been 3 shorter meetings today with just ~4 participiants each.

Is there anything what can be done to fix this?

damencho commented 3 months ago

Try updating the latest from testing, those versions will soon be pushed to stable.

damencho commented 3 months ago

The new stable is out, try it.

RhysRdm commented 3 months ago

Hey @damencho I have tested this against docker version stable-9646 and i still see the debug. Is the fix intended to be in this version?

damencho commented 3 months ago

Can you share the message you see with the latest version? What is the scenario you test? Is it easy to reproduces? What browser versions dod you use to reproduce this?

kiliankoller commented 3 months ago

Hi, I can confirm, yesterday we still got a 144487 of these messages in a total of 668055 lines:

JVB 2024-08-04 17:45:22.332 WARNING: [181] [confId=c4fba3a2d0995add conf_name=fsogtwuyp6zh2@conference.sichere-videokonferenz.de meeting_id=541d968a epId=c954fa82 stats_id=Imelda-mpU] VideoQualityLayerLookup.transform#44: Unable to find encoding matching packet! packet=RtpPacket: PT=100, Ssrc=3442339265, SeqNum=1652, M=false, X=true, Ts=2811464945, EncID=-1, TID=-1; sources=MediaSourceDesc[name=c954fa82-v0 owner=c954fa82, videoType=CAMERA, encodings=primary_ssrc=312312174,secondary_ssrcs={2793454169=RTX},layers=

This one comes up 43280 times too:

JVB 2024-06-17 11:19:00.467 WARNING: [377] [confId=25876e56a78b0d98 conf_name=knzd8etwyk8fp@conference.sichere-videokonferenz.de meeting_id=e5728590 epId=05b03d9f stats_id=Reynold-E5h] SingleSourceAllocation.improve#133: Oversending above maxOversendBitrate, layer bitrate 758864.0 bps

Versions:

ii  jitsi-meet                            2.0.9646-1                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-prosody                    1.0.8091-1                        all          Prosody configuration for Jitsi Meet
ii  jitsi-meet-tokens                     1.0.8091-1                        all          Prosody token authentication plugin for Jitsi Meet
ii  jitsi-meet-web                        1.0.8091-1                        all          WebRTC JavaScript video conferences
ii  jitsi-meet-web-config                 1.0.8091-1                        all          Configuration for web serving of Jitsi Meet
ii  jitsi-videobridge2                    2.3-160-g97a1f15b-1               all          WebRTC compatible Selective Forwarding Unit (SFU)
ii  lua-basexx                            0.4.1-jitsi1                      all          baseXX encoding/decoding library for Lua
ii  lua-cjson:amd64                       2.1.0.10-jitsi1                   amd64        JSON parser/encoder for Lua

I am not able to reproduce this specifically, I can only see it in a production system.

RhysRdm commented 3 months ago

Can you share the message you see with the latest version? What is the scenario you test? Is it easy to reproduces? What browser versions dod you use to reproduce this?

Sure:

Aug 19 10:56:52 ip-10-0-1-10.ec2.internal matrix-jitsi-jvb[25110]: JVB 2024-08-19 10:56:52.401 WARNING: [73] [confId=e445df009f6fefdb conf_name=efggwt3kj5iu6q2tn5ze22sbn5uvcuz2orsxg5bomnwgkylsonygc3tvmmx gg33n@muc.meet.jitsi meeting_id=8d9e741e epId=828f3baf stats_id=Jimmie-IYw] VideoQualityLayerLookup.transform#44: Unable to find encoding matching packet! packet=RtpPacket: PT=97, Ssrc=1842180385, SeqNum =24012, M=true, X=true, Ts=3327251257, EncID=-1; sources=

The scenario was a two party jitsi vudei conference and seems to produce every time.

Im using chrome Version 126.0.6478.126 (Official Build) (64-bit) and the lib jitsi meet client libraries

thisisnkc commented 3 months ago

+1, same happening on our jvb as well. my version is 8960 chrome version is 126.0.6478.126. @damencho @saghul please help this issue is causing to crash the jitsi meet frequently !

aslam-t commented 2 months ago

Started seeing this issue today on docker-jitsi-meet stable-8960 with mostly participants on Chrome. Few of them were kicked out. Could it be related to latest chrome itself?

fueleh commented 1 month ago

We got this problem when we enabled H264 on jitsi 9646. The jvb log was spamming about AV1 codec and same time screenshare got blurry and cameras died. we thought but we are not using AV1? why is it spamming about AV1? so we disabled AV1 in jicofo and it stopped and jitsi works alot better.

codec { video {

    vp8 {
      enabled = true
    }

    vp9 {
      enabled = false
    }

    av1 {
      enabled = false
    }

    h264 {
      enabled = true
    }

  }
  audio {

  }
}
kiliankoller commented 3 weeks ago

I added the following lines in the Jitsi Meet config /etc/jitsi/meet/HOSTNAME-config.js:

    videoQuality: {
        codecPreferenceOrder: ['VP9', 'VP8', 'H264'],
        mobileCodecPreferenceOrder: [ 'VP8', 'H264', 'VP9' ],
    },

    ...

    p2p: {
        codecPreferenceOrder: ['VP9', 'VP8', 'H264'],
        mobileCodecPreferenceOrder: [ 'VP8', 'H264', 'VP9' ],
    },

I can confirm, that this drastically reduced the no. of VideoQualityLayerLookup.transform#44: Unable to find encoding matching packet! errors in the log files.