heap-buffer-overflow after STUN request received

[carcharothellobo]

Hi, Im having some strange behaviour. My Servers start crashing ramdomly with heap-buffer-overflow after a message indicating an incoming STUN request. This only happen in my servers with version 0.11.7 and is not happening in my servers with version 0.6.1 I dont know how this is possible, I have only WSS and RTP ports opened in my firewall

stun request from 181.46.66.115:18239
=================================================================
==25==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60b000181361 at pc 0x7fbea799131e bp 0x7fbe92430550 sp 0x7fbe9242fd00
READ of size 192 at 0x60b000181361 thread T276
    #0 0x7fbea799131d  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x3f31d)
    #1 0x7fbe9ba99a98 in stun_parse_attribute (/usr/lib/libsofia-sip-ua.so.0+0x128a98)
    #2 0x7fbe9ba99b4c in stun_parse_message (/usr/lib/libsofia-sip-ua.so.0+0x128b4c)
    #3 0x7fbe9ba9a0a9  (/usr/lib/libsofia-sip-ua.so.0+0x1290a9)
    #4 0x7fbe9ba9a884 in stun_mini_request (/usr/lib/libsofia-sip-ua.so.0+0x129884)
    #5 0x7fbe9ba8b70f in tport_recv_stun_dgram (/usr/lib/libsofia-sip-ua.so.0+0x11a70f)
    #6 0x7fbe9ba85b0b in tport_recv_dgram (/usr/lib/libsofia-sip-ua.so.0+0x114b0b)
    #7 0x7fbe9ba81fd0 in tport_recv_event (/usr/lib/libsofia-sip-ua.so.0+0x110fd0)
    #8 0x7fbe9ba84502  (/usr/lib/libsofia-sip-ua.so.0+0x113502)
    #9 0x7fbe9ba6f99a  (/usr/lib/libsofia-sip-ua.so.0+0xfe99a)
    #10 0x7fbe9ba6d8f1 in su_base_port_run (/usr/lib/libsofia-sip-ua.so.0+0xfc8f1)
    #11 0x7fbe9ba6e084  (/usr/lib/libsofia-sip-ua.so.0+0xfd084)
    #12 0x7fbea68fefa2 in start_thread /build/glibc-vjB4T1/glibc-2.28/nptl/pthread_create.c:486
    #13 0x7fbea682f4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)

0x60b000181361 is located 0 bytes to the right of 97-byte region [0x60b000181300,0x60b000181361)
allocated by thread T276 here:
    #0 0x7fbea7a3b720 in __interceptor_realloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9720)
    #1 0x7fbe9ba674cf in su_realloc (/usr/lib/libsofia-sip-ua.so.0+0xf64cf)

Thread T276 created by T275 (sip 541433008) here:
    #0 0x7fbea79a2db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x7fbe9ba6e35d in su_pthreaded_port_start (/usr/lib/libsofia-sip-ua.so.0+0xfd35d)
    #2 0x10000012b  (<unknown module>)

Thread T275 (sip 541433008) created by T14 (sip handler) here:
    #0 0x7fbea79a2db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x7fbea7457ddf  (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x97ddf)

Thread T14 (sip handler) created by T0 here:
    #0 0x7fbea79a2db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x7fbea7457ddf  (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x97ddf)

[lminiero]

Since it's happening deep into Sofia SIP, you may want to make sure you're using a recent enough version of that library. If not, update Sofia and recompile Janus. Besides, 0.11.7 is almost a year old: please try Janus 0.13.0 as well.

[atoppi]

Also consider that Janus is now compatible with FreeSwitch Sofia SIP. Recent versions should be fetched from their repo, since they are the only maintainers of the project as far as I know.

[lminiero]

@carcharothellobo any update on the feedback above?

[carcharothellobo]

I had updated to the last version 1.1.1, but problem is still there. What I cant understand is why a simple STUN request can crash the whole service. I will try the freeswitch version of Sofia sip.

stun request from 186.130.105.249:56332
=================================================================
==25==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60b00047b0c1 at pc 0x7f2e2809131e bp 0x7f2e1265c570 sp 0x7f2e1265bd20
READ of size 192 at 0x60b00047b0c1 thread T36
    #0 0x7f2e2809131d  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x3f31d)
    #1 0x7f2e1c58d610 in stun_parse_attribute (/usr/lib/libsofia-sip-ua.so.0+0x127610)
    #2 0x7f2e1c58d6d4 in stun_parse_message (/usr/lib/libsofia-sip-ua.so.0+0x1276d4)
    #3 0x7f2e1c58dc1b  (/usr/lib/libsofia-sip-ua.so.0+0x127c1b)
    #4 0x7f2e1c58e346 in stun_mini_request (/usr/lib/libsofia-sip-ua.so.0+0x128346)
    #5 0x7f2e1c57fc47 in tport_recv_stun_dgram (/usr/lib/libsofia-sip-ua.so.0+0x119c47)
    #6 0x7f2e1c57a27b in tport_recv_dgram (/usr/lib/libsofia-sip-ua.so.0+0x11427b)
    #7 0x7f2e1c57681f in tport_recv_event (/usr/lib/libsofia-sip-ua.so.0+0x11081f)
    #8 0x7f2e1c578cc2  (/usr/lib/libsofia-sip-ua.so.0+0x112cc2)
    #9 0x7f2e1c564589  (/usr/lib/libsofia-sip-ua.so.0+0xfe589)
    #10 0x7f2e1c562561 in su_base_port_run (/usr/lib/libsofia-sip-ua.so.0+0xfc561)
    #11 0x7f2e1c562cea  (/usr/lib/libsofia-sip-ua.so.0+0xfccea)
    #12 0x7f2e271fafa2 in start_thread /build/glibc-6iIyft/glibc-2.28/nptl/pthread_create.c:486
    #13 0x7f2e2712c06e in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf906e)

[lminiero]

I had updated to the last version 1.1.1, but problem is still there

The problem is in your instance of Sofia, so no Janus update would fix it. You'll indeed have to install a more recent version of Sofia and reconfigure (touch configure.ac) and recompile (make clean before a new make install) Janus accordingly to see if that changes.

[carcharothellobo]

Ok, I had updated to last version of Freeswitch Sofia SIP (1.13.9). I don´t have any errors related to Stun requests, maybe this version fixed it, but now I have another error here, and also this error restart my Janus Server. It seems that it is triggered by a SUBSCRIBE, but I have other similars logs that don't crash the Janus Service.

[Mon Nov  7 09:09:18 2022] ESC[33m[WARN]ESC[0m [54128117] SUBSCRIBE failed: 481 Subscription does not exist
[Mon Nov  7 09:09:18 2022] [7030893788979851] Sending event to transport...
[Mon Nov  7 09:09:18 2022]   >> Pushing event to peer: 0 (Success)
[Mon Nov  7 09:09:18 2022] [54128117][nua_r_subscribe]: 900 Internal error at nua_client.c:713
[Mon Nov  7 09:09:18 2022] ESC[33m[WARN]ESC[0m [54128117] SUBSCRIBE failed: 900 Internal error at nua_client.c:713
AddressSanitizer:DEADLYSIGNAL
=================================================================
==24==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000090 (pc 0x7fb15875d753 bp 0x7fb148e4e190 sp 0x7fb148e444f0 T108)
==24==The signal is caused by a READ memory access.
==24==Hint: address points to the zero page.
    #0 0x7fb15875d752 in janus_sip_sofia_callback plugins/janus_sip.c:6254
    #1 0x7fb1585d8d52 in nua_application_event (/usr/lib/libsofia-sip-ua.so.0+0xc7d52)
    #2 0x7fb158646113 in su_base_port_execute_msgs (/usr/lib/libsofia-sip-ua.so.0+0x135113)
    #3 0x7fb158645ea1 in su_base_port_getmsgs (/usr/lib/libsofia-sip-ua.so.0+0x134ea1)
    #4 0x7fb158646208 in su_base_port_run (/usr/lib/libsofia-sip-ua.so.0+0x135208)
    #5 0x7fb1586424f8 in su_port_run (/usr/lib/libsofia-sip-ua.so.0+0x1314f8)
    #6 0x7fb1586435d3 in su_root_run (/usr/lib/libsofia-sip-ua.so.0+0x1325d3)
    #7 0x7fb158721ed3 in janus_sip_sofia_thread plugins/janus_sip.c:7263
    #8 0x7fb163c3c4d4  (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x764d4)
    #9 0x7fb163300fa2 in start_thread /build/glibc-6iIyft/glibc-2.28/nptl/pthread_create.c:486
    #10 0x7fb16323206e in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf906e)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV plugins/janus_sip.c:6254 in janus_sip_sofia_callback
Thread T108 (sip 54128117) created by T14 (sip handler) here:
    #0 0x7fb1641a8db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x7fb163c5dddf  (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x97ddf)

Thread T14 (sip handler) created by T0 here:
    #0 0x7fb1641a8db0 in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x50db0)
    #1 0x7fb163c5dddf  (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x97ddf)

==24==ABORTING

[lminiero]

Please open a different issue for this, so that we can mark the buffer overflow one as fixed.

[lminiero]

Even though it may be an easy fix: I see a dereference given for granted, so a simple check may do the trick. I just added a commit for that, please let me know if it fixes it for you: ef984223dd0cfbaa07445c18748f506899678305

[carcharothellobo]

Excellent, I will try it, if persists, I will open a new issue. Anyway while writing that STUN problem is solved my server crashed again with this problem. So this is not solved with last version of FreeSwitch Sofia SIP. I will try to get more infromation and I will place an Issue in Freeswitch Sofia SIP

[lminiero]

Ack, please keep us posted on the updates on that, if any!

[carcharothellobo]

Lorenzo Im still trying to get some response from Sofia Devs, but they are not responding since 6 days ago. They said that maybe is some miss validation. I think that if they put this validation we can avoid these crashes. But nobody es responding by now from Freeswitch Sofia SIP

https://github.com/freeswitch/sofia-sip/issues/180

[lminiero]

No, they're asking you for Sofia debug logs but you haven't provided any. Launch Janus with these env variables set, e.g.:

SOFIA_DEBUG=9 NUA_DEBUG=9 NTA_DEBUG=9 /opt/janus/bin/janus

(or whatever the path to Janus is on your system), and the logs will include debug Sofia logs too.

[carcharothellobo]

Excellent, thanks a lot, I will go for it

[carcharothellobo]

Lorenzo, I have an answer from Freeswitch SofiaSIP. I think they have some Security bug reported several years ago but it is not fixed in current version. I fixed by my own just adding some simple validation, but I think that this could be a problem for Janus. Maybe you want to take a look to that issue.

[lminiero]

Thanks for the heads up! I've subscribed to the thread to know when there's updates. Since it's an issue in Sofia, there's nothing we can do in Janus except wait until it's fixed there.

[carcharothellobo]

You can close this Issue by the way

[lminiero]

As a side note, did you configure the SIP plugin with behind_nat = true? https://github.com/meetecho/janus-gateway/blob/master/conf/janus.plugin.sip.jcfg.sample#L19-L22

[lminiero]

You can close this Issue by the way

I think it makes sense to keep this open until we know a timeline for the Sofia bug, so people encountering the problem have a reference.

[atoppi]

@carcharothellobo have you configured sofia with --disable-stun when compiling ? e.g.

make clean
./configure --prefix=/usr --with-pic --with-doxygen=no --disable-nth --disable-stun --enable-shared --disable-static
make -j$(nproc)
sudo make install

[carcharothellobo]

@carcharothellobo have you configured sofia with --disable-stun when compiling ? e.g.

make clean
./configure --prefix=/usr --with-pic --with-doxygen=no --disable-nth --disable-stun --enable-shared --disable-static
make -j$(nproc)
sudo make install

I didn't, I can try it, but as I understand this problem happens when Janus receive a STUN request for some reason I can't understand from some client. This is not a STUN response on a prevous Janus STUN request. And this is something a want to know. Our clients are using Google STUN server, I don't know why we are receiving these requests. Does Janus have some STUN failover pointing to the server?

[lminiero]

I think you're confused by what's happening. This is a STUN request sent within the context of a SIP call, so handled by Sofia. The STUN support on the WebRTC side (Janus itself) is an entirely different matter that happens in a completely different place, and handled by different libraries/code, so it has nothing to do with this issue.

What about the question I asked below yesterday?

As a side note, did you configure the SIP plugin with behind_nat = true? https://github.com/meetecho/janus-gateway/blob/master/conf/janus.plugin.sip.jcfg.sample#L19-L22

[carcharothellobo]

I couldn't test this option yet, I will let you know when I do it

[lminiero]

I just need to know if you're currently using it or not :hand_over_mouth: From what I can see in the code, STUN is only implicitly used in the SIP plugin via Sofia if behind_nat is set to true, otherwise we pass no-natify when creating the stack and so never ask Sofia to do any STUN for us. If that's the case, it means that by default Janus is not affected, since by default that property is not enabled. If you enabled without knowing what it actually did (and possibly assuming it had to do with WebRTC STUN, which it doesn't), disabling it should solve the crashes for you even without the patch.

[carcharothellobo]

this is my janus.plugin.sip.jcfg Im connecting via SIP to my Inernal PBXs, so there is no NAT between them.

general: {
        # Specify which local IP address to use. If not set it will be automatically
        # guessed from the system
        #local_ip = "1.2.3.4"

        # Enable local keep-alives to keep the registration open. Keep-alives are
        # sent in the form of OPTIONS requests, at the given interval inseconds.
        # (0 to disable)
        keepalive_interval = 120

        # Indicate if the server is behind NAT. If so, the server will use STUN
        # to guess its own public IP address and use it in the Contact header of
        # outgoing requests
        behind_nat = false

        # User-Agent string to be used
        # user_agent = "Cool WebRTC Gateway"

        # Expiration time for registrations
        register_ttl = 3600

        # Range of ports to use for RTP/RTCP (default=10000-60000)
        rtp_port_range = "57001-58000"

        # Whether events should be sent to event handlers (default is yes)
        #events = false
}

[lminiero]

    behind_nat = false

Ok, so looks like you have it explicitly disabled, which means my assumption was incorrect and the problem can occur nevertheless (not sure why Sofia would need to process STUN packets if the stack isn't sending any though). Let's wait for updates from the FreeSWITCH team.

[carcharothellobo]

Hi Lorenzo, Sofia SIP applied these changes in master, you can close this Issue I think

[lminiero]

Ack, closing, thanks!