sipwise / rtpengine

The Sipwise media proxy for Kamailio
GNU General Public License v3.0
787 stars 370 forks source link

Crash while rtpengine restoring data from redis #1488

Open kertor opened 2 years ago

kertor commented 2 years ago

Hello! After updating from:

10.0.0.0.20211202.ca0c6993

to

10.0.0.0.20220411.f160a303

I face with crashes in case when rtpengine restoring data from redis. I saw resolved issue https://github.com/sipwise/rtpengine/issues/1486 but not sure, that these are the same problems.

Steps: 1) Make one successful call through opensips; 2) When call will be establish, do rtpengine restart; 3) rtpengine will crash immediately after start, because in redis present data about established call. rtpengine try to restore this data, but here happened the problem.

In back trace i see this:

#0  0x000000000045a898 in json_link_streams (c=0x7f8d18020270, streams=0x7f8d1effc720, sfds=0x7f8d1effc740, medias=0x7f8d1effc700, root_reader=0x279ea60) at redis.c:1724
        sink = 0x7f8d18026810
        cs = 0x0
        ps_ml = 0x0
        i = 0
        ps = 0x7f8d18026070
        q = {head = 0x7f8d1801d4e0, tail = 0x7f8d1801d4e0, length = 1}
        l = 0x7f8d1801d4e0
#1  0x000000000045bdf3 in json_restore_call (r=0x27b4240, callid=0x7f8d1effc890, foreign=false) at redis.c:1992
        rr_jsonStr = 0x7f8d18001750
        call = {ht = 0x7f8d180172a0}
        tags = {len = 2, rh = 0x7f8d1801e410, ptrs = 0x7f8d1801e3f0}
        sfds = {len = 4, rh = 0x7f8d1801eec0, ptrs = 0x7f8d1801eef0}
        streams = {len = 4, rh = 0x7f8d1801f8a0, ptrs = 0x7f8d1801f8d0}
        medias = {len = 2, rh = 0x7f8d18009df0, ptrs = 0x7f8d18007780}
        maps = {len = 2, rh = 0x7f8d18008c50, ptrs = 0x7f8d18007760}
        c = 0x7f8d18020270
        s = {s = 0x0, len = 0}
        id = {s = 0x7f8d18009e70 "1.1.1.1", len = 14}
        meta = {s = 0x0, len = 0}
        last_signal = 1650972903
        err = 0x532157 "failed to link streams"
        i = 0
        root_reader = 0x279ea60
        parser = 0x27afe20
#2  0x000000000045ca85 in restore_thread (call_p=0x27a6610, ctx_p=0x7ffc2ee6b730) at redis.c:2083
        ctx = 0x7ffc2ee6b730
        call = 0x27a6610
        r = 0x27b4240
        callid = {s = 0x27a3a90 "NWJlOGIwZDE0MDkxYjJiNzAzMzA2NzNhODNiY2Q5MDM.", len = 44}
#3  0x00007f8d2e6118a3 in g_thread_pool_thread_proxy (data=<optimized out>) at gthreadpool.c:307
        task = 0x27a6610
        pool = 0x27a58a0
#4  0x00007f8d2e610e9a in g_thread_proxy (data=0x279eb20) at gthread.c:784
        thread = 0x279eb20
        __func__ = "g_thread_proxy"
#5  0x00007f8d2b5c615a in start_thread (arg=<optimized out>) at pthread_create.c:479
        ret = <optimized out>
        pd = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140244087199488, -4778417005054423560, 140721095357822, 140721095357823, 140721095357952, 140244087196160, 4806918543667596792, 4807013940485262840}, mask_was_saved = 0}}, priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007f8d2b2f5dd3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

In rtpengine logs i see only this:

rtpengine[870345]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: DEBUG: [core] Restoring calls from Redis...
rtpengine[870346]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: INFO: [core] Established initial connection to Redis 127.0.0.1:6379
rtpengine[870346]: DEBUG: [core] Processing call ID 'BLmIOTJEdaRw6bRuNL-P8g..' from Redis
rtpengine[870346]: DEBUG: [core] Processing call ID 'NWJlOGIwZDE0MDkxYjJiNzAzMzA2NzNhODNiY2Q5MDM.' from Redis

Redis version is:

hiredis.x86_64                         0.13.3-13.el8                                @ol8_EPEL         
redis.x86_64                           5.0.3-2.module+el8.0.0.z+5250+19ca22c8       @ol8_appstream

Could you please check this? Thank you!

rfuchs commented 2 years ago

First I would suggest to upgrade to a newer version because 10.0 is no longer supported.

kertor commented 2 years ago

Thank you, will update to current version.