search

signalwire / freeswitch

FreeSWITCH is a Software Defined Telecom Stack enabling the digital transformation from proprietary telecom switches to a versatile software implementation that runs on any commodity hardware. From a Raspberry PI to a multi-core server, FreeSWITCH can unlock the telecommunications potential of any device.
https://freeswitch.com/#getting-started
Other
3.5k stars 1.4k forks source link

Blacklist #983

Open tomlynn opened 3 years ago

tomlynn commented 3 years ago

Running version FreeSWITCH Version 1.10.5-release-6-25569c1631~32bit (-release-6-25569c1631 32bit) installed from packages ARM based Raspberry Pi system backtrace.log

Steps taken to cause the issue:

Received "Segmentation fault (core dumped)"

Expected +OK

Existng blacklist had only one entry: +12065238918

jungle-boogie commented 3 years ago

I'm also using a raspberry pi and don't have this issue with master: FreeSWITCH Version 1.10.6-dev+git~20201120T142952Z~8045c96458~32bit (git 8045c96 2020-11-20 14:29:52Z 32bit)

I wasn't aware of mod_blacklist, so I compiled freeswitch with this module on and enabled in freeswitch.

jungle-boogie commented 3 years ago

Actually! I think I'm running into the same issue:

freeswitch> blacklist save blocked
Socket interrupted, bye!
tomlynn commented 3 years ago

It happens for me more often than not. I can recall maybe two instances where it didn't produce the fault.

On Sat, Dec 12, 2020 at 1:14 PM jungle-boogie notifications@github.com wrote:

Actually! I think I'm running into the same issue:

freeswitch> blacklist save blocked Socket interrupted, bye!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/signalwire/freeswitch/issues/983#issuecomment-743880195, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFU3N54UHQWEC6STBV5KPRLSUPMKNANCNFSM4UXAVRBQ .

jungle-boogie commented 3 years ago

No problem on amd64:

freeswitch> blacklist add blocked 12065238918
+OK

2020-12-12 15:46:41.602457 [INFO] mod_blacklist.c:228 Added [12065238918] to list [blocked]
freeswitch> blacklist save blocked
+OK

2020-12-12 15:46:44.042457 [INFO] mod_blacklist.c:280 Saving blocked to /usr/local/freeswitch/blacklists/blocked.list
2020-12-12 15:46:44.042457 [DEBUG] mod_blacklist.c:288 adding 12065238918 to the dump file

version:

FreeSWITCH Version 1.10.6-dev+git~20201120T142952Z~8045c96458~64bit (git 8045c96 2020-11-20 14:29:52Z 64bit)
tomlynn commented 3 years ago

Where does that leave us for ARM?

On Sat, Dec 12, 2020 at 3:49 PM jungle-boogie notifications@github.com wrote:

No problem on amd64:

freeswitch> blacklist add blocked 12065238918 +OK

2020-12-12 15:46:41.602457 [INFO] mod_blacklist.c:228 Added [12065238918] to list [blocked] freeswitch> blacklist save blocked +OK

2020-12-12 15:46:44.042457 [INFO] mod_blacklist.c:280 Saving blocked to /usr/local/freeswitch/blacklists/blocked.list 2020-12-12 15:46:44.042457 [DEBUG] mod_blacklist.c:288 adding 12065238918 to the dump file

version:

FreeSWITCH Version 1.10.6-dev+git~20201120T142952Z~8045c96458~64bit (git 8045c96 2020-11-20 14:29:52Z 64bit)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/signalwire/freeswitch/issues/983#issuecomment-743921604, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFU3N524MABILYPGNCIBI3DSUP6O7ANCNFSM4UXAVRBQ .

jungle-boogie commented 3 years ago

Well I suppose we could bisect. Do you know when it last functioned?

tomlynn commented 3 years ago

No, I'm afraid I do not. I've only just begun to use the feature.

On Mon, Dec 14, 2020 at 6:34 PM jungle-boogie notifications@github.com wrote:

Well I suppose we could bisect. Do you know when it last functioned?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/signalwire/freeswitch/issues/983#issuecomment-745012695, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFU3N57IR33V5TJJXVRDVTLSU3DLNANCNFSM4UXAVRBQ .

Len-PGH commented 3 years ago

@tomlynn @jungle-boogie What model of raspberry pi are you testing on and what OS image is on the pi?

tomlynn commented 3 years ago

I'm using a model 4b with 2gb of ram. the os-release shows: pi@freeswitch:~ $ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

lsb_release -a shows:

pi@freeswitch:~ $ lsb_release -a No LSB modules are available. Distributor ID: Raspbian Description: Raspbian GNU/Linux 10 (buster) Release: 10 Codename: buster

On Fri, Dec 18, 2020 at 6:49 AM Len notifications@github.com wrote:

@tomlynn https://github.com/tomlynn @jungle-boogie https://github.com/jungle-boogie What model of raspberry pi are you testing on and what OS image is on the pi?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/signalwire/freeswitch/issues/983#issuecomment-748124012, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFU3N55AH36K5OKMTRQPBVTSVNTV5ANCNFSM4UXAVRBQ .

jungle-boogie commented 3 years ago

Hi,

I'm using a raspberry pi 4 with 4GB of RAM.

$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
$ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:        10
Codename:       buster
$ cc -v
Using built-in specs.
COLLECT_GCC=cc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/8/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 8.3.0-6+rpi1' --with-bugurl=fil
e:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --pr
efix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=arm-linux-gnueabihf
- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --en
able-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enab
le-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-uni
que-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin
--with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-
sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-che
cking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabih
f
Thread model: posix
gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)
$ uname -a
Linux freeswitch.in.example.net 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux
tomlynn commented 3 years ago

pi@freeswitch:~ $ cc -v Using built-in specs. COLLECT_GCC=cc COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/8/lto-wrapper Target: arm-linux-gnueabihf Configured with: ../src/configure -v --with-pkgversion='Raspbian 8.3.0-6+rpi1' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf Thread model: posix gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)

Remember, I've installed from packages, so didn't build this myself.

On Fri, Dec 18, 2020 at 10:00 AM jungle-boogie notifications@github.com wrote:

Hi,

I'm using a raspberry pi 4 with 4GB of RAM.

$ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

$ lsb_release -a No LSB modules are available. Distributor ID: Raspbian Description: Raspbian GNU/Linux 10 (buster) Release: 10 Codename: buster

$ cc -v Using built-in specs. COLLECT_GCC=cc COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/8/lto-wrapper Target: arm-linux-gnueabihf Configured with: ../src/configure -v --with-pkgversion='Raspbian 8.3.0-6+rpi1' --with-bugurl=fil e:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --pr efix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=arm-linux-gnueabihf

  • --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --en able-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enab le-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-uni que-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable- sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-che cking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabih f Thread model: posix gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)

$ uname -a Linux freeswitch.in.example.net 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/signalwire/freeswitch/issues/983#issuecomment-748233159, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFU3N5ZJN2SQDJ7Q47SAVZDSVOKC5ANCNFSM4UXAVRBQ .

tomlynn commented 3 years ago

pi@freeswitch:~ $ uname -a Linux freeswitch 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux

On Fri, Dec 18, 2020 at 12:40 PM Tom Lynn tom@tomlynn.com wrote:

pi@freeswitch:~ $ cc -v Using built-in specs. COLLECT_GCC=cc COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/8/lto-wrapper Target: arm-linux-gnueabihf Configured with: ../src/configure -v --with-pkgversion='Raspbian 8.3.0-6+rpi1' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf Thread model: posix gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)

Remember, I've installed from packages, so didn't build this myself.

On Fri, Dec 18, 2020 at 10:00 AM jungle-boogie notifications@github.com wrote:

Hi,

I'm using a raspberry pi 4 with 4GB of RAM.

$ cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

$ lsb_release -a No LSB modules are available. Distributor ID: Raspbian Description: Raspbian GNU/Linux 10 (buster) Release: 10 Codename: buster

$ cc -v Using built-in specs. COLLECT_GCC=cc COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/8/lto-wrapper Target: arm-linux-gnueabihf Configured with: ../src/configure -v --with-pkgversion='Raspbian 8.3.0-6+rpi1' --with-bugurl=fil e:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --pr efix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=arm-linux-gnueabihf

  • --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --en able-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enab le-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-uni que-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable- sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-che cking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabih f Thread model: posix gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)

$ uname -a Linux freeswitch.in.example.net 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/signalwire/freeswitch/issues/983#issuecomment-748233159, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFU3N5ZJN2SQDJ7Q47SAVZDSVOKC5ANCNFSM4UXAVRBQ .

mihing commented 3 years ago

Same issue on Intel Core i5-9500T, Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux, FreeSWITCH (Version 1.10.6 -release-18-1ff9d0a60e 64bit)

jungle-boogie commented 3 years ago

Interesting. It's still fine for me on amd64 with FS head.

> blacklist add blocked 12065238918
+OK

2021-05-27 12:57:39.524581 [INFO] mod_blacklist.c:228 Added [12065238918] to list [blocked]
> freeswitch@lv-freeswitch> blacklist save blocked
+OK

2021-05-27 12:57:47.144580 [INFO] mod_blacklist.c:280 Saving blocked to /usr/local/freeswitch/blacklists/blocked.list
2021-05-27 12:57:47.144580 [DEBUG] mod_blacklist.c:288 adding 170230002306 to the dump file
2021-05-27 12:57:47.144580 [DEBUG] mod_blacklist.c:288 adding 12065238918 to the dump file
> freeswitch@lv-freeswitch>
> version
FreeSWITCH Version 1.10.7-dev+git~20210517T135406Z~7f86d25c6c~64bit (git 7f86d25 2021-05-17 13:54:06Z 64bit)

and it's in the file:

$ cat /usr/local/freeswitch/blacklists/blocked.list
170230002306
12065238918
mihing commented 3 years ago 
freeswitch@allo> blacklist add blocked 74951343145
+OK

freeswitch@allo> blacklist save blocked
Socket interrupted, bye!
root@allo:~#
root@allo:~# cat /etc/freeswitch/blacklists/blocked.list
79035951015
79587638702
79912153801
74956600540
79119292049
74957390481
79646103873
79259296580
79675553574
74951505345
79691185540
...
mihing commented 3 years ago

This construction in dialplan:

        <action application="set" data="fake=${blacklist(del blocked 7${RF_NUM})}" />
        <action application="event" data="${blacklist(save blocked)}" />

doing something bad (sometimes):

0eb6c4fe-4e51-4775-bfbe-85ebe6bc7c70 EXECUTE [depth=0] sofia/internal/201@***:5060 record_session(/var/log/freeswitch/recordings/farm/201_20210525_123551_to_8*******219.mp3)
2021-05-25 12:35:51.626195 [INFO] mod_blacklist.c:250 Removed [7*******219] from list [blocked]
0eb6c4fe-4e51-4775-bfbe-85ebe6bc7c70 EXECUTE [depth=0] sofia/internal/201@***:5060 set(fake=+OK
0eb6c4fe-4e51-4775-bfbe-85ebe6bc7c70 )
2021-05-25 12:35:51.939683 [NOTICE] switch_loadable_module.c:412 Adding API Function 'cdr_csv'
2021-05-25 12:35:51.940044 [NOTICE] switch_loadable_module.c:350 Adding Application 'socket'
2021-05-25 12:35:51.940055 [NOTICE] switch_loadable_module.c:412 Adding API Function 'event_sink'
2021-05-25 12:35:51.941222 [INFO] mod_sofia.c:6260 Starting initial message thread.
2021-05-25 12:35:51.941399 [INFO] sofia.c:6103 Setting MAX Auth Validity to 0 Attempts
2021-05-25 12:35:51.941421 [NOTICE] sofia.c:6270 Started Profile external [sofia_reg_external]
2021-05-25 12:35:51.941653 [INFO] sofia.c:6103 Setting MAX Auth Validity to 0 Attempts
2021-05-25 12:35:51.941677 [NOTICE] sofia.c:6270 Started Profile internal [sofia_reg_internal]
2021-05-25 12:35:51.941728 [WARNING] sofia.c:2318 MSG Thread 0 Started
2021-05-25 12:35:51.944881 [INFO] switch_core_sqldb.c:1899 sofia:external Starting SQL thread.
2021-05-25 12:35:51.944934 [INFO] switch_core_sqldb.c:1899 sofia:internal Starting SQL thread.

The call goes into emptiness, there is silence in the headset.

jungle-boogie commented 3 years ago

I don't get the coredump or FS termination at all. Is it possible for you to build FS from trunk/master and see what happens?

mihing commented 3 years ago

Maybe later. This pbx now in prod. I have second pbx at core2duo e6600 with same problem. But there is not enough time for this yet.

mihing commented 3 years ago

Sometimes it crashes with this: freeswitch@allo> fs_cli: literal.c:136: literal_get: Assertionl->l_idx > (size _t)idx' failed. `

mihing commented 3 years ago

Works stably with this:

        <action application="set" data="api_result=${reload(mod_blacklist)}" />
        <action application="set" data="fake=${blacklist(del blocked 7${RF_NUM})}" />
        <action application="event" data="${blacklist(save blocked)}" />

If I reload the mod_blacklist first. No crashes.

mjerris commented 3 years ago

we need backtraces for the crashes

jungle-boogie commented 3 years ago

For me on arm64, I get a Socket interrupted, bye!.

Can a backtrace or core dump be generated from this?

mjerris commented 3 years ago

a backtrace is generated from a core dump. you need your system to generate a core dump on the crash

jungle-boogie commented 3 years ago

@mjerris, thanks that was the clue I needed.

here's from my freeswitch system. https://pastebin.freeswitch.org/view/8237e2be

FreeSWITCH Version 1.10.7-dev+git~20210517T135406Z~7f86d25c6c~32bit (git 7f86d25 2021-05-17 13:54:06Z 32bit)

Raspberry pi 4

jungle-boogie commented 3 years ago

For everyone else, review this article on how to create the backtrace file https://signalwire.force.com/help/s/article/FreeSWITCH-Crash-Getting-a-Backtrace-From-a-Core-Dump

mjerris commented 3 years ago

mod_blacklist.c:280 filename is a bad pointer:

   filename = 0xb50c86e3 <error: Cannot access memory at address 0xb50c86e3>
mjerris commented 3 years ago

function load_list is called with file name arg that is allocated from the xml, we should be duplicating this value because we are pushing that pointer into the globals.files hash table and then freeing the memory (when we free the xml structure)