Segmentation fault error [Ubuntu 20.04 LTS/20.10]

[ghost]

Segmentation fault error on trying to run ./dnsseed Segmentation fault (core dumped) on another machine

Ubuntu 20.04/64 both

[dryabov]

Could yry to found exact error line with the aid of gdb debugger (by loading the generated core dump): gdb ./dnsseed core What do you see?

[ghost]

gdb ./dnsseed core

Reading symbols from ./dnsseed...
(No debugging symbols found in ./dnsseed)
/home/xxx/twister-seeder/core: No such file or directory.
(gdb)

I have compiled dnsseed binary using bitcoin-seeder instructions (just by the make command)

Is the twisterd process required for seeder running? maybe core means some process id?

Not familiar with subject but trying to understand how can I support the network as have 24/7 machine. All that I found, following archived article with some informative comments:

http://web.archive.org/web/20150318175101/http://twister.net.co/?p=410

Thanks for any help

[dryabov]

Replace -g0 in the Makefile by just -g, rebuild and run again (until the segmentation fault error).

[ghost]

Changed CXXFLAGS to CXXFLAGS = -O3 -g -march=native, cloned & compiled again, but the same issue.

--

Should I to setup DNS records before? Maybe it's stupid, but currently trying to run seeder using demo example (just trying how it works before purchase the domain): ./dnsseed -h dnsseed.example.com -n vps.example.com

and get following response: Starting 4 DNS threads for dnsseed.example.com on vps.example.com (port 53)....Segmentation fault

--

Anyway, gdb ./dnsseed core still returning the same output:

Reading symbols from ./dnsseed...
(No debugging symbols found in ./dnsseed)
/home/xxx/twister-seeder/core: No such file or directory.
(gdb)

--

Compilation log

make
gcc -pthread -std=c99 -O3 -g -march=native dns.c -c -o dns.o
g++ -DUSE_IPV6 -pthread -O3 -g -march=native -Wno-invalid-offsetof -c -o bitcoin.o bitcoin.cpp
bitcoin.cpp: In member function ‘bool CNode::ProcessMessage(std::string, CDataStream&)’:
bitcoin.cpp:142:16: warning: too many arguments for format [-Wformat-extra-args]
  142 |         printf("%s: got address %s\n", ToString(you).c_str(), addr.ToString().c_str(), (int)(vAddr->size()));
      |                ^~~~~~~~~~~~~~~~~~~~~~
g++ -DUSE_IPV6 -pthread -O3 -g -march=native -Wno-invalid-offsetof -c -o netbase.o netbase.cpp
g++ -DUSE_IPV6 -pthread -O3 -g -march=native -Wno-invalid-offsetof -c -o protocol.o protocol.cpp
g++ -DUSE_IPV6 -pthread -O3 -g -march=native -Wno-invalid-offsetof -c -o db.o db.cpp
g++ -DUSE_IPV6 -pthread -O3 -g -march=native -Wno-invalid-offsetof -c -o main.o main.cpp
main.cpp:380:20: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
  380 |         fprintf(d, "%-47s  %4d  %11"PRId64"  %6.2f%% %6.2f%% %6.2f%% %6.2f%% %6.2f%%  %6i  %08"PRIx64"  %5i \"%s\"\n", rep.ip.ToString().c_str(), (int)rep.fGood, rep.lastSuccess, 100.0*rep.uptime[0], 100.0*rep.uptime[1], 100.0*rep.uptime[2], 100.0*rep.uptime[3], 100.0*rep.uptime[4], rep.blocks, rep.services, rep.clientVersion, rep.clientSubVersion.c_str());
      |                    ^
main.cpp:380:43: warning: invalid suffix on literal; C++11 requires a space between literal and string macro [-Wliteral-suffix]
  380 |         fprintf(d, "%-47s  %4d  %11"PRId64"  %6.2f%% %6.2f%% %6.2f%% %6.2f%% %6.2f%%  %6i  %08"PRIx64"  %5i \"%s\"\n", rep.ip.ToString().c_str(), (int)rep.fGood, rep.lastSuccess, 100.0*rep.uptime[0], 100.0*rep.uptime[1], 100.0*rep.uptime[2], 100.0*rep.uptime[3], 100.0*rep.uptime[4], rep.blocks, rep.services, rep.clientVersion, rep.clientSubVersion.c_str());
      |                                           ^
main.cpp: In function ‘void* ThreadDNS(void*)’:
main.cpp:344:1: warning: no return statement in function returning non-void [-Wreturn-type]
  344 | }
      | ^
g++ -DUSE_IPV6 -pthread -O3 -g -march=native -Wno-invalid-offsetof -c -o util.o util.cpp
g++ -pthread -O3 -g -march=native -o dnsseed dns.o bitcoin.o netbase.o protocol.o db.o main.o util.o -lcrypto

--

By the way, I have had compilation issues (libtorrent) on Ubuntu 18.04 using latest twister-core repository slice. But successfully compiled it on Ubuntu 20.04. Maybe twister-seeder use some deprecated dependencies as repository was updated in 2015

[dryabov]

Try to run dnsseed under gdb:

gdb ./dnsseed
(gdb) r
...
Segmentation fault
(gdb) bt
...
(gdb) thread apply all bt
...

The r command runs the file, bt prints backtrace, and thread apply all bt prints backraces of all threads (just in case).

PS. The only dependence is libssl.

[ghost]

yep, already tried - there is no troubles, it seems I need to setup DNS records first. thank you so much.

here is gdb output anyway (last commands included)

gdb ./dnsseed
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./dnsseed...
(gdb) r
Starting program: /home/xxx/twister-seeder/dnsseed 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Wanna dance? RTFM, babe!

Twister-seeder
Usage: /home/xxx/twister-seeder/dnsseed -h <host> -n <ns> [-m <mbox>] [-t <threads>] [-p <port>]

Options:
-h <host>       Hostname of the DNS seed
-n <ns>         Hostname of the nameserver
-m <mbox>       E-Mail address reported in SOA records
-t <threads>    Number of crawlers to run in parallel (default 96)
-d <threads>    Number of DNS server threads (default 4)
-p <port>       UDP port to listen on (default 53)
-o <ip:port>    Tor proxy IP/Port
--testnet       Use testnet
--wipeban       Wipe list of banned nodes
--wipeignore    Wipe list of ignored nodes
-?, --help      Show this text

[Inferior 1 (process 1189) exited normally]
(gdb) bt
No stack.
(gdb) thread apply all bt
(gdb)

[ghost]

@dryabov just one question: when you running following (demo) command as is: ./dnsseed -h dnsseed.example.com -n vps.example.com

then you don't get the same 'Segmentation fault error'? If not, it could be deprecated OS dependencies issue

[dryabov]

@D4708 I run it on a server with Ubuntu 18.04 (and the code was compiled years ago for Ubuntu 12.04 or 14.04), and I've never saw segmentation fault error there.

[ghost]

got it, thoughts that is the reason. thank you!

[ghost]

@dryabov that's it! tried on another (Debian machine) and it works.

uname -a Linux debian 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux

root@debian:/home/xxx/twister-seeder# ./dnsseed -h dnsseed.example.com -n vps.example.com
Starting 4 DNS threads for dnsseed.example.com on vps.example.com (port 53).......done
Starting seeder...done
Starting 96 crawler threads...done
[21-03-03 09:39:59] 0/0 available (0 tried in 1614782399s, 0 new, 0 active), 0 banned; 0 DNS requests, 4 db queries
[21-03-03 09:40:00] 0/1 available (0 tried in 1614782400s, 1 new, 0 active), 0 banned; 0 DNS requests, 4 db queries
[21-03-03 09:40:01] 0/1 available (0 tried in 1614782401s, 1 new, 0 active), 0 banned; 0 DNS requests, 4 db queries
[21-03-03 09:40:02] 0/1 available (0 tried in 1614782402s, 1 new, 0 active), 0 banned; 0 DNS requests, 4 db queries
^C

Have no idea how to downgrade my current OS as on the older distros I'll not be able to compile latest twisterd version... Is there any ideas how to upgrade twister-seeder to support latest operation systems?)

[ghost]

latest version of bitcoin-seeder works fine on Ubuntu 20.04, maybe some things like boost api should be upgraded... https://github.com/miguelfreitas/twister-core/commit/1efa0961aea2a3b3c898dbac37d0f3f972a7e224

[ghost]

Found the answer why gdb returns empty results.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Wanna dance? RTFM, babe!

The solution:

The problem was that gdb was not able to find and load both libpthread.so.0 and libthrea_db.so.1.

Therefore, create a .gdbinit in your $HOME directory and add the following lines in it

set auto-load safe-path / set libthread-db-search-path /lib/x86_64-linux-gnu (64-bit users) set env LD_PRELOAD /lib/x86_64-linux-gnu/libpthread.so.0 (64-bit users)

So get following the run output:

(gdb) run
Starting program: /home/xxx/twister-seeder/dnsseed 
ERROR: ld.so: object '(64-bit' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object 'users)' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '(64-bit' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object 'users)' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Wanna dance? RTFM, babe!

Still #RTFM so :)

[ghost]

Ah, forgot to provide host flags to seeder so was starting without fail (just the help output):

sudo gdb ./dnsseed
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./dnsseed...
(No debugging symbols found in ./dnsseed)

(gdb) run -h xxxx -n xxxx
Starting program: /home/xxxx/twister-seeder/dnsseed -h xxxx -n xxxx
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Starting 4 DNS threads for xxxx on xxxx (port 53)...[New Thread 0x7ffff7763700 (LWP 15258)]
.
Thread 2 "dnsseed" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7763700 (LWP 15258)]
0x000055555556fcf4 in StatCompare(CAddrReport const&, CAddrReport const&) ()

(gdb) bt
#0  0x000055555556fcf4 in StatCompare(CAddrReport const&, CAddrReport const&) ()
#1  0x00007ffff7763700 in ?? ()
#2  0x00007ffff796a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#3  0x00007ffff7891293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

(gdb) thread apply all bt
Thread 2 (Thread 0x7ffff7763700 (LWP 15258)):
#0  0x000055555556fcf4 in StatCompare(CAddrReport const&, CAddrReport const&) ()
#1  0x00007ffff7763700 in ?? ()
#2  0x00007ffff796a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#3  0x00007ffff7891293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7ffff7764740 (LWP 15254)):
#0  0x00007ffff784f3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7fffffffe3d0, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007ffff7855047 in __GI___nanosleep (requested_time=<optimized out>, remaining=<optimized out>) at nanosleep.c:27
#2  0x0000555555559dbc in main ()

The same results on both different machines running Ubuntu 20.04

[dryabov]

There are several things I'd try: 1) to remove dnsseed.dat before the run, maybe incorrect data were stored there during previous run. 2) to increase ulimit -s limit (but on my server the default 8Mb limit is sufficient; note the size is set in Kb). 3) print values of a.uptime and b.uptime at the beginning of StatCompare in main.cpp (most likely a and b are stored n registers, and gdb will not be able to print them using print command):

printf("&a=%p, &b=%p\n", (void*)&a, (void*)&b);
printf("a.uptime=%g %g, b.uptime=%g %g\n", a.uptime[3], a.uptime[4], b.uptime[3], b.uptime[4]);

Patch, rebuild and run again.

[ghost]

1. it seems that dnsseed.dat yet not created as the seeder crashes before creating it
2. not helps, I suppose that is the sys lib dependency issue... or the distributive library version bug (will try another distributive)
3. get following backtrace:

(gdb) run -h xxx -n xxx
Starting program: /home/xxx/twister-seeder/dnsseed -h xxx -n xxx
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Starting 4 DNS threads for xxx on xxx (port 53)...[New Thread 0x7ffff7763700 (LWP 17174)]
.
Thread 2 "dnsseed" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7763700 (LWP 17174)]
buffered_vfprintf (s=s@entry=0x7ffff795b6a0 <_IO_2_1_stdout_>, 
    format=format@entry=0x55555557da0c "&a=%p, &b=%p\n", args=args@entry=0x7ffff7762df8, 
    mode_flags=mode_flags@entry=2) at vfprintf-internal.c:2377
2377    vfprintf-internal.c: No such file or directory.
(gdb) thread apply all bt

Thread 2 (Thread 0x7ffff7763700 (LWP 17174)):
#0  buffered_vfprintf (s=s@entry=0x7ffff795b6a0 <_IO_2_1_stdout_>, format=format@entry=0x55555557da0c "&a=%p, &b=%p\n", args=args@entry=0x7ffff7762df8, mode_flags=mode_flags@entry=2) at vfprintf-internal.c:2377
#1  0x00007ffff77e8ea4 in __vfprintf_internal (s=0x7ffff795b6a0 <_IO_2_1_stdout_>, format=0x55555557da0c "&a=%p, &b=%p\n", ap=ap@entry=0x7ffff7762df8, mode_flags=2) at vfprintf-internal.c:1346
#2  0x00007ffff78a00eb in ___printf_chk (flag=<optimized out>, format=<optimized out>) at printf_chk.c:33
#3  0x000055555556fd99 in StatCompare(CAddrReport const&, CAddrReport const&) ()
#4  0x00007ffff7763700 in ?? ()
#5  0x00007ffff796a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007ffff7891293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7ffff7764740 (LWP 17170)):
#0  0x00007ffff784f3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7fffffffe3f0, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007ffff7855047 in __GI___nanosleep (requested_time=<optimized out>, remaining=<optimized out>) at nanosleep.c:27
#2  0x0000555555559e3c in main ()

[dryabov]

Try to apply this patch: https://github.com/dryabov/twister-seeder/commit/4d5f88e301c4ffb51938d3cb444e427f07d0fe1e (works on Ubuntu 20.10).

An alternative solution is to build with -O0 option instead of -O3 in the Makefile, but I wouldn't recommend it (from performance point of view).

[ghost]

Thank you so much!

Confirm, it works on 20.04. Also tested on 18.04 - so compatible with previous versions. Not sure about issue closing as the official repository should include this patch.

\o/

[dryabov]

PR: https://github.com/miguelfreitas/twister-seeder/pull/9

[ghost]

[21-03-15 12:05:28] 0/2 available (2 tried in 810s, 0 new, 0 active), 0 banned; 0 DNS requests, 4 db queries
[21-03-15 12:05:29] 0/2 available (2 tried in 811s, 0 new, 0 active), 0 banned; 0 DNS requests, 4 db queries
[21-03-15 12:05:30] 0/2 available (2 tried in 812s, 0 new, 0 active), 0 banned; 0 DNS requests, 4 db queries
[21-03-15 12:05:31] 0/2 available (2 tried in 813s, 0 new, 0 active), 0 banned; 0 DNS requests, 4 db queries
Segmentation fault

just running again with gdb

I suppose it would be local/RAM issue as server uses all of it resources.

[dryabov]

@D4708 I've run debug build (with all -fsanitize options enabled), will try to reproduce. Most likely 2 hour of running will be sufficient to catch that segmentation fault (if it was related to the code).

[ghost]

Just running on the another machine (20.04) with a free memory/cpu - all works. The system resources issue 99% not the code.

P.S. running under the gdb console works without errors on the overloaded machine (20.04 too), just crashes using native ./dnsseed call. That confirm my suppose imho. P.P.S. not sure how to use -fsanitize, as I understand, that is compilation flag? just searching for info.

Thank you for advice.

[dryabov]

P.P.S. not sure how to use -fsanitize, as I understand, that is compilation flag?

@D4708 I just append -fsanitize=address -fsanitize=undefined -fno-sanitize-recover to CXXFLAGS, it adds runtime checks for correctness of each variable access (and don't forget to replace -g0 by -g to get explicit backtrace in error messages). Of course, it slows down execution, but not as much as valgrind does.