Raspberry Pi Wallet Version 3.6.1.0 Crash

mattpotok commented 7 years ago

I set up a wallet a Raspberry Pi 3 running Raspbian more or less following the instructions at this link. However, I have noticed that the wallet has been crashing continuously in the past couple of weeks or so. Initially, I was running at version 3.6.0.2 when the crashes started occuring so I upgraded to 3.6.1.0 but the wallet keeps crashing.

I added debug lines to my gridcoinresearchd.conf file and the ouput can be found at this pastebin.

Additionally, I attached the 'gridcoinresearchd' process in gdb and after running it for some time I got the following error: Thread 13 "grc-msghand" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x703ff440 (LWP 18022)] 0x76963a3c in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6

denravonska commented 7 years ago

Interesting.

Does this Pi have any swap?
Can you do a 'bt' in gdb to see if we can get a backtrace?

mattpotok commented 7 years ago

The Pi has 1GB of swap. I changed this from the original (don't remember what it was) because I was having issues with compiling gridcoin. When I monitor gridcoin with top, Gridcoin uses up most of the RAM available on the Pi but the total memory used is ~50%.

I ran 'bt' and the output is below:

Thread 12 "grc-msghand" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x70bff440 (LWP 18811)]
0x76963a3c in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
(gdb) bt
#0  0x76963a3c in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#1  0x0004420c in ?? ()
#2  0x000b53c8 in ?? ()
#3  0x000f0b08 in ?? ()
#4  0x000f0bf8 in ?? ()
#5  0x76eccdc8 in ?? () from /usr/lib/arm-linux-gnueabihf/libboost_thread.so.1.62.0
#6  0x76a27fc4 in start_thread (arg=0x70bff440) at pthread_create.c:335
#7  0x767bfc68 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

One noteworthy thing I forgot to mention was that when I upgraded the wallet to 3.6.1.0, I was experiencing the same issue as the prior version so I wiped the .GridcoinResearch directory and started from 0 blocks two days ago. Today, I noticed that it began crashing continuously after it reached 1033144 blocks which seems to coincide with the time when it downloaded all the blocks.

tomasbrod commented 7 years ago

@mattpotok The trace is incomplete because the binary is Missing debug symbols. @denravonska How to properly enable -g when building?

mattpotok commented 7 years ago

@tomasbrod I can try to recompile Gridcoin with debug flag enabled. I was hesitant about doing it for now because compilation takes quite a bit on the Pi.

denravonska commented 7 years ago

Please do. Debug symbols are enabled by default, so if you self compile you should get them.

Scalextrix commented 7 years ago

I have found you can use the -j2 flag to speed up compiling a bit.

sudo make -f makefile.unix -j2 -e PIE=1

tomasbrod commented 7 years ago

1) never run make with sudo

except when make install, but even then you should use distro tools (checkinstall, etc) 2) none of the switches enable debugging, and no, when I compile myself, I do NOT get them, it is not default @denravonska. I always edit makefile and fight with qmake because I don't know how to do it properly 3) you can even use -j16 if you have enough ram and cpus

Scalextrix commented 7 years ago

My comment was to help the user re-compile quicker specifically on a Raspberry Pi 2 or 3, based on my own, -j2 is as fast as it gets.

denravonska commented 7 years ago

@tomasbrod Debug symbols are in the binary. At least the headless one:

$ make -f makefile.unix obj/net.o
g++ -c -O2  -std=c++11 -pthread -Wall -Wextra -Wno-ignored-qualifiers -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -I/home/marco/dev/Gridcoin-Research/src -I/home/marco/dev/Gridcoin-Research/src/obj  -DBOOST_NO_CXX11_SCOPED_ENUMS -DUSE_UPNP=0 -I/home/marco/dev/Gridcoin-Research/src/leveldb/include -I/home/marco/dev/Gridcoin-Research/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -MMD -MF obj/net.d -o obj/net.o net.cpp
                                                                                                                       ^^^

$ file gridcoinresearchd 
gridcoinresearchd: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=623a70bcc250ac230f0fa4de1635771717c1fdcf, not stripped

In order to get debug symbols in the Qt wallet, do:

$ qmake CONFIG+=force_deubg_info

mattpotok commented 7 years ago

Here's an update on what happened over the past few days:

After leaving my comment 6 days ago, I wasn't able to work on the Pi for a couple of days. When I came back to it, I found that my auto-upgrade script had upgraded gridcoinresearchd to version 3.6.2.0. I ran the new upgraded wallet and left it running. Till now, the wallet hasn't crashed so I will assume that whatever was causing it to crash was fixed in the most recent upgrade.

I am going close the issue for now since the wallet is functional. Thanks for all the help and the quick responses!

tomasbrod commented 7 years ago

@denravonska Did not work. Actually debug symbols are enabled with

qmake CONFIG+=deubg
make clean
make

iFoggz commented 7 years ago

LOL deubg

CONFIG += qt debug according to documentation.

nno consistency in documentation these days :) let me know which one is correct i'm using all 3 lol

gridcoin-community / Gridcoin-Research

Raspberry Pi Wallet Version 3.6.1.0 Crash #652