Closed ateleshev closed 4 years ago
@ateleshev could you please provide a better trace may be using '# valgrind --track-origins=yes /usr/local/sbin/gluster-blockd' ?
@pkalever yes, of course
[root@7a33d06e1b1a] ~ # valgrind --track-origins=yes /usr/sbin/gluster-blockd
==165== Memcheck, a memory error detector
==165== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==165== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==165== Command: /usr/sbin/gluster-blockd
==165==
Error opening log file: No such file or directory
Logging to stderr.
[2019-10-30 14:46:25.194853] CRIT: trying to change logDir from /usr/var/log/gluster-block to /usr/var/log/gluster-block [at utils.c+495 :<initLogDirAndFiles>]
==165== Invalid read of size 4
==165== at 0x4A186C0: pthread_mutex_lock (pthread_mutex_lock.c:67)
==165== by 0x110EB0: glusterBlockCleanGlobals (gluster-blockd.c:65)
==165== by 0x10C961: main (gluster-blockd.c:695)
==165== Address 0x546c028 is not stack'd, malloc'd or (recently) free'd
==165==
==165== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==165== Access not within mapped region at address 0x546C028
==165== at 0x4A186C0: pthread_mutex_lock (pthread_mutex_lock.c:67)
==165== by 0x110EB0: glusterBlockCleanGlobals (gluster-blockd.c:65)
==165== by 0x10C961: main (gluster-blockd.c:695)
==165== If you believe this happened as a result of a stack
==165== overflow in your program's main thread (unlikely but
==165== possible), you can try to increase the size of the
==165== main thread stack using the --main-stacksize= flag.
==165== The main thread stack size used in this run was 8388608.
==165==
==165== HEAP SUMMARY:
==165== in use at exit: 457 bytes in 7 blocks
==165== total heap usage: 34 allocs, 27 frees, 64,187 bytes allocated
==165==
==165== LEAK SUMMARY:
==165== definitely lost: 0 bytes in 0 blocks
==165== indirectly lost: 0 bytes in 0 blocks
==165== possibly lost: 320 bytes in 1 blocks
==165== still reachable: 137 bytes in 6 blocks
==165== suppressed: 0 bytes in 0 blocks
==165== Rerun with --leak-check=full to see details of leaked memory
==165==
==165== For counts of detected and suppressed errors, rerun with: -v
==165== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault
And, the same in verbose mode:
[root@7a33d06e1b1a] ~ # valgrind -v --track-origins=yes /usr/sbin/gluster-blockd
==215== Memcheck, a memory error detector
==215== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==215== Using Valgrind-3.14.0-353a3587bb-20181007X and LibVEX; rerun with -h for copyright info
==215== Command: /usr/sbin/gluster-blockd
==215==
--215-- Valgrind options:
--215-- -v
--215-- --track-origins=yes
--215-- Contents of /proc/version:
--215-- Linux version 4.19.0-6-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20)
--215--
--215-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-lzcnt-rdtscp-sse3-avx-avx2-bmi
--215-- Page sizes: currently 4096, max supported 4096
--215-- Valgrind library directory: /usr/lib/x86_64-linux-gnu/valgrind
--215-- Reading syms from /usr/sbin/gluster-blockd
--215-- Reading syms from /lib/x86_64-linux-gnu/ld-2.28.so
--215-- Considering /usr/lib/debug/.build-id/f2/5dfd7b95be4ba386fd71080accae8c0732b711.debug ..
--215-- .. build-id is valid
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/valgrind/memcheck-amd64-linux
--215-- Considering /usr/lib/debug/.build-id/32/2e77af97f403c3d34ff09edf60b089e72ec889.debug ..
--215-- .. build-id is valid
--215-- object doesn't have a dynamic symbol table
--215-- Scheduler: using generic scheduler lock implementation.
--215-- Reading suppressions file: /usr/lib/x86_64-linux-gnu/valgrind/default.supp
==215== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-215-by-???-on-7a33d06e1b1a
==215== embedded gdbserver: writing to /tmp/vgdb-pipe-to-vgdb-from-215-by-???-on-7a33d06e1b1a
==215== embedded gdbserver: shared mem /tmp/vgdb-pipe-shared-mem-vgdb-215-by-???-on-7a33d06e1b1a
==215==
==215== TO CONTROL THIS PROCESS USING vgdb (which you probably
==215== don't want to do, unless you know exactly what you're doing,
==215== or are doing some strange experiment):
==215== /usr/lib/x86_64-linux-gnu/valgrind/../../bin/vgdb --pid=215 ...command...
==215==
==215== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==215== /path/to/gdb /usr/sbin/gluster-blockd
==215== and then give GDB the following command
==215== target remote | /usr/lib/x86_64-linux-gnu/valgrind/../../bin/vgdb --pid=215
==215== --pid is optional if only one valgrind process is running
==215==
--215-- REDIR: 0x401e2f0 (ld-linux-x86-64.so.2:strlen) redirected to 0x580c9ce2 (vgPlain_amd64_linux_REDIR_FOR_strlen)
--215-- REDIR: 0x401e0d0 (ld-linux-x86-64.so.2:index) redirected to 0x580c9cfc (vgPlain_amd64_linux_REDIR_FOR_index)
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_core-amd64-linux.so
--215-- Considering /usr/lib/debug/.build-id/34/fbc4cdfbfce1bfc3d2e522a01a050bf724c930.debug ..
--215-- .. build-id is valid
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so
--215-- Considering /usr/lib/debug/.build-id/ca/bb7898ea982d936cbfadd70ae831c6eb63623f.debug ..
--215-- .. build-id is valid
==215== WARNING: new redirection conflicts with existing -- ignoring it
--215-- old: 0x0401e2f0 (strlen ) R-> (0000.0) 0x580c9ce2 vgPlain_amd64_linux_REDIR_FOR_strlen
--215-- new: 0x0401e2f0 (strlen ) R-> (2007.0) 0x04838d80 strlen
--215-- REDIR: 0x401ab10 (ld-linux-x86-64.so.2:strcmp) redirected to 0x4839e40 (strcmp)
--215-- REDIR: 0x401e830 (ld-linux-x86-64.so.2:mempcpy) redirected to 0x483d860 (mempcpy)
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libjson-c.so.3.0.1
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libgfapi.so.0.0.0
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0.0.1
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libgfrpc.so.0.0.1
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libgfxdr.so.0.0.1
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libacl.so.1.1.2253
--215-- object doesn't have a symbol table
--215-- Reading syms from /lib/x86_64-linux-gnu/libuuid.so.1.3.0
--215-- object doesn't have a symbol table
--215-- Reading syms from /lib/x86_64-linux-gnu/libpthread-2.28.so
--215-- Considering /usr/lib/debug/.build-id/e9/1114987a0147bd050addbd591eb8994b29f4b3.debug ..
--215-- .. build-id is valid
--215-- Reading syms from /lib/x86_64-linux-gnu/libc-2.28.so
--215-- Considering /usr/lib/debug/.build-id/18/b9a9a8c523e5cfe5b5d946d605d09242f09798.debug ..
--215-- .. build-id is valid
--215-- Reading syms from /lib/x86_64-linux-gnu/libtirpc.so.3.0.0
--215-- object doesn't have a symbol table
--215-- Reading syms from /lib/x86_64-linux-gnu/libz.so.1.2.11
--215-- object doesn't have a symbol table
--215-- Reading syms from /lib/x86_64-linux-gnu/libm-2.28.so
--215-- Considering /usr/lib/debug/.build-id/88/5dda4b4a5cea600e7b5b98c1ad86996c8d2299.debug ..
--215-- .. build-id is valid
--215-- Reading syms from /lib/x86_64-linux-gnu/librt-2.28.so
--215-- Considering /usr/lib/debug/.build-id/5d/cf98ad684962be494af28a1051793fd39e4ebc.debug ..
--215-- .. build-id is valid
--215-- Reading syms from /lib/x86_64-linux-gnu/libdl-2.28.so
--215-- Considering /usr/lib/debug/.build-id/d3/583c742dd47aaa860c5ae0c0c5bdbcd2d54f61.debug ..
--215-- .. build-id is valid
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libattr.so.1.1.2448
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
--215-- object doesn't have a symbol table
--215-- Reading syms from /lib/x86_64-linux-gnu/libcom_err.so.2.1
--215-- object doesn't have a symbol table
--215-- Reading syms from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
--215-- object doesn't have a symbol table
--215-- Reading syms from /lib/x86_64-linux-gnu/libkeyutils.so.1.8
--215-- object doesn't have a symbol table
--215-- Reading syms from /lib/x86_64-linux-gnu/libresolv-2.28.so
--215-- Considering /usr/lib/debug/.build-id/02/6c3ba167f64f631eb8781fca2269fbc2ee7ca5.debug ..
--215-- .. build-id is valid
--215-- REDIR: 0x4ab7f70 (libc.so.6:memmove) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab7160 (libc.so.6:strncpy) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab8250 (libc.so.6:strcasecmp) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab6b70 (libc.so.6:strcat) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab7190 (libc.so.6:rindex) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab9890 (libc.so.6:rawmemchr) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ad2230 (libc.so.6:wmemchr) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ad1cd0 (libc.so.6:wcscmp) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab80e0 (libc.so.6:mempcpy) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab7f10 (libc.so.6:bcmp) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab7100 (libc.so.6:strncmp) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab6be0 (libc.so.6:strcmp) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab8040 (libc.so.6:memset) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ad1ca0 (libc.so.6:wcschr) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab70a0 (libc.so.6:strnlen) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab6c70 (libc.so.6:strcspn) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab82a0 (libc.so.6:strncasecmp) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab6c40 (libc.so.6:strcpy) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab83e0 (libc.so.6:memcpy@@GLIBC_2.14) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ad34d0 (libc.so.6:wcsnlen) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab71c0 (libc.so.6:strpbrk) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab6ba0 (libc.so.6:index) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab7070 (libc.so.6:strlen) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4abe280 (libc.so.6:memrchr) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab82f0 (libc.so.6:strcasecmp_l) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab7ee0 (libc.so.6:memchr) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ad1db0 (libc.so.6:wcslen) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab7460 (libc.so.6:strspn) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab8220 (libc.so.6:stpncpy) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab81f0 (libc.so.6:stpcpy) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab98c0 (libc.so.6:strchrnul) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab8340 (libc.so.6:strncasecmp_l) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4b36950 (libc.so.6:__memcpy_chk) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4ab7e20 (libc.so.6:strstr) redirected to 0x482b1b0 (_vgnU_ifunc_wrapper)
--215-- REDIR: 0x4b8afa0 (libc.so.6:__strrchr_avx2) redirected to 0x4838790 (rindex)
--215-- REDIR: 0x4b8b170 (libc.so.6:__strlen_avx2) redirected to 0x4838c60 (strlen)
--215-- REDIR: 0x4ab4120 (libc.so.6:calloc) redirected to 0x4837ab0 (calloc)
--215-- REDIR: 0x4ab3350 (libc.so.6:malloc) redirected to 0x4835710 (malloc)
--215-- REDIR: 0x4b8b6b0 (libc.so.6:__memcpy_avx_unaligned_erms) redirected to 0x483c8a0 (memmove)
--215-- REDIR: 0x4ab7980 (libc.so.6:__GI_strstr) redirected to 0x483dac0 (__strstr_sse2)
--215-- REDIR: 0x4b87000 (libc.so.6:__memchr_avx2) redirected to 0x4839ee0 (memchr)
--215-- REDIR: 0x4ab39a0 (libc.so.6:free) redirected to 0x4836940 (free)
--215-- REDIR: 0x4b8ab80 (libc.so.6:__strchr_avx2) redirected to 0x4838910 (index)
--215-- REDIR: 0x4b866b0 (libc.so.6:__strcmp_avx2) redirected to 0x4839d00 (strcmp)
--215-- REDIR: 0x4acd710 (libc.so.6:__stpcpy_sse2_unaligned) redirected to 0x483c3f0 (__stpcpy_sse2_unaligned)
--215-- REDIR: 0x4b8b690 (libc.so.6:__mempcpy_avx_unaligned_erms) redirected to 0x483d4a0 (mempcpy)
--215-- REDIR: 0x4b8adb0 (libc.so.6:__strchrnul_avx2) redirected to 0x483d390 (strchrnul)
Error opening log file: No such file or directory
Logging to stderr.
--215-- REDIR: 0x4b87780 (libc.so.6:__memcmp_avx2_movbe) redirected to 0x483c060 (bcmp)
--215-- REDIR: 0x4b8bb30 (libc.so.6:__memset_avx2_unaligned_erms) redirected to 0x483c790 (memset)
[2019-10-30 14:49:06.664691] CRIT: trying to change logDir from /usr/var/log/gluster-block to /usr/var/log/gluster-block [at utils.c+495 :<initLogDirAndFiles>]
==215== Invalid read of size 4
==215== at 0x4A186C0: pthread_mutex_lock (pthread_mutex_lock.c:67)
==215== by 0x110EB0: glusterBlockCleanGlobals (gluster-blockd.c:65)
==215== by 0x10C961: main (gluster-blockd.c:695)
==215== Address 0x546c028 is not stack'd, malloc'd or (recently) free'd
==215==
==215== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==215== Access not within mapped region at address 0x546C028
==215== at 0x4A186C0: pthread_mutex_lock (pthread_mutex_lock.c:67)
==215== by 0x110EB0: glusterBlockCleanGlobals (gluster-blockd.c:65)
==215== by 0x10C961: main (gluster-blockd.c:695)
==215== If you believe this happened as a result of a stack
==215== overflow in your program's main thread (unlikely but
==215== possible), you can try to increase the size of the
==215== main thread stack using the --main-stacksize= flag.
==215== The main thread stack size used in this run was 8388608.
==215==
==215== HEAP SUMMARY:
==215== in use at exit: 457 bytes in 7 blocks
==215== total heap usage: 34 allocs, 27 frees, 64,187 bytes allocated
==215==
==215== Searching for pointers to 7 not-freed blocks
==215== Checked 8,818,208 bytes
==215==
==215== LEAK SUMMARY:
==215== definitely lost: 0 bytes in 0 blocks
==215== indirectly lost: 0 bytes in 0 blocks
==215== possibly lost: 320 bytes in 1 blocks
==215== still reachable: 137 bytes in 6 blocks
==215== suppressed: 0 bytes in 0 blocks
==215== Rerun with --leak-check=full to see details of leaked memory
==215==
==215== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
==215==
==215== 1 errors in context 1 of 1:
==215== Invalid read of size 4
==215== at 0x4A186C0: pthread_mutex_lock (pthread_mutex_lock.c:67)
==215== by 0x110EB0: glusterBlockCleanGlobals (gluster-blockd.c:65)
==215== by 0x10C961: main (gluster-blockd.c:695)
==215== Address 0x546c028 is not stack'd, malloc'd or (recently) free'd
==215==
==215== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault
@pkalever and here is log data
16:27 [root@7a33d06e1b1a] ~ # cat /usr/var/log/gluster-block/gluster-blockd.log
[2019-10-30 14:45:52.059999] CRIT: logDir now is /usr/var/log/gluster-block [at utils.c+527 :<initLogDirAndFiles>]
[2019-10-30 14:45:52.060028] INFO: Block Hosting Volfile Server Set to: localhost [at utils.c+348 :<fetchGlfsVolServerFromEnv>]
[2019-10-30 14:45:52.060114] ERROR: creat(/usr/var/run/gluster-blockd.lock) failed[No such file or directory] [at gluster-blockd.c+616 :<main>]
[2019-10-30 14:46:25.233937] CRIT: logDir now is /usr/var/log/gluster-block [at utils.c+527 :<initLogDirAndFiles>]
[2019-10-30 14:46:25.239920] INFO: Block Hosting Volfile Server Set to: localhost [at utils.c+348 :<fetchGlfsVolServerFromEnv>]
[2019-10-30 14:46:25.320141] ERROR: creat(/usr/var/run/gluster-blockd.lock) failed[No such file or directory] [at gluster-blockd.c+616 :<main>]
[2019-10-30 14:49:06.706128] CRIT: logDir now is /usr/var/log/gluster-block [at utils.c+527 :<initLogDirAndFiles>]
[2019-10-30 14:49:06.711851] INFO: Block Hosting Volfile Server Set to: localhost [at utils.c+348 :<fetchGlfsVolServerFromEnv>]
[2019-10-30 14:49:06.790534] ERROR: creat(/usr/var/run/gluster-blockd.lock) failed[No such file or directory] [at gluster-blockd.c+616 :<main>]
@pkalever If I create the folder before run gluster-blockd, the daemon works fine.
mkdir -p /usr/var/run
Should the make script create this folder?
@ateleshev cool. yes the make install should have created it based on the prefix, feel free to send a PR.
Atleast in RHEL, Centos and Fedora,
utils/utils.h:# define GB_INFODIR DATADIR "/run"
DATADIR will be '/var' thus GB_INFODIR would expand to /var/run/, that is why may be we are lucky and didn't hit this before.
Cheers!
Closing this now, feel free to open if required. Thanks!
When I run gluster-blockd in debian container, I receive - Segmentation fault
Kind of issue
Bug
Observed behavior
Details on how to reproduce (minimal and precise)
Logs and Information about the environment:
Other useful information
Dockerfile you can find here: https://github.com/dboxx/containers/blob/master/gluster-block/Dockerfile
Logs: