pvfs2-rm segfaults on SLES12. This is causing several of the sysint tests to fail.
Output of gdb is below. Unfortunately, I am unable to install the debuginfo libraries because we don't have access to the SLES repositories.
This problem only occurs on SLES12.
Jim
ec2-user@ofsnode-001:~> /opt/orangefs/bin/pvfs2-rm /tmp/mount/orangefs/miscfile
Segmentation fault (core dumped)
ec2-user@ofsnode-001:~> gdb /opt/orangefs/bin/pvfs2-rm core
GNU gdb (GDB; SUSE Linux Enterprise 12) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://bugs.opensuse.org/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
warning: /etc/gdbinit.d/gdb-heap.py: No such file or directory
Reading symbols from /opt/orangefs/bin/pvfs2-rm...done.
[New LWP 1016]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/opt/orangefs/bin/pvfs2-rm /tmp/mount/orangefs/miscfile'.
Program terminated with signal SIGSEGV, Segmentation fault.
0 0x00007f51b826331b in fread (ptr=0x7fff59cb1d80, size=, nmemb=, stream=0x3c) at src/client/usrint/stdio.c:1034
1034 if (!stream || !ISMAGICSET(stream, _P_IO_MAGIC))
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.19-17.72.x86_64 libopenssl1_0_0-debuginfo-1.0.1i-17.1.x86_64 libz1-debuginfo-1.2.8-5.1.x86_64
(gdb) bt
0 0x00007f51b826331b in fread (ptr=0x7fff59cb1d80, size=, nmemb=, stream=0x3c) at src/client/usrint/stdio.c:1034
1 0x00007f51b6eb7e2c in RAND_load_file () from /lib64/libcrypto.so.1.0.0
2 0x00007f51b6eb7828 in ssleay_rand_bytes () from /lib64/libcrypto.so.1.0.0
3 0x00007f51b6eb8193 in ?? () from /lib64/libcrypto.so.1.0.0
4 0x00007f51b6f26455 in ?? () from /lib64/libcrypto.so.1.0.0
5 0x00007f51b6f26b7f in FIPS_drbg_instantiate () from /lib64/libcrypto.so.1.0.0
6 0x00007f51b6eb8733 in RAND_init_fips () from /lib64/libcrypto.so.1.0.0
7 0x00007f51b6e320d0 in OPENSSL_init_library () from /lib64/libcrypto.so.1.0.0
8 0x00007f51b878792a in call_init.part () from /lib64/ld-linux-x86-64.so.2
9 0x00007f51b8787a13 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
10 0x00007f51b877a1ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
If LD_PRELOAD is set, running ANY program causes the same core dump.
ec2-user@ofsnode-001:~> export LD_PRELOAD=/opt/orangefs/lib/libofs.so:/opt/orangefs/lib/libpvfs2.so
ec2-user@ofsnode-001:~> gdb
Segmentation fault (core dumped)
ec2-user@ofsnode-001:~> export LD_PRELOAD=
ec2-user@ofsnode-001:~> gdb gdb core
GNU gdb (GDB; SUSE Linux Enterprise 12) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://bugs.opensuse.org/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
warning: /etc/gdbinit.d/gdb-heap.py: No such file or directory
Reading symbols from gdb...(no debugging symbols found)...done.
[New LWP 1237]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `gdb'.
Program terminated with signal SIGSEGV, Segmentation fault.
0 0x00007f1d4b01231b in fread (ptr=0x7fff52ab0e60, size=, nmemb=, stream=0x3c) at src/client/usrint/stdio.c:1034
1034 if (!stream || !ISMAGICSET(stream, _P_IO_MAGIC))
Missing separate debuginfos, use: zypper install gdb-debuginfo-7.7-7.32.x86_64
(gdb) bt
0 0x00007f1d4b01231b in fread (ptr=0x7fff52ab0e60, size=, nmemb=, stream=0x3c) at src/client/usrint/stdio.c:1034
1 0x00007f1d48a2ae2c in RAND_load_file () from /lib64/libcrypto.so.1.0.0
2 0x00007f1d48a2a828 in ssleay_rand_bytes () from /lib64/libcrypto.so.1.0.0
3 0x00007f1d48a2b193 in ?? () from /lib64/libcrypto.so.1.0.0
4 0x00007f1d48a99455 in ?? () from /lib64/libcrypto.so.1.0.0
5 0x00007f1d48a99b7f in FIPS_drbg_instantiate () from /lib64/libcrypto.so.1.0.0
6 0x00007f1d48a2b733 in RAND_init_fips () from /lib64/libcrypto.so.1.0.0
7 0x00007f1d489a50d0 in OPENSSL_init_library () from /lib64/libcrypto.so.1.0.0
8 0x00007f1d4b53692a in call_init.part () from /lib64/ld-linux-x86-64.so.2
9 0x00007f1d4b536a13 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
10 0x00007f1d4b5291ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
11 0x0000000000000001 in ?? ()
12 0x00007fff52ab29a9 in ?? ()
13 0x0000000000000000 in ?? ()
Ticket change
Time: 1466453740
Author: jburto2
Field: status
This was due to a mismatch between parameters between fread in glibc and fread in OrangeFS.
This bug was not seen on other distributions because other distributions initialized the (unset) value of stream to zero, which would cause the program to work. SLES12 just left garbage in it, which caused it to fail and the bug to be seen. This has been fixed.
Imported from Trac
ID: 56 Type: defect Time: 1432757020 Change time: 1466453740 Component: User Interface Severity: None Priority: major Owner: parl Reporter: jburto2 Cc: Version: latest Milestone: Status: closed Resolution: fixed Summary: pvfs2-rm segfaults on SLES 12 Keywords:
pvfs2-rm segfaults on SLES12. This is causing several of the sysint tests to fail.
Output of gdb is below. Unfortunately, I am unable to install the debuginfo libraries because we don't have access to the SLES repositories.
This problem only occurs on SLES12.
Jim
ec2-user@ofsnode-001:~> /opt/orangefs/bin/pvfs2-rm /tmp/mount/orangefs/miscfile Segmentation fault (core dumped) ec2-user@ofsnode-001:~> gdb /opt/orangefs/bin/pvfs2-rm core GNU gdb (GDB; SUSE Linux Enterprise 12) 7.7 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://bugs.opensuse.org/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"...
warning: /etc/gdbinit.d/gdb-heap.py: No such file or directory Reading symbols from /opt/orangefs/bin/pvfs2-rm...done. [New LWP 1016] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/opt/orangefs/bin/pvfs2-rm /tmp/mount/orangefs/miscfile'. Program terminated with signal SIGSEGV, Segmentation fault.
0 0x00007f51b826331b in fread (ptr=0x7fff59cb1d80, size=, nmemb=, stream=0x3c) at src/client/usrint/stdio.c:1034
1034 if (!stream || !ISMAGICSET(stream, _P_IO_MAGIC)) Missing separate debuginfos, use: zypper install glibc-debuginfo-2.19-17.72.x86_64 libopenssl1_0_0-debuginfo-1.0.1i-17.1.x86_64 libz1-debuginfo-1.2.8-5.1.x86_64 (gdb) bt
0 0x00007f51b826331b in fread (ptr=0x7fff59cb1d80, size=, nmemb=, stream=0x3c) at src/client/usrint/stdio.c:1034
1 0x00007f51b6eb7e2c in RAND_load_file () from /lib64/libcrypto.so.1.0.0
2 0x00007f51b6eb7828 in ssleay_rand_bytes () from /lib64/libcrypto.so.1.0.0
3 0x00007f51b6eb8193 in ?? () from /lib64/libcrypto.so.1.0.0
4 0x00007f51b6f26455 in ?? () from /lib64/libcrypto.so.1.0.0
5 0x00007f51b6f26b7f in FIPS_drbg_instantiate () from /lib64/libcrypto.so.1.0.0
6 0x00007f51b6eb8733 in RAND_init_fips () from /lib64/libcrypto.so.1.0.0
7 0x00007f51b6e320d0 in OPENSSL_init_library () from /lib64/libcrypto.so.1.0.0
8 0x00007f51b878792a in call_init.part () from /lib64/ld-linux-x86-64.so.2
9 0x00007f51b8787a13 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
10 0x00007f51b877a1ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
11 0x0000000000000002 in ?? ()
12 0x00007fff59cb296a in ?? ()
13 0x00007fff59cb2985 in ?? ()
14 0x0000000000000000 in ?? ()
(gdb) list 1029 if (stream == stdin || stream == stdout || stream == stderr) 1030 { 1031 return stdio_ops.fread(ptr, size, nmemb, stream); 1032 } 1033 #endif 1034 if (!stream || !ISMAGICSET(stream, _P_IO_MAGIC)) 1035 { 1036 if (stream && ISMAGICSET(stream, _IO_MAGIC)) 1037 { 1038 return stdio_ops.fread(ptr, size, nmemb, stream);
Ticket change Time: 1432757730 Author: jburto2 Field: component
User Interface
Ticket change Time: 1432757730 Author: jburto2 Field: comment
If LD_PRELOAD is set, running ANY program causes the same core dump.
ec2-user@ofsnode-001:~> export LD_PRELOAD=/opt/orangefs/lib/libofs.so:/opt/orangefs/lib/libpvfs2.so ec2-user@ofsnode-001:~> gdb Segmentation fault (core dumped) ec2-user@ofsnode-001:~> export LD_PRELOAD= ec2-user@ofsnode-001:~> gdb gdb core GNU gdb (GDB; SUSE Linux Enterprise 12) 7.7 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://bugs.opensuse.org/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"...
warning: /etc/gdbinit.d/gdb-heap.py: No such file or directory Reading symbols from gdb...(no debugging symbols found)...done. [New LWP 1237] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `gdb'. Program terminated with signal SIGSEGV, Segmentation fault.
0 0x00007f1d4b01231b in fread (ptr=0x7fff52ab0e60, size=, nmemb=, stream=0x3c) at src/client/usrint/stdio.c:1034
1034 if (!stream || !ISMAGICSET(stream, _P_IO_MAGIC)) Missing separate debuginfos, use: zypper install gdb-debuginfo-7.7-7.32.x86_64 (gdb) bt
0 0x00007f1d4b01231b in fread (ptr=0x7fff52ab0e60, size=, nmemb=, stream=0x3c) at src/client/usrint/stdio.c:1034
1 0x00007f1d48a2ae2c in RAND_load_file () from /lib64/libcrypto.so.1.0.0
2 0x00007f1d48a2a828 in ssleay_rand_bytes () from /lib64/libcrypto.so.1.0.0
3 0x00007f1d48a2b193 in ?? () from /lib64/libcrypto.so.1.0.0
4 0x00007f1d48a99455 in ?? () from /lib64/libcrypto.so.1.0.0
5 0x00007f1d48a99b7f in FIPS_drbg_instantiate () from /lib64/libcrypto.so.1.0.0
6 0x00007f1d48a2b733 in RAND_init_fips () from /lib64/libcrypto.so.1.0.0
7 0x00007f1d489a50d0 in OPENSSL_init_library () from /lib64/libcrypto.so.1.0.0
8 0x00007f1d4b53692a in call_init.part () from /lib64/ld-linux-x86-64.so.2
9 0x00007f1d4b536a13 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
10 0x00007f1d4b5291ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
11 0x0000000000000001 in ?? ()
12 0x00007fff52ab29a9 in ?? ()
13 0x0000000000000000 in ?? ()
Ticket change Time: 1466453740 Author: jburto2 Field: status
closed
Ticket change Time: 1466453740 Author: jburto2 Field: resolution
fixed
Ticket change Time: 1466453740 Author: jburto2 Field: comment
This was due to a mismatch between parameters between fread in glibc and fread in OrangeFS.
This bug was not seen on other distributions because other distributions initialized the (unset) value of stream to zero, which would cause the program to work. SLES12 just left garbage in it, which caused it to fail and the bug to be seen. This has been fixed.