Open renzhengeek opened 9 years ago
n1:/opt/ocfs2-test/log # journalctl -b 0 | grep error
Sep 01 17:59:35 zl-linux67 systemd-vconsole-setup[130]: /usr/bin/setfont failed with error code 1.
Sep 01 17:59:50 n1 dbus[809]: [system] Rejected send message, 1 matched rules; type="method_call", sender=":1.14" (uid=0 pid=888 comm="/usr/sbin/gdm ") interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply="0" destination=":1.15" (uid=0 pid=906 comm="/usr/lib/gdm/gdm-simple-slave --display-id /org/gn")
Sep 01 18:00:44 n1 external/libvirt(stonith-libvirt)[1904]: ERROR: error: failed to get domain 'n2'
error: Domain not found: no domain with matching name 'n2'
Sep 01 18:00:47 n1 iscsid[1523]: iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Sep 01 18:00:47 n1 kernel: connection1:0: detected conn error (1020)
Sep 01 18:00:48 n1 external/libvirt(stonith-libvirt)[1956]: ERROR: error: failed to get domain 'n2'
error: Domain not found: no domain with matching name 'n2'
Sep 01 18:00:49 n1 stonith-ng[1625]: error: Operation 'reboot' [1905] (call 2 from crmd.1629) for host 'n2' with device 'stonith-libvirt' returned: -201 (Generic Pacemaker error)
Sep 01 18:00:49 n1 stonith-ng[1625]: error: Operation reboot of n2 by <no-one> for crmd.1629@n1.db1458a3: No route to host
Sep 01 18:01:50 n1 dlm_controld[2020]: 137 cpg_dispatch error 9
Sep 01 18:05:16 n1 dlm_controld[2020]: 343 cpg_dispatch error 9
Sep 01 18:06:43 n1 libvirtd[1470]: nl_recv returned with error: No buffer space available
Sep 02 06:16:44 n1 dlm_controld[2020]: 44231 cpg_dispatch error 9
Sep 02 06:16:46 n1 dlm_controld[2020]: 44233 cpg_dispatch error 9
Sep 02 16:59:06 n1 dlm_controld[2020]: 82772 cpg_dispatch error 9
Sep 02 16:59:07 n1 dlm_controld[2020]: 82774 cpg_dispatch error 9
n1:/opt/ocfs2-test/log # cat multiple-x86_64-2015-09-01-18-05-14.log
Tue Sep 1 18:05:14 CST 2015 - Starting Multiple Nodes Regress test
Tue Sep 1 18:05:14 CST 2015
xattr-test
=====================Multiple nodes xattr testing starts: Tue Sep 1 18:05:15 CST 2015=====================
++++++++++xattr tests with "-b 512 -C 4096"++++++++++
=======================================================================================
Mkfsing device(-b 512 -C 4096): [ PASS ]
Mounting device /dev/sda to nodes(n1,n2): [ PASS ]
[1] Check Namespace&Filetype of Multinode Xattr on Ocfs2: [ PASS ]
[2] Check Utility of Multinode Xattr on Ocfs2: [ PASS ]
[3] Check Max Multinode Xattr EA_Name_Length: [ PASS ]
[4] Check Max Multinode Xattr EA_Size: [ PASS ]
[5] Check Huge Multinode Xattr EA_Entry_Nums: [ PASS ]
[6] Check All Max Multinode Xattr Arguments Together: [ PASS ]
[7] Launch Concurrent Adding Test: [ PASS ]
[8] Launch MultiNode Xattr Stress Test: [ PASS ]
Umounting device /dev/sda from nodes(n1,n2): [ PASS ]
=======================================================================================
++++++++++xattr tests with "-b 512 -C 32768"++++++++++
=======================================================================================
Mkfsing device(-b 512 -C 32768): [ PASS ]
Mounting device /dev/sda to nodes(n1,n2): [ PASS ]
[9] Check Namespace&Filetype of Multinode Xattr on Ocfs2: [ PASS ]
[10] Check Utility of Multinode Xattr on Ocfs2: [ PASS ]
[11] Check Max Multinode Xattr EA_Name_Length: [ PASS ]
[12] Check Max Multinode Xattr EA_Size: [ PASS ]
[13] Check Huge Multinode Xattr EA_Entry_Nums: [ PASS ]
[14] Check All Max Multinode Xattr Arguments Together: [ PASS ]
[15] Launch Concurrent Adding Test: [ PASS ]
[16] Launch MultiNode Xattr Stress Test: [ PASS ]
Umounting device /dev/sda from nodes(n1,n2): [ PASS ]
=======================================================================================
++++++++++xattr tests with "-b 512 -C 1048576"++++++++++
=======================================================================================
Mkfsing device(-b 512 -C 1048576): [ PASS ]
Mounting device /dev/sda to nodes(n1,n2): [ PASS ]
[17] Check Namespace&Filetype of Multinode Xattr on Ocfs2: [ PASS ]
[18] Check Utility of Multinode Xattr on Ocfs2: [ PASS ]
[19] Check Max Multinode Xattr EA_Name_Length: [ PASS ]
[20] Check Max Multinode Xattr EA_Size: [ PASS ]
[21] Check Huge Multinode Xattr EA_Entry_Nums:n1:/opt/ocfs2-test/log #
Mounting device /dev/sda to nodes(n1,n2):+ /usr/lib64/mpi/gcc/openmpi/bin/mpirun --allow-run-as-root -mca btl tcp,self -mca plm_rsh_agent ssh:rsh --allow-run-as-root -np 2 --host n1,n2 udevadm trigger
+ /usr/lib64/mpi/gcc/openmpi/bin/mpirun --allow-run-as-root -mca btl tcp,self -mca plm_rsh_agent ssh:rsh --allow-run-as-root -np 2 --host n1,n2 /opt/ocfs2-test/bin/command.py --mount -l ocfs2-xattr-multi-test-x86_64 -m /mnt/shared
[ PASS ]
[17] Check Namespace&Filetype of Multinode Xattr on Ocfs2: [ PASS ]
[18] Check Utility of Multinode Xattr on Ocfs2: [ PASS ]
[19] Check Max Multinode Xattr EA_Name_Length: [ PASS ]
[20] Check Max Multinode Xattr EA_Size: [ PASS ]
[21] Check Huge Multinode Xattr EA_Entry_Nums:n1:/opt/ocfs2-test/log/multi-xattr-test #
==========================================================
[21] Check Huge Multinode Xattr EA_Entry_Nums:
==========================================================
Testing Binary: /usr/lib64/mpi/gcc/openmpi/bin/mpirun -mca plm_rsh_agent ssh:rsh --allow-run-as-root -mca btl tcp,self --allow-run-as-root -np 4 --host n1,n2 /opt/ocfs2-test/bin/xattr-multi-test -i 1 -x 10000 -n user -t normal -l 100 -s 200 /mnt/shared/multi_xattr_test_place
Warning:list size exceed,due to (xattr_name_sz + 1) * xattr_nums was greater than 65536,Warning:list size exceed,due to (xattr_name_sz + 1) * xattr_nums was greater than 65536,will not launch list test!
******************************************************
**************Round 0 test running...*****************
will not launch list test!
******************************************************
Test 1: Creating commonfile /mnt/shared/multi_xattr_test_place/test_normal-0 on n1(rank 0),to perform 10000 EAs.
Warning:list size exceed,due to (xattr_name_sz + 1) * xattr_nums was greater than 65536,will not launch list test!
Warning:list size exceed,due to (xattr_name_sz + 1) * xattr_nums was greater than 65536,will not launch list test!
Test 2: Performancing Xattrs on /mnt/shared/multi_xattr_test_place/test_normal-0,all ranks take race to do update/remove/add/read 10000 EAs.
crash> bt 28037
PID: 28037 TASK: ffff880036f2c640 CPU: 2 COMMAND: "xattr-multi-tes"
#0 [ffff88012a02f8b8] schedule at ffffffff8151ecbf
#1 [ffff88012a02f928] schedule_timeout at ffffffff8151d3c9
#2 [ffff88012a02f9d8] wait_for_completion at ffffffff8151e3df
#3 [ffff88012a02fa30] __ocfs2_cluster_lock at ffffffffa05ab1b8 [ocfs2]
#4 [ffff88012a02fb10] ocfs2_inode_lock_full_nested at ffffffffa05acf78 [ocfs2]
#5 [ffff88012a02fb80] ocfs2_xattr_set at ffffffffa0612b89 [ocfs2]
#6 [ffff88012a02fd30] generic_setxattr at ffffffff811c910c
#7 [ffff88012a02fd60] __vfs_setxattr_noperm at ffffffff811c9941
#8 [ffff88012a02fda8] vfs_setxattr at ffffffff811c9b44
#9 [ffff88012a02fde8] setxattr at ffffffff811c9c5c
#10 [ffff88012a02ff38] sys_fsetxattr at ffffffff811ca0bd
#11 [ffff88012a02ff80] system_call_fastpath at ffffffff81529809
RIP: 00007f86d36c7fba RSP: 00007ffcd8bf9d10 RFLAGS: 00000246
RAX: 00000000000000be RBX: ffffffff81529809 RCX: 00007f86d39860b8
RDX: 0000000001c5b770 RSI: 0000000001c57ae0 RDI: 000000000000000f
RBP: 0000000000000000 R8: 0000000000000001 R9: 00007f86d3986120
R10: 0000000000001388 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffcd8bf9f60 R14: 00000000004018c0 R15: 00007ffcd8bf9e40
ORIG_RAX: 00000000000000be CS: 0033 SS: 002b
crash> bt -f 28037
PID: 28037 TASK: ffff880036f2c640 CPU: 2 COMMAND: "xattr-multi-tes"
#0 [ffff88012a02f8b8] schedule at ffffffff8151ecbf
ffff88012a02f8c0: 0000000000000082 ffff880036f2c640
ffff88012a02f8d0: ffff88012a02ffd8 ffff88012a02ffd8
ffff88012a02f8e0: ffff88012a02ffd8 ffff88013a784300
ffff88012a02f8f0: ffff880036f2c640 ffff88012a02fab0
ffff88012a02f900: 7fffffffffffffff ffff88012a02faa8
ffff88012a02f910: ffff880036f2c640 ffff88008e6ae280
ffff88012a02f920: ffff88012a02f9d0 ffffffff8151d3c9
#1 [ffff88012a02f928] schedule_timeout at ffffffff8151d3c9
ffff88012a02f930: 7fffffffffffffff ffff88012a02fe05
ffff88012a02f940: ffffffffa060eaa6 ffff88008f123c30
ffff88012a02f950: ffff88012a02fc50 0000000000000000
ffff88012a02f960: ffff880091584880 ffff880091584880
ffff88012a02f970: ffff88012a02fa68 ffffffffa05fc905
ffff88012a02f980: ffffffff811d6e62 000000000000008a
ffff88012a02f990: ffffffffa05ab0df 000000001374e17e
ffff88012a02f9a0: ffffffff81093b88 ffff88012a02fab0
ffff88012a02f9b0: 7fffffffffffffff ffff88012a02faa8
ffff88012a02f9c0: ffff880036f2c640 ffff88008e6ae280
ffff88012a02f9d0: ffff88012a02fa28 ffffffff8151e3df
#2 [ffff88012a02f9d8] wait_for_completion at ffffffff8151e3df
ffff88012a02f9e0: 0000000000000001 ffff880036f2c640
ffff88012a02f9f0: ffffffff8108afb0 ffff88012a02fab8
ffff88012a02fa00: ffff88012a02fab8 0000000000000005
ffff88012a02fa10: ffff88008e6ae2ec 0000000000000246
ffff88012a02fa20: ffff880036f2c640 ffff88012a02fb08
ffff88012a02fa30: ffffffffa05ab1b8
#3 [ffff88012a02fa30] __ocfs2_cluster_lock at ffffffffa05ab1b8 [ocfs2]
ffff88012a02fa38: ffff88013fd11f40 ffffffff81088bc5
ffff88012a02fa48: ffff8801393652c0 ffff88012a02fa78
ffff88012a02fa58: ffffffff81088bf9 ffff8801393652c0
ffff88012a02fa68: ffff880036f2cf08 ffff880036f2c640
ffff88012a02fa78: ffff8801383ea000 0000000000000008
ffff88012a02fa88: 0000000000000000 ffff880137659c70
ffff88012a02fa98: ffff88008e6ae2a0 ffff880100000000
ffff88012a02faa8: 0000000000000000 0000000000070007
ffff88012a02fab8: ffff88012a02f9f8 ffff88012a02f9f8
ffff88012a02fac8: 0000000000000004 0000000000000000
ffff88012a02fad8: 0000000000008050 ffff88012a02fbe0
ffff88012a02fae8: ffff88008e6ae280 0000000000000000
ffff88012a02faf8: ffff88008e6ae578 ffff8801383ea000
ffff88012a02fb08: 0000000000000001 ffffffffa05acf78
#4 [ffff88012a02fb10] ocfs2_inode_lock_full_nested at ffffffffa05acf78 [ocfs2]
ffff88012a02fb18: ffff88012a02fc80 ffffffff8107bc24
ffff88012a02fb28: ffff880000008050 0000000000000000
ffff88012a02fb38: ffff88008e6ae578 0000000000000008
ffff88012a02fb48: 0000000000000000 ffff88008e6ae578
ffff88012a02fb58: 0000000000000000 ffff88012a02fe05
ffff88012a02fb68: 0000000000000001 ffff8801383ea000
ffff88012a02fb78: ffff88012a02fc80 ffffffffa0612b89
#5 [ffff88012a02fb80] ocfs2_xattr_set at ffffffffa0612b89 [ocfs2]
ffff88012a02fb88: ffff88012a02fc10 0000000000000001
ffff88012a02fb98: ffff88013932c000 ffff880100000001
ffff88012a02fba8: ffff88008e679cf8 0000000000011f40
ffff88012a02fbb8: ffff8801393652c0 0000000000000002
ffff88012a02fbc8: 0000000000000000 ffffffff00000000
ffff88012a02fbd8: ffff88013f007400 0000000000000000
ffff88012a02fbe8: 0000000000000000 ffff880100000065
ffff88012a02fbf8: ffffffff81093b88 ffff880036f2c6a8
ffff88012a02fc08: ffff88013fd11fb8 0000000000000001
ffff88012a02fc18: 0000000000000001 0000000000000001
ffff88012a02fc28: ffff880000000001 ffff88012a02fe05
ffff88012a02fc38: ffff88010000000c ffff88013932c000
ffff88012a02fc48: 0000000000001388 0000000000000000
ffff88012a02fc58: 0000000000000000 0000000000000000
ffff88012a02fc68: 0000000000000000 0000000000000000
ffff88012a02fc78: 0000000000000000 0000000000000000
ffff88012a02fc88: 0000000000000000 0000000000000000
ffff88012a02fc98: 0000000000000000 0000000000000000
ffff88012a02fca8: 0000000000000000 0000000000000000
ffff88012a02fcb8: 00000000ffffffc3 0000000000000000
ffff88012a02fcc8: 0000000000000000 0000000000000000
ffff88012a02fcd8: ffff880139ed7dc0 0000000000000000
ffff88012a02fce8: 0000000000000000 0000000000000000
ffff88012a02fcf8: 00000000ffffffc3 ffff88013932c000
ffff88012a02fd08: ffff880138e85600 0000000000001388
ffff88012a02fd18: 0000000000000001 0000000000001388
ffff88012a02fd28: 0000000000000001 ffffffff811c910c
#6 [ffff88012a02fd30] generic_setxattr at ffffffff811c910c
ffff88012a02fd38: ffff88012a02fe05 ffff880138e85600
ffff88012a02fd48: ffff880138e85600 ffff88012a02fe00
ffff88012a02fd58: ffff88013932c000 ffffffff811c9941
#7 [ffff88012a02fd60] __vfs_setxattr_noperm at ffffffff811c9941
ffff88012a02fd68: ffff880138e85600 ffff88013932c000
ffff88012a02fd78: ffff880138e85600 ffff88012a02fe00
ffff88012a02fd88: ffff88013932c000 0000000000001388
ffff88012a02fd98: 0000000000000001 ffff88008e6ae620
ffff88012a02fda8: ffffffff811c9b44
#8 [ffff88012a02fda8] vfs_setxattr at ffffffff811c9b44
ffff88012a02fdb0: 0000000000000286 0000000000001388
ffff88012a02fdc0: ffff88012a02fe00 0000000000000001
ffff88012a02fdd0: 0000000000000000 ffff88013932c000
ffff88012a02fde0: fffffffffffffff2 ffffffff811c9c5c
#9 [ffff88012a02fde8] setxattr at ffffffff811c9c5c
ffff88012a02fdf0: 0000000001c5b770 ffff880138e85600
ffff88012a02fe00: 2d316e2e72657375 30392d306b6e6172
ffff88012a02fe10: 0000000000000034 ffffffff8107bbb2
ffff88012a02fe20: ffff8800b6baa400 0000000000000246
ffff88012a02fe30: ffffffff81365ffa ffff88013991e380
ffff88012a02fe40: ffff8800b6baa4d8 ffff8800b6baa000
ffff88012a02fe50: 00000001b6baa620 ffff8800b6baa620
ffff88012a02fe60: 0000000000000202 0000000000000001
ffff88012a02fe70: 0000000000000001 0000000000000004
ffff88012a02fe80: ffff880036f2c640 ffffffff811e53ec
ffff88012a02fe90: ffff880139365328 ffff88013fc11fb8
ffff88012a02fea0: ffff880139365328 ffff880138e85600
ffff88012a02feb0: ffffffff81090f1b ffff88008f747800
ffff88012a02fec0: 0000000000000001 ffffffff811a8fb2
ffff88012a02fed0: ffff88013fd11f40 ffff8800b42525e0
ffff88012a02fee0: 0000000000000000 000060fec0000e78
ffff88012a02fef0: 0000000000000246 ffff8800367b54c0
ffff88012a02ff00: 000000001374e17e 0000000000000001
ffff88012a02ff10: ffff880138e85600 ffff8800367b54c0
ffff88012a02ff20: 0000000001c5b770 0000000000001388
ffff88012a02ff30: 0000000000000001 ffffffff811ca0bd
#10 [ffff88012a02ff38] sys_fsetxattr at ffffffff811ca0bd
ffff88012a02ff40: 0000000001c57ae0 00000001004046a2
ffff88012a02ff50: 00000000004046a2 00007ffcd8bf9e40
ffff88012a02ff60: 00000000004018c0 00007ffcd8bf9f60
ffff88012a02ff70: 0000000000000000 0000000000000000
ffff88012a02ff80: ffffffff81529809
#11 [ffff88012a02ff80] system_call_fastpath at ffffffff81529809
RIP: 00007f86d36c7fba RSP: 00007ffcd8bf9d10 RFLAGS: 00000246
RAX: 00000000000000be RBX: ffffffff81529809 RCX: 00007f86d39860b8
RDX: 0000000001c5b770 RSI: 0000000001c57ae0 RDI: 000000000000000f
RBP: 0000000000000000 R8: 0000000000000001 R9: 00007f86d3986120
R10: 0000000000001388 R11: 0000000000000246 R12: 0000000000000000