search

Seagate / cortx-motr

CORTX Motr is a distributed object and key-value storage system targeting mass capacity storage configurations. It's the core component of CORTX storage system.
https://github.com/Seagate/cortx
Apache License 2.0
59 stars 142 forks source link

Ctgdump failed and all2all test #2052

Closed faradawn closed 2 years ago

faradawn commented 2 years ago

Problem

Hi! Been trying to run ctgdump_test, but got a failure status in the end:

[cc@skylake-loop-1 addb_folder]$ sudo sh /home/cc/cortx-motr/cas/st/ctgdump_tesh.sh test
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(2,1,1,4) nr_ios=4 multiple_pools=0
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00102744 s, 1.0 GB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00104994 s, 999 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.000907324 s, 1.2 GB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.000950443 s, 1.1 GB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00107433 s, 976 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.000933436 s, 1.1 GB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00100124 s, 1.0 GB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00093895 s, 1.1 GB/s

[104:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
      [1: "4 2 1 1"],
      [1: ^n|1:2],
      [1: ^S|1:6],
      [3: ^o|1:9, ^o|20:1, ^o|2:9],
      [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.52.0.33@tcp:12345:33:1", [1: ^s|1:101])}, 
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.52.0.33@tcp:12345:33:900", [8: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0,                     ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])}, 
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.52.0.33@tcp:12345:33:901", [8: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1,                     ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])}, 
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.52.0.33@tcp:12345:33:902", [8: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2,                     ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])}, 
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.52.0.33@tcp:12345:33:903", [8: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3,                     ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])}, 
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.52.0.33@tcp:12345:33:800", [3: ^s|2:0, ^s|12:0, ^s|3:4])}, 
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.52.0.33@tcp:12345:34:1",
                          [3: ^s|1:6, ^s|1:7, ^s|3:6])}, 
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.52.0.33@tcp:12345:33:100",
                          [2: ^s|8:0, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.52.0.33@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.52.0.33@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.52.0.33@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.52.0.33@tcp:12345:33:1"], [0], [0])},

 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.52.0.33@tcp:12345:33:800"], [0], [0])}, 
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.52.0.33@tcp:12345:33:800"], [0], [0])}, 
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.52.0.33@tcp:12345:33:800"], [0], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.52.0.33@tcp:12345:33:900"], [0], [1: ^d|1:1])}, 
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.52.0.33@tcp:12345:33:900"], [0], [0])},                    
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.52.0.33@tcp:12345:33:900"], [0], [0])}, 
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.52.0.33@tcp:12345:33:900"], [0], [0])}, 
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.52.0.33@tcp:12345:33:900"], [0], [0])}, 
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.52.0.33@tcp:12345:33:900"], [0], [1: ^d|20:0])}, 
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.52.0.33@tcp:12345:33:900"], [0], [0])}, 
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.52.0.33@tcp:12345:33:900"], [0], [0])}, 
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.52.0.33@tcp:12345:33:901"], [0], [1: ^d|1:2])}, 
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.52.0.33@tcp:12345:33:901"], [0], [0])},                    
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.52.0.33@tcp:12345:33:901"], [0], [0])}, 
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.52.0.33@tcp:12345:33:901"], [0], [0])}, 
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.52.0.33@tcp:12345:33:901"], [0], [0])}, 
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.52.0.33@tcp:12345:33:901"], [0], [1: ^d|20:1])}, 
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.52.0.33@tcp:12345:33:901"], [0], [0])}, 
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.52.0.33@tcp:12345:33:901"], [0], [0])}, 
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.52.0.33@tcp:12345:33:902"], [0], [1: ^d|1:3])}, 
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.52.0.33@tcp:12345:33:902"], [0], [0])},                    
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.52.0.33@tcp:12345:33:902"], [0], [0])}, 
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.52.0.33@tcp:12345:33:902"], [0], [0])}, 
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.52.0.33@tcp:12345:33:902"], [0], [0])}, 
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.52.0.33@tcp:12345:33:902"], [0], [1: ^d|20:2])}, 
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.52.0.33@tcp:12345:33:902"], [0], [0])}, 
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.52.0.33@tcp:12345:33:902"], [0], [0])}, 
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.52.0.33@tcp:12345:33:903"], [0], [1: ^d|1:4])}, 
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.52.0.33@tcp:12345:33:903"], [0], [0])},                    
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.52.0.33@tcp:12345:33:903"], [0], [0])}, 
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.52.0.33@tcp:12345:33:903"], [0], [0])}, 
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.52.0.33@tcp:12345:33:903"], [0], [0])}, 
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.52.0.33@tcp:12345:33:903"], [0], [1: ^d|20:3])}, 
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.52.0.33@tcp:12345:33:903"], [0], [0])}, 
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.52.0.33@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.52.0.33@tcp:12345:34:1"], [0], [0])}, 
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.52.0.33@tcp:12345:33:100"], [0], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop1")}, 
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])}, 
 {0x6a| ((^j|1:1), ^k|1:1, [0])}, 
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop2")}, 
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])}, 
 {0x6a| ((^j|1:2), ^k|1:2, [0])}, 
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop3")}, 
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])}, 
 {0x6a| ((^j|1:3), ^k|1:3, [0])}, 
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop4")}, 
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])}, 
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [8: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (2, 1,
                                            1, 4,
                                            [5: 0, 0, 0, 0, 1],
                                            [1: ^j|1:5])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:5), ^S|1:6, [1: ^j|1:6])},
  {0x6a| ((^j|1:6), ^a|1:6, [1: ^j|1:7])},
  {0x6a| ((^j|1:7), ^e|1:7, [1: ^j|1:8])},
  {0x6a| ((^j|1:8), ^c|1:8, [4: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4])} , 
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])}, 
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:5])})}, 
{0x6a| ((^j|2:5), ^S|1:6, [1: ^j|2:6])}, 
{0x6a| ((^j|2:6), ^a|1:6, [1: ^j|2:7])}, 
{0x6a| ((^j|2:7), ^e|1:7, [1: ^j|2:8])}, 
{0x6a| ((^j|2:8), ^c|1:8, [4: ^j|2:1, ^j|2:2, ^j|2:3, ^j|2:4])}, 
{0x6a| ((^j|2:1), ^k|1:1, [0])}, 
 {0x6a| ((^j|2:2), ^k|1:2, [0])}, 
 {0x6a| ((^j|2:3), ^k|1:3, [0])}, 
 {0x6a| ((^j|2:4), ^k|1:4, [0])}  , 
{0x64| ((^d|20:0), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")}, 
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])}, 
{0x6a| ((^j|20:100), ^k|20:0, [0])}, 
{0x64| ((^d|20:1), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")}, 
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])}, 
{0x6a| ((^j|20:101), ^k|20:1, [0])}, 
{0x64| ((^d|20:2), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")}, 
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])}, 
{0x6a| ((^j|20:102), ^k|20:2, [0])}, 
{0x64| ((^d|20:3), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")}, 
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])}, 
{0x6a| ((^j|20:103), ^k|20:3, [0])}, 
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])}, 
{0x76| ((^v|1:20), {0| (1, 3, 0,
                                                    4,
                                                    [5: 0, 0, 0, 0, 3],
                                                    [1: ^j|20:1])})}, 
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])}, 
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])}, 
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])}, 
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/systest-278078/confd && exec /home/cc/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                -w 4 -m 65536               -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.52.0.33@tcp:12345:35:1             -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-278078/confd && exec /home/cc/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.52.0.33@tcp:12345:33:100 -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0d.log
cd /var/motr/systest-278078/ha && exec /home/cc/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.52.0.33@tcp:12345:35:1 -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0mkfs.log
motr[278387]:  8cc0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/systest-278078/mds1 && exec            /home/cc/cortx-motr/utils/mkfs/m0mkfs -F -T ad      -D db -S stobs -A linuxstob:addb-stobs               -w 4 -m 65536               -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:35:800          -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-278078/ios1 && exec /home/cc/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-278078/ios2 && exec /home/cc/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-278078/ios3 && exec /home/cc/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-278078/ios4 && exec /home/cc/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-278078/ha && exec /home/cc/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs           -w 4 -m 65536               -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.52.0.33@tcp:12345:34:1            -c /var/motr/systest-278078/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.52.0.33@tcp:12345:34:1 |& tee -a m0d.log
motr[01757]:  5780  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/systest-278078/mds1 && exec /home/cc/cortx-motr/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.52.0.33@tcp:12345:34:1 -c /var/motr/systest-278078/confd/conf.xc |& tee -a m0d.log
motr[01970]:  bf90  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/systest-278078/ios1 && exec /home/cc/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.52.0.33@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/systest-278078/ios2 && exec /home/cc/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.52.0.33@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/systest-278078/ios3 && exec /home/cc/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.52.0.33@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/systest-278078/ios4 && exec /home/cc/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.52.0.33@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.52.0.33@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
motr[02193]:  1930  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[02197]:  f0e0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[02203]:  6880  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[02211]:  de70  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
/home/cc/cortx-motr/utils/m0kv -l 10.52.0.33@tcp:12345:33:1000  -h 10.52.0.33@tcp:12345:34:1 -p 0x7000000000000001:0            -f 0x7200000000000001:64 -s  index create 7800000000012345:12345
Client Command Line tool: m0kv
Usage: ./m0kv -l local_addr -h ha_addr -p profile -f proc_fid [-s] [subsystem] [subsystem commands]

Use -? for more verbose help on common arguments.
Usage example for common arguments: 
./m0kv -l 10.0.2.15@tcp:12345:33:100 -h 10.0.2.15@tcp:12345:34:1 -p '<0x7000000000000001:0>' -f '<0x7200000000000000:0>' [subsystem] [subsystem commands]

-s  Enable string format and it is optional.
-e  Enable M0_ENF_META flag and it is optional.
-v 7600000000000001:30 it is mandatory with -e for PUT, GET, DEL and NEXT operations.
-L  Enable M0_OIF_SKIP_LAYOUT flag and it is optional.
-v 7600000000000001:30 it is mandatory with -L for other operations except CREATE.
-C  Disable M0_OIF_CROW flag and it is optional.
Available subsystems and subsystem-specific commands are listed below.

    INDEX subsystem
         'index [commands]'- name of subsystem
         Commands:
            -'create FID_PARAM, create index'
            -'drop FID_PARAM, drop existing index'
            -'list FID NUM, get indicies'
            -'lookup FID_PARAM, lookup index in storage'
            -'put FID_PARAM KEY_PARAM VAL_PARAM, put record'
            -'del FID_PARAM KEY_PARAM, delete record'
            -'get FID KEY_PARAM, lookup and returns values by key'
            -'next FID KEY CNT, returns records larger than KEY '
            -'genf CNT FILE, generate file with several FID'
            -'genv CNT SIZE FILE, generate file with several KEY_PARAM/VAL_PARAM. Note: SIZE > 16'
            -'wait FILE, await a file to appear'
    NOTE:
    - FID_PARAM - single FID value or @FIDFILENAME
    - KEY_PARAM - single KEY value or @VFILENAME
    - VAL_PARAM - single VALUE param or @VFILENAME
    - FID - value in m0_fid_sscanf format, e.g. '<0x780000000000000b:1>', '1:5' and etc
    - KEY or VALUE - value in m0_xcode_read format, e.g. '[0xa:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0a]'
    Example: 
        >m0kv [common args] index create "1:5" 
        >m0kv [common args] index list "1:5" 2 
        >m0kv [common args] index genf 10 fid.txt 
        >m0kv [common args] index genv 10 20 keys.txt 
        >m0kv [common args] index genv 10 20 vals.txt 
        >m0kv [common args] index put "1:5" @keys.txt @vals.txt 
        >m0kv [common args] index get "1:5" @keys.txt 
        >m0kv [common args] index next "1:5" '[0x02:0x01,0x02]' 3 
        >m0kv [common args] index next "1:5" "0" 3 -s 
    Possible to supply multiple commands on command line e.g.:
        >m0kv [common args] index create "1:5" put "1:5" "[0x02:0x01,0x02]" "[0x09:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09]"
        >m0kv [common args] index drop "1:5" create "1:5" put "1:5" @keys.txt @vals.txt
        >m0kv [common args] -s index put "1:5" "Department" "Testing" 
        Note: If key already exists put over-write the old value.
        >m0kv [common args] -s index get "1:5" "Department" 
        Note: DIX pool version info on console if pass -e or -Lflag for index create.
        >m0kv [common args] -e index create "1:5" 
        Note: Pass pool version info for PUT/GET/DEL/NEXT operations if pass -e flag 
        >m0kv [common args] -e -v 7600000000000001:30 -s index put "1:5" "Department" "Testing" 
        >m0kv [common args] -L index create "1:6" 
        Note: Pass pool version info for other operations if pass -L flag 
        >m0kv [common args] -L -v 7600000000000001:30 -s index put "1:6" "Department" "Testing" 
        >m0kv [common args] -C index create "1:6" 
motr[03069]:  8600  ERROR  [m0kv/cmd_main.c:166:opts_get]  <! rc=-22
Got error -22
Done, rc:  22
m0kv failed
Client Command Line tool: m0kv
Usage: ./m0kv -l local_addr -h ha_addr -p profile -f proc_fid [-s] [subsystem] [subsystem commands]

Use -? for more verbose help on common arguments.
Usage example for common arguments: 
./m0kv -l 10.0.2.15@tcp:12345:33:100 -h 10.0.2.15@tcp:12345:34:1 -p '<0x7000000000000001:0>' -f '<0x7200000000000000:0>' [subsystem] [subsystem commands]

-s  Enable string format and it is optional.
-e  Enable M0_ENF_META flag and it is optional.
-v 7600000000000001:30 it is mandatory with -e for PUT, GET, DEL and NEXT operations.
-L  Enable M0_OIF_SKIP_LAYOUT flag and it is optional.
-v 7600000000000001:30 it is mandatory with -L for other operations except CREATE.
-C  Disable M0_OIF_CROW flag and it is optional.
Available subsystems and subsystem-specific commands are listed below.

    INDEX subsystem
         'index [commands]'- name of subsystem
         Commands:
            -'create FID_PARAM, create index'
            -'drop FID_PARAM, drop existing index'
            -'list FID NUM, get indicies'
            -'lookup FID_PARAM, lookup index in storage'
            -'put FID_PARAM KEY_PARAM VAL_PARAM, put record'
            -'del FID_PARAM KEY_PARAM, delete record'
            -'get FID KEY_PARAM, lookup and returns values by key'
            -'next FID KEY CNT, returns records larger than KEY '
            -'genf CNT FILE, generate file with several FID'
            -'genv CNT SIZE FILE, generate file with several KEY_PARAM/VAL_PARAM. Note: SIZE > 16'
            -'wait FILE, await a file to appear'
    NOTE:
    - FID_PARAM - single FID value or @FIDFILENAME
    - KEY_PARAM - single KEY value or @VFILENAME
    - VAL_PARAM - single VALUE param or @VFILENAME
    - FID - value in m0_fid_sscanf format, e.g. '<0x780000000000000b:1>', '1:5' and etc
    - KEY or VALUE - value in m0_xcode_read format, e.g. '[0xa:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0a]'
    Example: 
        >m0kv [common args] index create "1:5" 
        >m0kv [common args] index list "1:5" 2 
        >m0kv [common args] index genf 10 fid.txt 
        >m0kv [common args] index genv 10 20 keys.txt 
        >m0kv [common args] index genv 10 20 vals.txt 
        >m0kv [common args] index put "1:5" @keys.txt @vals.txt 
        >m0kv [common args] index get "1:5" @keys.txt 
        >m0kv [common args] index next "1:5" '[0x02:0x01,0x02]' 3 
        >m0kv [common args] index next "1:5" "0" 3 -s 
    Possible to supply multiple commands on command line e.g.:
        >m0kv [common args] index create "1:5" put "1:5" "[0x02:0x01,0x02]" "[0x09:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09]"
        >m0kv [common args] index drop "1:5" create "1:5" put "1:5" @keys.txt @vals.txt
        >m0kv [common args] -s index put "1:5" "Department" "Testing" 
        Note: If key already exists put over-write the old value.
        >m0kv [common args] -s index get "1:5" "Department" 
        Note: DIX pool version info on console if pass -e or -Lflag for index create.
        >m0kv [common args] -e index create "1:5" 
        Note: Pass pool version info for PUT/GET/DEL/NEXT operations if pass -e flag 
        >m0kv [common args] -e -v 7600000000000001:30 -s index put "1:5" "Department" "Testing" 
        >m0kv [common args] -L index create "1:6" 
        Note: Pass pool version info for other operations if pass -L flag 
        >m0kv [common args] -L -v 7600000000000001:30 -s index put "1:6" "Department" "Testing" 
        >m0kv [common args] -C index create "1:6" 
motr[03097]:  a740  ERROR  [m0kv/cmd_main.c:166:opts_get]  <! rc=-22
Got error -22
Done, rc:  22
Client Command Line tool: m0kv
Usage: ./m0kv -l local_addr -h ha_addr -p profile -f proc_fid [-s] [subsystem] [subsystem commands]

Use -? for more verbose help on common arguments.
Usage example for common arguments: 
./m0kv -l 10.0.2.15@tcp:12345:33:100 -h 10.0.2.15@tcp:12345:34:1 -p '<0x7000000000000001:0>' -f '<0x7200000000000000:0>' [subsystem] [subsystem commands]

-s  Enable string format and it is optional.
-e  Enable M0_ENF_META flag and it is optional.
-v 7600000000000001:30 it is mandatory with -e for PUT, GET, DEL and NEXT operations.
-L  Enable M0_OIF_SKIP_LAYOUT flag and it is optional.
-v 7600000000000001:30 it is mandatory with -L for other operations except CREATE.
-C  Disable M0_OIF_CROW flag and it is optional.
Available subsystems and subsystem-specific commands are listed below.

    INDEX subsystem
         'index [commands]'- name of subsystem
         Commands:
            -'create FID_PARAM, create index'
            -'drop FID_PARAM, drop existing index'
            -'list FID NUM, get indicies'
            -'lookup FID_PARAM, lookup index in storage'
            -'put FID_PARAM KEY_PARAM VAL_PARAM, put record'
            -'del FID_PARAM KEY_PARAM, delete record'
            -'get FID KEY_PARAM, lookup and returns values by key'
            -'next FID KEY CNT, returns records larger than KEY '
            -'genf CNT FILE, generate file with several FID'
            -'genv CNT SIZE FILE, generate file with several KEY_PARAM/VAL_PARAM. Note: SIZE > 16'
            -'wait FILE, await a file to appear'
    NOTE:
    - FID_PARAM - single FID value or @FIDFILENAME
    - KEY_PARAM - single KEY value or @VFILENAME
    - VAL_PARAM - single VALUE param or @VFILENAME
    - FID - value in m0_fid_sscanf format, e.g. '<0x780000000000000b:1>', '1:5' and etc
    - KEY or VALUE - value in m0_xcode_read format, e.g. '[0xa:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0a]'
    Example: 
        >m0kv [common args] index create "1:5" 
        >m0kv [common args] index list "1:5" 2 
        >m0kv [common args] index genf 10 fid.txt 
        >m0kv [common args] index genv 10 20 keys.txt 
        >m0kv [common args] index genv 10 20 vals.txt 
        >m0kv [common args] index put "1:5" @keys.txt @vals.txt 
        >m0kv [common args] index get "1:5" @keys.txt 
        >m0kv [common args] index next "1:5" '[0x02:0x01,0x02]' 3 
        >m0kv [common args] index next "1:5" "0" 3 -s 
    Possible to supply multiple commands on command line e.g.:
        >m0kv [common args] index create "1:5" put "1:5" "[0x02:0x01,0x02]" "[0x09:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09]"
        >m0kv [common args] index drop "1:5" create "1:5" put "1:5" @keys.txt @vals.txt
        >m0kv [common args] -s index put "1:5" "Department" "Testing" 
        Note: If key already exists put over-write the old value.
        >m0kv [common args] -s index get "1:5" "Department" 
        Note: DIX pool version info on console if pass -e or -Lflag for index create.
        >m0kv [common args] -e index create "1:5" 
        Note: Pass pool version info for PUT/GET/DEL/NEXT operations if pass -e flag 
        >m0kv [common args] -e -v 7600000000000001:30 -s index put "1:5" "Department" "Testing" 
        >m0kv [common args] -L index create "1:6" 
        Note: Pass pool version info for other operations if pass -L flag 
        >m0kv [common args] -L -v 7600000000000001:30 -s index put "1:6" "Department" "Testing" 
        >m0kv [common args] -C index create "1:6" 
motr[03120]:  bce0  ERROR  [m0kv/cmd_main.c:166:opts_get]  <! rc=-22
Got error -22
Done, rc:  22
Client Command Line tool: m0kv
Usage: ./m0kv -l local_addr -h ha_addr -p profile -f proc_fid [-s] [subsystem] [subsystem commands]

Use -? for more verbose help on common arguments.
Usage example for common arguments: 
./m0kv -l 10.0.2.15@tcp:12345:33:100 -h 10.0.2.15@tcp:12345:34:1 -p '<0x7000000000000001:0>' -f '<0x7200000000000000:0>' [subsystem] [subsystem commands]

-s  Enable string format and it is optional.
-e  Enable M0_ENF_META flag and it is optional.
-v 7600000000000001:30 it is mandatory with -e for PUT, GET, DEL and NEXT operations.
-L  Enable M0_OIF_SKIP_LAYOUT flag and it is optional.
-v 7600000000000001:30 it is mandatory with -L for other operations except CREATE.
-C  Disable M0_OIF_CROW flag and it is optional.
Available subsystems and subsystem-specific commands are listed below.

    INDEX subsystem
         'index [commands]'- name of subsystem
         Commands:
            -'create FID_PARAM, create index'
            -'drop FID_PARAM, drop existing index'
            -'list FID NUM, get indicies'
            -'lookup FID_PARAM, lookup index in storage'
            -'put FID_PARAM KEY_PARAM VAL_PARAM, put record'
            -'del FID_PARAM KEY_PARAM, delete record'
            -'get FID KEY_PARAM, lookup and returns values by key'
            -'next FID KEY CNT, returns records larger than KEY '
            -'genf CNT FILE, generate file with several FID'
            -'genv CNT SIZE FILE, generate file with several KEY_PARAM/VAL_PARAM. Note: SIZE > 16'
            -'wait FILE, await a file to appear'
    NOTE:
    - FID_PARAM - single FID value or @FIDFILENAME
    - KEY_PARAM - single KEY value or @VFILENAME
    - VAL_PARAM - single VALUE param or @VFILENAME
    - FID - value in m0_fid_sscanf format, e.g. '<0x780000000000000b:1>', '1:5' and etc
    - KEY or VALUE - value in m0_xcode_read format, e.g. '[0xa:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0a]'
    Example: 
        >m0kv [common args] index create "1:5" 
        >m0kv [common args] index list "1:5" 2 
        >m0kv [common args] index genf 10 fid.txt 
        >m0kv [common args] index genv 10 20 keys.txt 
        >m0kv [common args] index genv 10 20 vals.txt 
        >m0kv [common args] index put "1:5" @keys.txt @vals.txt 
        >m0kv [common args] index get "1:5" @keys.txt 
        >m0kv [common args] index next "1:5" '[0x02:0x01,0x02]' 3 
        >m0kv [common args] index next "1:5" "0" 3 -s 
    Possible to supply multiple commands on command line e.g.:
        >m0kv [common args] index create "1:5" put "1:5" "[0x02:0x01,0x02]" "[0x09:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09]"
        >m0kv [common args] index drop "1:5" create "1:5" put "1:5" @keys.txt @vals.txt
        >m0kv [common args] -s index put "1:5" "Department" "Testing" 
        Note: If key already exists put over-write the old value.
        >m0kv [common args] -s index get "1:5" "Department" 
        Note: DIX pool version info on console if pass -e or -Lflag for index create.
        >m0kv [common args] -e index create "1:5" 
        Note: Pass pool version info for PUT/GET/DEL/NEXT operations if pass -e flag 
        >m0kv [common args] -e -v 7600000000000001:30 -s index put "1:5" "Department" "Testing" 
        >m0kv [common args] -L index create "1:6" 
        Note: Pass pool version info for other operations if pass -L flag 
        >m0kv [common args] -L -v 7600000000000001:30 -s index put "1:6" "Department" "Testing" 
        >m0kv [common args] -C index create "1:6" 
motr[03143]:  58c0  ERROR  [m0kv/cmd_main.c:166:opts_get]  <! rc=-22
Got error -22
Done, rc:  22
Client Command Line tool: m0kv
Usage: ./m0kv -l local_addr -h ha_addr -p profile -f proc_fid [-s] [subsystem] [subsystem commands]

Use -? for more verbose help on common arguments.
Usage example for common arguments: 
./m0kv -l 10.0.2.15@tcp:12345:33:100 -h 10.0.2.15@tcp:12345:34:1 -p '<0x7000000000000001:0>' -f '<0x7200000000000000:0>' [subsystem] [subsystem commands]

-s  Enable string format and it is optional.
-e  Enable M0_ENF_META flag and it is optional.
-v 7600000000000001:30 it is mandatory with -e for PUT, GET, DEL and NEXT operations.
-L  Enable M0_OIF_SKIP_LAYOUT flag and it is optional.
-v 7600000000000001:30 it is mandatory with -L for other operations except CREATE.
-C  Disable M0_OIF_CROW flag and it is optional.
Available subsystems and subsystem-specific commands are listed below.

... deleted a portion of output ...

Use -? for more verbose help on common arguments.
Usage example for common arguments: 
./m0kv -l 10.0.2.15@tcp:12345:33:100 -h 10.0.2.15@tcp:12345:34:1 -p '<0x7000000000000001:0>' -f '<0x7200000000000000:0>' [subsystem] [subsystem commands]

-s  Enable string format and it is optional.
-e  Enable M0_ENF_META flag and it is optional.
-v 7600000000000001:30 it is mandatory with -e for PUT, GET, DEL and NEXT operations.
-L  Enable M0_OIF_SKIP_LAYOUT flag and it is optional.
-v 7600000000000001:30 it is mandatory with -L for other operations except CREATE.
-C  Disable M0_OIF_CROW flag and it is optional.
Available subsystems and subsystem-specific commands are listed below.

    INDEX subsystem
         'index [commands]'- name of subsystem
         Commands:
            -'create FID_PARAM, create index'
            -'drop FID_PARAM, drop existing index'
            -'list FID NUM, get indicies'
            -'lookup FID_PARAM, lookup index in storage'
            -'put FID_PARAM KEY_PARAM VAL_PARAM, put record'
            -'del FID_PARAM KEY_PARAM, delete record'
            -'get FID KEY_PARAM, lookup and returns values by key'
            -'next FID KEY CNT, returns records larger than KEY '
            -'genf CNT FILE, generate file with several FID'
            -'genv CNT SIZE FILE, generate file with several KEY_PARAM/VAL_PARAM. Note: SIZE > 16'
            -'wait FILE, await a file to appear'
    NOTE:
    - FID_PARAM - single FID value or @FIDFILENAME
    - KEY_PARAM - single KEY value or @VFILENAME
    - VAL_PARAM - single VALUE param or @VFILENAME
    - FID - value in m0_fid_sscanf format, e.g. '<0x780000000000000b:1>', '1:5' and etc
    - KEY or VALUE - value in m0_xcode_read format, e.g. '[0xa:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0a]'
    Example: 
        >m0kv [common args] index create "1:5" 
        >m0kv [common args] index list "1:5" 2 
        >m0kv [common args] index genf 10 fid.txt 
        >m0kv [common args] index genv 10 20 keys.txt 
        >m0kv [common args] index genv 10 20 vals.txt 
        >m0kv [common args] index put "1:5" @keys.txt @vals.txt 
        >m0kv [common args] index get "1:5" @keys.txt 
        >m0kv [common args] index next "1:5" '[0x02:0x01,0x02]' 3 
        >m0kv [common args] index next "1:5" "0" 3 -s 
    Possible to supply multiple commands on command line e.g.:
        >m0kv [common args] index create "1:5" put "1:5" "[0x02:0x01,0x02]" "[0x09:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09]"
        >m0kv [common args] index drop "1:5" create "1:5" put "1:5" @keys.txt @vals.txt
        >m0kv [common args] -s index put "1:5" "Department" "Testing" 
        Note: If key already exists put over-write the old value.
        >m0kv [common args] -s index get "1:5" "Department" 
        Note: DIX pool version info on console if pass -e or -Lflag for index create.
        >m0kv [common args] -e index create "1:5" 
        Note: Pass pool version info for PUT/GET/DEL/NEXT operations if pass -e flag 
        >m0kv [common args] -e -v 7600000000000001:30 -s index put "1:5" "Department" "Testing" 
        >m0kv [common args] -L index create "1:6" 
        Note: Pass pool version info for other operations if pass -L flag 
        >m0kv [common args] -L -v 7600000000000001:30 -s index put "1:6" "Department" "Testing" 
        >m0kv [common args] -C index create "1:6" 
motr[03419]:  d600  ERROR  [m0kv/cmd_main.c:166:opts_get]  <! rc=-22
Got error -22
Done, rc:  22
Client Command Line tool: m0kv
Usage: ./m0kv -l local_addr -h ha_addr -p profile -f proc_fid [-s] [subsystem] [subsystem commands]

Use -? for more verbose help on common arguments.
Usage example for common arguments: 
./m0kv -l 10.0.2.15@tcp:12345:33:100 -h 10.0.2.15@tcp:12345:34:1 -p '<0x7000000000000001:0>' -f '<0x7200000000000000:0>' [subsystem] [subsystem commands]

-s  Enable string format and it is optional.
-e  Enable M0_ENF_META flag and it is optional.
-v 7600000000000001:30 it is mandatory with -e for PUT, GET, DEL and NEXT operations.
-L  Enable M0_OIF_SKIP_LAYOUT flag and it is optional.
-v 7600000000000001:30 it is mandatory with -L for other operations except CREATE.
-C  Disable M0_OIF_CROW flag and it is optional.
Available subsystems and subsystem-specific commands are listed below.

    INDEX subsystem
         'index [commands]'- name of subsystem
         Commands:
            -'create FID_PARAM, create index'
            -'drop FID_PARAM, drop existing index'
            -'list FID NUM, get indicies'
            -'lookup FID_PARAM, lookup index in storage'
            -'put FID_PARAM KEY_PARAM VAL_PARAM, put record'
            -'del FID_PARAM KEY_PARAM, delete record'
            -'get FID KEY_PARAM, lookup and returns values by key'
            -'next FID KEY CNT, returns records larger than KEY '
            -'genf CNT FILE, generate file with several FID'
            -'genv CNT SIZE FILE, generate file with several KEY_PARAM/VAL_PARAM. Note: SIZE > 16'
            -'wait FILE, await a file to appear'
    NOTE:
    - FID_PARAM - single FID value or @FIDFILENAME
    - KEY_PARAM - single KEY value or @VFILENAME
    - VAL_PARAM - single VALUE param or @VFILENAME
    - FID - value in m0_fid_sscanf format, e.g. '<0x780000000000000b:1>', '1:5' and etc
    - KEY or VALUE - value in m0_xcode_read format, e.g. '[0xa:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0a]'
    Example: 
        >m0kv [common args] index create "1:5" 
        >m0kv [common args] index list "1:5" 2 
        >m0kv [common args] index genf 10 fid.txt 
        >m0kv [common args] index genv 10 20 keys.txt 
        >m0kv [common args] index genv 10 20 vals.txt 
        >m0kv [common args] index put "1:5" @keys.txt @vals.txt 
        >m0kv [common args] index get "1:5" @keys.txt 
        >m0kv [common args] index next "1:5" '[0x02:0x01,0x02]' 3 
        >m0kv [common args] index next "1:5" "0" 3 -s 
    Possible to supply multiple commands on command line e.g.:
        >m0kv [common args] index create "1:5" put "1:5" "[0x02:0x01,0x02]" "[0x09:0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09]"
        >m0kv [common args] index drop "1:5" create "1:5" put "1:5" @keys.txt @vals.txt
        >m0kv [common args] -s index put "1:5" "Department" "Testing" 
        Note: If key already exists put over-write the old value.
        >m0kv [common args] -s index get "1:5" "Department" 
        Note: DIX pool version info on console if pass -e or -Lflag for index create.
        >m0kv [common args] -e index create "1:5" 
        Note: Pass pool version info for PUT/GET/DEL/NEXT operations if pass -e flag 
        >m0kv [common args] -e -v 7600000000000001:30 -s index put "1:5" "Department" "Testing" 
        >m0kv [common args] -L index create "1:6" 
        Note: Pass pool version info for other operations if pass -L flag 
        >m0kv [common args] -L -v 7600000000000001:30 -s index put "1:6" "Department" "Testing" 
        >m0kv [common args] -C index create "1:6" 
motr[03442]:  cea0  ERROR  [m0kv/cmd_main.c:166:opts_get]  <! rc=-22
Got error -22
Done, rc:  22
Test failed with error 22
stopping /home/cc/cortx-motr/motr/m0d processes...
=== pids of services: 1757 1970 2193 2197 2203 2211 278387 ===
Shutting down services one by one. mdservice is the last.
----- 1757 stopping--------lt-m0d: got signal 1
----- 1757 stopped --------
----- 1970 stopping--------lt-m0d: got signal 1
motr[01970]:  4860   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1ea85a0 ep=10.52.0.33@tcp:12345:34:1 lq_tags=(confirmed=98 delivered=98 next=98 assign=122)
motr[01970]:  4860   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=48 hl=0x1ea85a0 ep=10.52.0.33@tcp:12345:34:1
motr[01970]:  4c10   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 1970 stopped --------
----- 2193 stopping--------lt-m0d: got signal 1
motr[02193]:  5860   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x312c0b0 ep=10.52.0.33@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=102)
motr[02193]:  5860   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x312c0b0 ep=10.52.0.33@tcp:12345:34:1
motr[02193]:  5c10   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2193 stopped --------
----- 2197 stopping--------lt-m0d: got signal 1
motr[02197]:  8860   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1f670b0 ep=10.52.0.33@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=116)
motr[02197]:  8860   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x1f670b0 ep=10.52.0.33@tcp:12345:34:1
motr[02197]:  8c10   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2197 stopped --------
----- 2203 stopping--------lt-m0d: got signal 1
motr[02203]:  6860   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x385d0b0 ep=10.52.0.33@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=130)
motr[02203]:  6860   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x385d0b0 ep=10.52.0.33@tcp:12345:34:1
motr[02203]:  6c10   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2203 stopped --------
----- 2211 stopping--------lt-m0d: got signal 1
motr[02211]:  1860   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x26090b0 ep=10.52.0.33@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=144)
motr[02211]:  1860   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x26090b0 ep=10.52.0.33@tcp:12345:34:1
motr[02211]:  1c10   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2211 stopped --------
----- 278387 stopping--------lt-m0d: got signal 1
----- 278387 stopped --------
Motr services stopped.
ctgdump: FAILURE 22

Expected behavior

As described in the ctgdump fix pull request, the test should pass.

How to reproduce

On a Skylake, CENTOS 7.9, can use the exact following commands to deploy motr:

# === First build motr === #
# clone repository
cd /home/cc
git clone --recursive https://github.com/Seagate/cortx-motr.git

# install pip and python
sudo yum group install -y "Development Tools"
sudo yum install -y python-devel ansible tmux
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py -o get-pip.py
python get-pip.py pip==19.3.1            
sudo pip install --target=/usr/lib64/python2.7/site-packages ipaddress

# force ansible to use python2
sudo bash -c "echo 'all:' >> /etc/ansible/hosts"
sudo bash -c "echo '  ansible_python_interpreter: \"/usr/bin/python2\"' >> /etc/ansible/hosts"

# run build dependencies (9 min)
cd /home/cc/cortx-motr
time sudo ./scripts/install-build-deps

# configure Luster interface to eth0
sudo sed -i 's|tcp(eth1)|tcp(eth0)|g' /etc/modprobe.d/lnet.conf
cat /etc/modprobe.d/lnet.conf
sudo modprobe lnet

# configure libfab interface to etho0
sudo sed -i 's|tcp(eth1)|tcp(eth0)|g' /etc/libfab.conf

# [Important!] build motr (1 min with 48 cores, 7 min with 1 core)
cd /home/cc/cortx-motr
sudo chown -R cc .
./autogen.sh && ./configure

# [option 1: default make]
make -j48

# [option 2: all2all make] 7 min
# time { MAKE_OPTS=-j48 CONFIGURE_OPTS=--enable-dtm0\ --disable-altogether-mode\ --enable-debug\ --with-trace-ubuf-size=32 sudo ./scripts/m0 rebuild || echo FAIL; }

# === Second build hare === #

# complie python util 
cd /home/cc
sudo yum install -y gcc rpm-build python36 python36-pip python36-devel python36-setuptools openssl-devel libffi-devel python36-dbus
git clone --recursive https://github.com/Seagate/cortx-utils -b main
cd cortx-utils
./jenkins/build.sh -v 2.0.0 -b 2
sudo pip3 install -r https://raw.githubusercontent.com/Seagate/cortx-utils/main/py-utils/python_requirements.txt
sudo pip3 install -r https://raw.githubusercontent.com/Seagate/cortx-utils/main/py-utils/python_requirements.ext.txt
cd py-utils/dist
sudo yum install -y cortx-py-utils-*.noarch.rpm

# clone repo
cd /home/cc
git clone https://github.com/Seagate/cortx-hare.git && cd cortx-hare

# install fecter
sudo yum -y install python3 python3-devel yum-utils
sudo yum localinstall -y https://yum.puppetlabs.com/puppet/el/7/x86_64/puppet-agent-7.0.0-1.el7.x86_64.rpm
sudo ln -sf /opt/puppetlabs/bin/facter /usr/bin/facter

# install consul
sudo yum -y install yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install consul-1.9.1

# build and install motr
cd /home/cc/cortx-motr
sudo ./scripts/install-motr-service --link
export M0_SRC_DIR=$PWD

# build hare (2 min, 55 passed, 3 skipped, 36 warnings; 0.5min)
cd /home/cc/cortx-hare
make
sudo make install

# create hare group
sudo groupadd --force hare
sudo usermod --append --groups hare $USER
sudo chown -R cc /var/lib/hare

# add path
PATH=/opt/seagate/cortx/hare/bin:$PATH

Then, can run the test

sudo sh /home/cc/cortx-motr/cas/st/ctgdump_tesh.sh test

Deployment information

Skylake bare metal, CENTOS 7.9, one node. Cortx-motr Aug. 4 commit: 60c43dd74dc3dbe43b090bc488eec1901cf3d76a

Additional information

In addition, ran the all2all test, which mostly succeeded, except the ctgdump in the end:

INFO: Phase recovery:shutdown
INFO: m0d pids: 242545 242995 243447
Stopping m0d@0x7200000000000001:0x2 (ios) at skylake-loop-2.novalocal:4c6c2bd1... 
Stopping m0d@0x7200000000000001:0x3 (ios) at skylake-loop-2.novalocal:4c6c2bd1... 
Stopping m0d@0x7200000000000001:0x4 (ios) at skylake-loop-2.novalocal:4c6c2bd1... 
Stopped m0d@0x7200000000000001:0x3 (ios) at skylake-loop-2.novalocal:4c6c2bd1
Stopped m0d@0x7200000000000001:0x2 (ios) at skylake-loop-2.novalocal:4c6c2bd1
Stopped m0d@0x7200000000000001:0x4 (ios) at skylake-loop-2.novalocal:4c6c2bd1
Stopping m0d@0x7200000000000001:0x1 (confd) at skylake-loop-2.novalocal:4c6c2bd1... 
Stopped m0d@0x7200000000000001:0x1 (confd) at skylake-loop-2.novalocal:4c6c2bd1
Stopping hare-hax at skylake-loop-2.novalocal:4c6c2bd1... 
Stopped hare-hax at skylake-loop-2.novalocal:4c6c2bd1
Making sure that RC leader can be re-elected next time
Stopping hare-consul-agent at skylake-loop-2.novalocal:4c6c2bd1... 
Stopped hare-consul-agent at skylake-loop-2.novalocal:4c6c2bd1
Shutting down RC Leader at skylake-loop-2.novalocal... 
INFO: Dumping /var/motr/m0d-0x7200000000000001:0x2/addb-stobs-242545/o/100000000000000:2 -> /tmp/a2a-addb-out/addb_2.dump ...
OK.
INFO: Dumping /var/motr/m0d-0x7200000000000001:0x3/addb-stobs-242995/o/100000000000000:2 -> /tmp/a2a-addb-out/addb_3.dump ...
OK.
INFO: Dumping /var/motr/m0d-0x7200000000000001:0x4/addb-stobs-243447/o/100000000000000:2 -> /tmp/a2a-addb-out/addb_4.dump ...
OK.
INFO: Dumping /home/cc/cortx-motr/dtm0/it/all2all/addb_244010/o/100000000000000:2 -> /tmp/a2a-addb-out/addb_5.dump ...
OK.
INFO: /home/cc/cortx-motr/cas/m0ctgdump -e libfab:inet:tcp:10.52.2.194@21003 -A linuxstob:addb-stobs -f 0x7200000000000001:0x2 -T ad -S /var/motr/m0d-0x7200000000000001:0x2/stobs -D /var/motr/m0d-0x7200000000000001:0x2/db -m 524288 -q 16 -w 8 -U -r 134217728 -c /etc/motr/confd.xc str 1:5
ERR: ctgdump failed, terminating.

Just a side node: I was able to run example1.c without any problem!

Any suggestions would help!

Thanks in advance!

welcome[bot] commented 2 years ago

Thanks for opening this issue. A contributor should be by to give feedback soon. In the meantime, please check out the contributing guidelines and explore other ways you can get involved.

cortx-admin commented 2 years ago

For the convenience of the Seagate development team, this issue has been mirrored in a private Seagate Jira Server: https://jts.seagate.com/browse/CORTX-33858. Note that community members will not be able to access that Jira server but that is not a problem since all activity in that Jira mirror will be copied into this GitHub issue.

stale[bot] commented 2 years ago

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @nkommuri @mehjoshi @huanghua78 for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

hessio commented 2 years ago

You can this discussion here on Slack: https://cortxcommunity.slack.com/archives/C019S0SGWNQ/p1659478351921799

chandradharraval commented 2 years ago

Hi @madhavemuri, Do we need to enable DTM to run "all2all" test?

cortx-admin commented 2 years ago

Madhav Vemuri commented in Jira Server:

[~530903]: Yes, all2all test is specific to DTM.

cortx-admin commented 2 years ago

Chandradhar Raval commented in Jira Server:

Marking this issue Closed and the same can be tested with DTM enabled custom builds