Seagate / cortx-motr

CORTX Motr is a distributed object and key-value storage system targeting mass capacity storage configurations. It's the core component of CORTX storage system.
https://github.com/Seagate/cortx
Apache License 2.0
59 stars 142 forks source link

improve the disk full testing (73motr-io-small-disks) #2144

Closed huanghua78 closed 7 months ago

huanghua78 commented 2 years ago

Signed-off-by: Hua Huang hua.huang@seagate.com

Problem Statement

Design

Prepare the disk size when generating the conf. This size is passed to Motr and then passed to balloc when doing initialization. So, when disk space is consumed more and more, and finally the balloc will return -ENOSPC error. This error will be passed to upper layer and then to client.

Coding

Checklist for Author

Testing

Checklist for Author

Impact Analysis

Checklist for Author/Reviewer/GateKeeper

Review Checklist

Checklist for Author

Documentation

Checklist for Author

rkothiya commented 2 years ago

Jenkins CI Result : Motr#1759

Motr Test Summary

Test ResultCountInfo
:x:Failed0
:file_folder:

:checkered_flag:Skipped31
:file_folder:

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation
04motr-single-node/34sns-repair-1n-1f
04motr-single-node/08spiel-sns-repair-quiesce
04motr-single-node/28sys-kvs-kernel
04motr-single-node/11m0t1fs-rconfc-fail
04motr-single-node/08spiel-sns-repair
04motr-single-node/19sns-repair-abort
04motr-single-node/22sns-repair-ios-fail
05motr-single-node/18sns-repair-quiesce
05motr-single-node/12fwait
05motr-single-node/16sns-repair-multi
05motr-single-node/07mount-fail
05motr-single-node/15sns-repair-single
05motr-single-node/23sns-abort-quiesce
05motr-single-node/17sns-repair-concurrent-io
05motr-single-node/07mount
05motr-single-node/07mount-multiple
05motr-single-node/12fsync

:heavy_check_mark:Passed171
:file_folder:

01motr-single-node/00userspace-tests_rpc-packet-encdec-ut
01motr-single-node/00userspace-tests_fdmi-sd-ut
01motr-single-node/00userspace-tests_net-prov-ut
01motr-single-node/00userspace-tests_rpc-link-ut
01motr-single-node/00userspace-tests_fom-stats-ut
01motr-single-node/00userspace-tests_fdmi-pd-ut
01motr-single-node/00userspace-tests_rpc-formation-ut
01motr-single-node/00userspace-tests_balloc-ut
01motr-single-node/00userspace-tests_cob-ut
01motr-single-node/00userspace-tests_net-module
01motr-single-node/00userspace-tests_spiel-ci-ut
01motr-single-node/00userspace-tests_io-nw-xfer-ut
01motr-single-node/00userspace-tests_rpc-session-ut
01motr-single-node/00userspace-tests_spiel-conf-ut
01motr-single-node/00userspace-tests_addb2-storage
01motr-single-node/00userspace-tests_rm-ut
01motr-single-node/00userspace-tests_helpers-ufid-ut
01motr-single-node/00userspace-tests_io-req-ut
01motr-single-node/00userspace-tests_buffer_pool_ut
01motr-single-node/00userspace-tests_be-ut
01motr-single-node/43m0crate
01motr-single-node/00userspace-tests_dix-cm-iter
01motr-single-node/00userspace-tests_fis-ut
01motr-single-node/00userspace-tests_rpc-conn-pool-ut
01motr-single-node/00userspace-tests_idx-ut
01motr-single-node/00userspace-tests_ff2c-ut
01motr-single-node/05confgen
01motr-single-node/00userspace-tests_conf-ut
01motr-single-node/00userspace-tests_cob-foms-ut
01motr-single-node/00userspace-tests_addb2-consumer
01motr-single-node/00userspace-tests_capa-ut
01motr-single-node/00userspace-tests_isc-api-ut
01motr-single-node/00userspace-tests_snscm_storage-ut
01motr-single-node/00userspace-tests_net-bulk-mem
01motr-single-node/00userspace-tests_obj-ut
01motr-single-node/00userspace-tests_confc-ut
01motr-single-node/00userspace-tests_reqh-fop-allow-ut
01motr-single-node/00userspace-tests_mdservice-ut
01motr-single-node/00userspace-tests_reqh-ut
01motr-single-node/00userspace-tests_ios-bufferpool-ut
01motr-single-node/00userspace-tests_io-ut
01motr-single-node/00userspace-tests_rpc-rcv-session-ut
01motr-single-node/00userspace-tests_dtm-dtx-ut
01motr-single-node/00userspace-tests_idx-dix
01motr-single-node/00userspace-tests_io-req-fop-ut
01motr-single-node/00userspace-tests_net-bulk-if
01motr-single-node/00userspace-tests_net-misc-ut
01motr-single-node/00userspace-tests_parity_math_ssse3-ut
01motr-single-node/00userspace-tests_addb2-net
01motr-single-node/00userspace-tests_failure_domains_tree-ut
01motr-single-node/00userspace-tests_fdmi-filterc-ut
01motr-single-node/00userspace-tests_cm-cp-ut
01motr-single-node/00userspace-tests_conf-pvers-ut
01motr-single-node/00userspace-tests_addb2-base
01motr-single-node/00userspace-tests_rpc-at
01motr-single-node/00userspace-tests_dtm0-ut
01motr-single-node/00userspace-tests_parity_math-ut
01motr-single-node/00userspace-tests_net-test
01motr-single-node/00userspace-tests_spiel-ut
01motr-single-node/00userspace-tests_stats-ut
01motr-single-node/00userspace-tests_addb2-sys
01motr-single-node/00userspace-tests_fdmi-filter-eval-ut
01motr-single-node/00userspace-tests_snscm_xform-ut
01motr-single-node/00userspace-tests_ms-fom-ut
01motr-single-node/00userspace-tests_fop-lock-ut
01motr-single-node/00userspace-tests_dtm-nucleus-ut
01motr-single-node/00userspace-tests_xcode_bufvec_fop-ut
01motr-single-node/00userspace-tests_conf-diter-ut
01motr-single-node/00userspace-tests_libconsole-ut
01motr-single-node/00userspace-tests_sm-ut
01motr-single-node/00userspace-tests_di-ut
01motr-single-node/00userspace-tests_rpc-item-source-ut
01motr-single-node/00userspace-tests_addb2-histogram
01motr-single-node/00userspace-tests_isc-service-ut
01motr-single-node/00userspace-tests_sss-ut
01motr-single-node/00userspace-tests_dix-client-ut
01motr-single-node/00userspace-tests_ha-state-ut
01motr-single-node/00userspace-tests_poolmach-ut
01motr-single-node/00userspace-tests_dtm0-log-ut
01motr-single-node/06hagen
01motr-single-node/00userspace-tests_rpc-machine-ut
01motr-single-node/00userspace-tests_rm-rcredits-ut
01motr-single-node/00userspace-tests_snscm_net-ut
01motr-single-node/00userspace-tests_bytecount-ut
01motr-single-node/00userspace-tests_rconfc-ut
01motr-single-node/00userspace-tests_stob-ut
01motr-single-node/00userspace-tests_reqh-service-ut
01motr-single-node/00userspace-tests_rpc-lib-ut
01motr-single-node/00userspace-tests_failure_domains-ut
01motr-single-node/00userspace-tests_sync-ut
01motr-single-node/00userspace-tests_pi_ut
01motr-single-node/00userspace-tests_net-lnet
01motr-single-node/00userspace-tests_udb-ut
01motr-single-node/00userspace-tests_layout-ut
01motr-single-node/04initscripts
01motr-single-node/00userspace-tests_rpc-connection-ut
01motr-single-node/00userspace-tests_io-pargrp-ut
01motr-single-node/00userspace-tests_rpc-item-ut
01motr-single-node/00userspace-tests_fdmi-fol-fini-ut
01motr-single-node/00userspace-tests_confstr-ut
01motr-single-node/00userspace-tests_module-ut
01motr-single-node/00userspace-tests_cm-ut
01motr-single-node/00userspace-tests_sns-file-lock-ut
01motr-single-node/00userspace-tests_rm-rwlock-ut
01motr-single-node/00userspace-tests_conf-validation-ut
01motr-single-node/00userspace-tests_m0d-ut
01motr-single-node/00userspace-tests_dtm-transmit-ut
01motr-single-node/00userspace-tests_idx-dix-mt
01motr-single-node/00userspace-tests_xcode-ut
01motr-single-node/00userspace-tests_cas-client
01motr-single-node/00userspace-tests_bulk-server-ut
01motr-single-node/00userspace-tests_ha-ut
01motr-single-node/00userspace-tests_conf-walk-ut
01motr-single-node/00userspace-tests_client-ut
01motr-single-node/00userspace-tests_conf-load-ut
01motr-single-node/00userspace-tests_sns-cm-repair-ut
01motr-single-node/00userspace-tests_libm0-ut
01motr-single-node/00userspace-tests_btree-ut
01motr-single-node/00userspace-tests_reqh-service-ctx-ut
01motr-single-node/00userspace-tests_fom-timedwait-ut
01motr-single-node/52motr-singlenode-sanity
01motr-single-node/00userspace-tests_fdmi-fol-ut
01motr-single-node/00userspace-tests_conf-glob-ut
01motr-single-node/00userspace-tests_storage-dev-ut
01motr-single-node/00userspace-tests_cas-service
01motr-single-node/01net
01motr-single-node/00userspace-tests_bulk-client-ut
01motr-single-node/00userspace-tests_layout-access-plan-ut
01motr-single-node/01kernel-tests
01motr-single-node/03console
01motr-single-node/00userspace-tests_dtm0-clk-src-ut
01motr-single-node/02rpcping
01motr-single-node/00userspace-tests_fol-ut
01motr-single-node/00userspace-tests_fit-ut
01motr-single-node/00userspace-tests_libfab-ut
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/72spiel-sns-motr-repair-quiesce
03motr-single-node/08spiel-multi-confd
03motr-single-node/69sns-repair-motr-quiesce
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/70sns-failure-after-repair-quiesce
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/71spiel-sns-motr-repair
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
04motr-single-node/73motr-io-small-disks
04motr-single-node/48motr-raid0-io
04motr-single-node/74motr-di-corruption-detection
04motr-single-node/49motr-rpc-cancel
04motr-single-node/25m0kv
04motr-single-node/44motr-rm-lock-cc-io
04motr-single-node/45motr-rmw
05motr-single-node/23dix-repair-m0repair
05motr-single-node/43motr-sync-replication
05motr-single-node/42motr-utils
05motr-single-node/45motr-sns-repair-N-1
05motr-single-node/40motr-dgmode
05motr-single-node/23dix-repair-quiesce-m0repair
05motr-single-node/23spiel-dix-repair-quiesce
05motr-single-node/44motr-sns-repair
05motr-single-node/23spiel-dix-repair

Total202:link:

CppCheck Summary

   Cppcheck: No new warnings found :thumbsup:

rkothiya commented 2 years ago

Jenkins CI Result : Motr#1763

Motr Test Summary

Test ResultCountInfo
:x:Failed1
:file_folder:

01motr-single-node/00userspace-tests_spiel-ci-ut

:checkered_flag:Skipped31
:file_folder:

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation
04motr-single-node/34sns-repair-1n-1f
04motr-single-node/08spiel-sns-repair-quiesce
04motr-single-node/28sys-kvs-kernel
04motr-single-node/11m0t1fs-rconfc-fail
04motr-single-node/08spiel-sns-repair
04motr-single-node/19sns-repair-abort
04motr-single-node/22sns-repair-ios-fail
05motr-single-node/18sns-repair-quiesce
05motr-single-node/12fwait
05motr-single-node/16sns-repair-multi
05motr-single-node/07mount-fail
05motr-single-node/15sns-repair-single
05motr-single-node/23sns-abort-quiesce
05motr-single-node/17sns-repair-concurrent-io
05motr-single-node/07mount
05motr-single-node/07mount-multiple
05motr-single-node/12fsync

:heavy_check_mark:Passed170
:file_folder:

01motr-single-node/00userspace-tests_rpc-packet-encdec-ut
01motr-single-node/00userspace-tests_fdmi-sd-ut
01motr-single-node/00userspace-tests_net-prov-ut
01motr-single-node/00userspace-tests_rpc-link-ut
01motr-single-node/00userspace-tests_fom-stats-ut
01motr-single-node/00userspace-tests_fdmi-pd-ut
01motr-single-node/00userspace-tests_rpc-formation-ut
01motr-single-node/00userspace-tests_balloc-ut
01motr-single-node/00userspace-tests_cob-ut
01motr-single-node/00userspace-tests_net-module
01motr-single-node/00userspace-tests_io-nw-xfer-ut
01motr-single-node/00userspace-tests_rpc-session-ut
01motr-single-node/00userspace-tests_spiel-conf-ut
01motr-single-node/00userspace-tests_addb2-storage
01motr-single-node/00userspace-tests_rm-ut
01motr-single-node/00userspace-tests_helpers-ufid-ut
01motr-single-node/00userspace-tests_io-req-ut
01motr-single-node/00userspace-tests_buffer_pool_ut
01motr-single-node/00userspace-tests_be-ut
01motr-single-node/43m0crate
01motr-single-node/00userspace-tests_dix-cm-iter
01motr-single-node/00userspace-tests_fis-ut
01motr-single-node/00userspace-tests_rpc-conn-pool-ut
01motr-single-node/00userspace-tests_idx-ut
01motr-single-node/00userspace-tests_ff2c-ut
01motr-single-node/05confgen
01motr-single-node/00userspace-tests_conf-ut
01motr-single-node/00userspace-tests_cob-foms-ut
01motr-single-node/00userspace-tests_addb2-consumer
01motr-single-node/00userspace-tests_capa-ut
01motr-single-node/00userspace-tests_isc-api-ut
01motr-single-node/00userspace-tests_snscm_storage-ut
01motr-single-node/00userspace-tests_net-bulk-mem
01motr-single-node/00userspace-tests_obj-ut
01motr-single-node/00userspace-tests_confc-ut
01motr-single-node/00userspace-tests_reqh-fop-allow-ut
01motr-single-node/00userspace-tests_mdservice-ut
01motr-single-node/00userspace-tests_reqh-ut
01motr-single-node/00userspace-tests_ios-bufferpool-ut
01motr-single-node/00userspace-tests_io-ut
01motr-single-node/00userspace-tests_rpc-rcv-session-ut
01motr-single-node/00userspace-tests_dtm-dtx-ut
01motr-single-node/00userspace-tests_idx-dix
01motr-single-node/00userspace-tests_io-req-fop-ut
01motr-single-node/00userspace-tests_net-bulk-if
01motr-single-node/00userspace-tests_net-misc-ut
01motr-single-node/00userspace-tests_parity_math_ssse3-ut
01motr-single-node/00userspace-tests_addb2-net
01motr-single-node/00userspace-tests_failure_domains_tree-ut
01motr-single-node/00userspace-tests_fdmi-filterc-ut
01motr-single-node/00userspace-tests_cm-cp-ut
01motr-single-node/00userspace-tests_conf-pvers-ut
01motr-single-node/00userspace-tests_addb2-base
01motr-single-node/00userspace-tests_rpc-at
01motr-single-node/00userspace-tests_dtm0-ut
01motr-single-node/00userspace-tests_parity_math-ut
01motr-single-node/00userspace-tests_net-test
01motr-single-node/00userspace-tests_spiel-ut
01motr-single-node/00userspace-tests_stats-ut
01motr-single-node/00userspace-tests_addb2-sys
01motr-single-node/00userspace-tests_fdmi-filter-eval-ut
01motr-single-node/00userspace-tests_snscm_xform-ut
01motr-single-node/00userspace-tests_ms-fom-ut
01motr-single-node/00userspace-tests_fop-lock-ut
01motr-single-node/00userspace-tests_dtm-nucleus-ut
01motr-single-node/00userspace-tests_xcode_bufvec_fop-ut
01motr-single-node/00userspace-tests_conf-diter-ut
01motr-single-node/00userspace-tests_libconsole-ut
01motr-single-node/00userspace-tests_sm-ut
01motr-single-node/00userspace-tests_di-ut
01motr-single-node/00userspace-tests_rpc-item-source-ut
01motr-single-node/00userspace-tests_addb2-histogram
01motr-single-node/00userspace-tests_isc-service-ut
01motr-single-node/00userspace-tests_sss-ut
01motr-single-node/00userspace-tests_dix-client-ut
01motr-single-node/00userspace-tests_ha-state-ut
01motr-single-node/00userspace-tests_poolmach-ut
01motr-single-node/00userspace-tests_dtm0-log-ut
01motr-single-node/06hagen
01motr-single-node/00userspace-tests_rpc-machine-ut
01motr-single-node/00userspace-tests_rm-rcredits-ut
01motr-single-node/00userspace-tests_snscm_net-ut
01motr-single-node/00userspace-tests_bytecount-ut
01motr-single-node/00userspace-tests_rconfc-ut
01motr-single-node/00userspace-tests_stob-ut
01motr-single-node/00userspace-tests_reqh-service-ut
01motr-single-node/00userspace-tests_rpc-lib-ut
01motr-single-node/00userspace-tests_failure_domains-ut
01motr-single-node/00userspace-tests_sync-ut
01motr-single-node/00userspace-tests_pi_ut
01motr-single-node/00userspace-tests_net-lnet
01motr-single-node/00userspace-tests_udb-ut
01motr-single-node/00userspace-tests_layout-ut
01motr-single-node/04initscripts
01motr-single-node/00userspace-tests_rpc-connection-ut
01motr-single-node/00userspace-tests_io-pargrp-ut
01motr-single-node/00userspace-tests_rpc-item-ut
01motr-single-node/00userspace-tests_fdmi-fol-fini-ut
01motr-single-node/00userspace-tests_confstr-ut
01motr-single-node/00userspace-tests_module-ut
01motr-single-node/00userspace-tests_cm-ut
01motr-single-node/00userspace-tests_sns-file-lock-ut
01motr-single-node/00userspace-tests_rm-rwlock-ut
01motr-single-node/00userspace-tests_conf-validation-ut
01motr-single-node/00userspace-tests_m0d-ut
01motr-single-node/00userspace-tests_dtm-transmit-ut
01motr-single-node/00userspace-tests_idx-dix-mt
01motr-single-node/00userspace-tests_xcode-ut
01motr-single-node/00userspace-tests_cas-client
01motr-single-node/00userspace-tests_bulk-server-ut
01motr-single-node/00userspace-tests_ha-ut
01motr-single-node/00userspace-tests_conf-walk-ut
01motr-single-node/00userspace-tests_client-ut
01motr-single-node/00userspace-tests_conf-load-ut
01motr-single-node/00userspace-tests_sns-cm-repair-ut
01motr-single-node/00userspace-tests_libm0-ut
01motr-single-node/00userspace-tests_btree-ut
01motr-single-node/00userspace-tests_reqh-service-ctx-ut
01motr-single-node/00userspace-tests_fom-timedwait-ut
01motr-single-node/52motr-singlenode-sanity
01motr-single-node/00userspace-tests_fdmi-fol-ut
01motr-single-node/00userspace-tests_conf-glob-ut
01motr-single-node/00userspace-tests_storage-dev-ut
01motr-single-node/00userspace-tests_cas-service
01motr-single-node/01net
01motr-single-node/00userspace-tests_bulk-client-ut
01motr-single-node/00userspace-tests_layout-access-plan-ut
01motr-single-node/01kernel-tests
01motr-single-node/03console
01motr-single-node/00userspace-tests_dtm0-clk-src-ut
01motr-single-node/02rpcping
01motr-single-node/00userspace-tests_fol-ut
01motr-single-node/00userspace-tests_fit-ut
01motr-single-node/00userspace-tests_libfab-ut
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/72spiel-sns-motr-repair-quiesce
03motr-single-node/08spiel-multi-confd
03motr-single-node/69sns-repair-motr-quiesce
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/70sns-failure-after-repair-quiesce
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/71spiel-sns-motr-repair
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
04motr-single-node/73motr-io-small-disks
04motr-single-node/48motr-raid0-io
04motr-single-node/74motr-di-corruption-detection
04motr-single-node/49motr-rpc-cancel
04motr-single-node/25m0kv
04motr-single-node/44motr-rm-lock-cc-io
04motr-single-node/45motr-rmw
05motr-single-node/23dix-repair-m0repair
05motr-single-node/43motr-sync-replication
05motr-single-node/42motr-utils
05motr-single-node/45motr-sns-repair-N-1
05motr-single-node/40motr-dgmode
05motr-single-node/23dix-repair-quiesce-m0repair
05motr-single-node/23spiel-dix-repair-quiesce
05motr-single-node/44motr-sns-repair
05motr-single-node/23spiel-dix-repair

Total202:link:

CppCheck Summary

   Cppcheck: No new warnings found :thumbsup:

huanghua78 commented 2 years ago

Jenkins CI Result : Motr#1763

Motr Test Summary

Test Result Count Info ❌Failed 1
📁 🏁Skipped 31
📁 ✔️Passed 170 📁 Total 202 🔗

CppCheck Summary

   Cppcheck: No new warnings found 👍

[520428@ssc-vm-g3-rhev4-2904 cortx-motr]$ sudo ./scripts/m0 run-ut -t spiel-ci-ut
false
----- run_ut -t spiel-ci-ut -----
opening /dev/lnet failed: No such file or directory
hint: the kernel modules may not be loaded
IOC_LIBCFS_GET_NI error 2: No such file or directory
START Iteration: 1 out of 1
spiel-ci-ut
  service-cmds                                    2.95 sec  193 MiB
  process-cmds                                    2.52 sec  116 MiB
  service-order                                   2.48 sec  116 MiB
  process-services-list                           2.41 sec  115 MiB
  device-cmds                                     3.72 sec  120 MiB
  stats                                           6.86 sec  177 MiB
  bytecount-stats                                 2.41 sec  115 MiB
  pool-sns-repair                                 2.93 sec  122 MiB
  pool-sns-rebalance                              2.61 sec  119 MiB
  pool-dix-repair                                 2.52 sec  118 MiB
  pool-dix-rebalance                              2.45 sec  116 MiB
  [ time: 33.92 sec, mem: 1 GiB, leaked: 14 MiB ]

Time: 33.93 sec, Mem: 1 GiB, Leaked: 14 MiB, Asserts: 294
Unit tests status: SUCCESS
END   Iteration: 1 out of 1

utime 7.626437 stime 1.767240 maxrss 620272 nvcsw 48820 nivcsw 10684
minflt 132035 majflt 7 inblock 1472 oublock 383992
rchar 4556176 wchar 12692 syscr 12990 syscw 10864
read_bytes 753664 write_bytes 196603904 cancelled_write_bytes 0
huanghua78 commented 2 years ago
[520428@ssc-vm-g3-rhev4-2904 cortx-motr]$ sudo ./scripts/m0 run-ut -t spiel-ci-ut
[sudo] password for 520428:
false
----- run_ut -t spiel-ci-ut -----
opening /dev/lnet failed: No such file or directory
hint: the kernel modules may not be loaded
IOC_LIBCFS_GET_NI error 2: No such file or directory
START Iteration: 1 out of 1
spiel-ci-ut
  service-cmds                                    2.90 sec  193 MiB
  process-cmds                                    2.46 sec  116 MiB
  service-order                                   2.42 sec  116 MiB
  process-services-list                           2.42 sec  115 MiB
  device-cmds                                     3.70 sec  120 MiB
  stats                                           6.80 sec  177 MiB
  bytecount-stats                                 2.41 sec  115 MiB
  pool-sns-repair                                 2.87 sec  122 MiB
  pool-sns-rebalance                              2.70 sec  119 MiB
  pool-dix-repair                                 2.51 sec  118 MiB
  pool-dix-rebalance                              2.46 sec  116 MiB
  [ time: 33.72 sec, mem: 1 GiB, leaked: 14 MiB ]

Time: 33.73 sec, Mem: 1 GiB, Leaked: 14 MiB, Asserts: 290
Unit tests status: SUCCESS
END   Iteration: 1 out of 1

utime 7.591633 stime 1.628395 maxrss 610016 nvcsw 48369 nivcsw 10282
minflt 123060 majflt 3 inblock 800 oublock 351056
rchar 4554131 wchar 12799 syscr 13102 syscw 10971
read_bytes 409600 write_bytes 179740672 cancelled_write_bytes 16384
huanghua78 commented 2 years ago

The output of the improved test looks like this:

https://eos-jenkins.colo.seagate.com/job/Cortx-PR-Build/job/Motr/1763/testReport/04motr-single-node/73motr-io-small-disks/

https://eos-jenkins.colo.seagate.com/job/Cortx-PR-Build/job/Motr/1763/testReport/04motr-single-node/73motr-io-small-disks/attachments/73motr-io-small-disks.stdout.log

true
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(2,1,1,4) nr_ios=4 multiple_pools=0

[112:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
      [1: "4 2 1 1"],
      [1: ^n|1:2],
      [1: ^S|1:6],
      [3: ^o|1:9, ^o|20:1, ^o|2:9],
      [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.230.247.3@tcp:12345:34:101", [2: ^s|1:102, ^s|1:101])}, 
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.230.247.3@tcp:12345:33:900", [9: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0, ^s|18:0,                      ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])}, 
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.230.247.3@tcp:12345:33:901", [9: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1, ^s|18:1,                      ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])}, 
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.230.247.3@tcp:12345:33:902", [9: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2, ^s|18:2,                      ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])}, 
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.230.247.3@tcp:12345:33:903", [9: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3, ^s|18:3,                      ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])}, 
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.230.247.3@tcp:12345:33:800", [4: ^s|2:0, ^s|12:0, ^s|3:4, ^s|18:4])}, 
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.230.247.3@tcp:12345:34:1",
                          [4: ^s|1:6, ^s|1:7, ^s|18:6, ^s|3:6])}, 
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.230.247.3@tcp:12345:33:100",
                          [3: ^s|8:0, ^s|18:7, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.230.247.3@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.230.247.3@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.230.247.3@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.230.247.3@tcp:12345:34:101"], [0], [0])},
  {0x73| ((^s|1:102), @M0_CST_DTM0, [1: "10.230.247.3@tcp:12345:34:101"], [1: "origin:in-volatile"], [0])},

 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.230.247.3@tcp:12345:33:800"], [0], [0])}, 
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.230.247.3@tcp:12345:33:800"], [0], [0])}, 
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.230.247.3@tcp:12345:33:800"], [0], [0])}, 
 {0x73| ((^s|18:4), @M0_CST_DTM0, [1: "10.230.247.3@tcp:12345:33:800"], [1: "origin:in-volatile"], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.230.247.3@tcp:12345:33:900"], [0], [1: ^d|1:1])}, 
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.230.247.3@tcp:12345:33:900"], [0], [0])}, 
 {0x73| ((^s|18:0), @M0_CST_DTM0, [1: "10.230.247.3@tcp:12345:33:900"], [1: "origin:in-volatile"], [0])},                  
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.230.247.3@tcp:12345:33:900"], [0], [0])}, 
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.230.247.3@tcp:12345:33:900"], [0], [0])}, 
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.230.247.3@tcp:12345:33:900"], [0], [0])}, 
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.230.247.3@tcp:12345:33:900"], [0], [1: ^d|20:0])}, 
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.230.247.3@tcp:12345:33:900"], [0], [0])}, 
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.230.247.3@tcp:12345:33:900"], [0], [0])}, 
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.230.247.3@tcp:12345:33:901"], [0], [1: ^d|1:2])}, 
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.230.247.3@tcp:12345:33:901"], [0], [0])}, 
 {0x73| ((^s|18:1), @M0_CST_DTM0, [1: "10.230.247.3@tcp:12345:33:901"], [1: "origin:in-volatile"], [0])},                  
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.230.247.3@tcp:12345:33:901"], [0], [0])}, 
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.230.247.3@tcp:12345:33:901"], [0], [0])}, 
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.230.247.3@tcp:12345:33:901"], [0], [0])}, 
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.230.247.3@tcp:12345:33:901"], [0], [1: ^d|20:1])}, 
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.230.247.3@tcp:12345:33:901"], [0], [0])}, 
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.230.247.3@tcp:12345:33:901"], [0], [0])}, 
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.230.247.3@tcp:12345:33:902"], [0], [1: ^d|1:3])}, 
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.230.247.3@tcp:12345:33:902"], [0], [0])}, 
 {0x73| ((^s|18:2), @M0_CST_DTM0, [1: "10.230.247.3@tcp:12345:33:902"], [1: "origin:in-volatile"], [0])},                  
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.230.247.3@tcp:12345:33:902"], [0], [0])}, 
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.230.247.3@tcp:12345:33:902"], [0], [0])}, 
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.230.247.3@tcp:12345:33:902"], [0], [0])}, 
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.230.247.3@tcp:12345:33:902"], [0], [1: ^d|20:2])}, 
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.230.247.3@tcp:12345:33:902"], [0], [0])}, 
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.230.247.3@tcp:12345:33:902"], [0], [0])}, 
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.230.247.3@tcp:12345:33:903"], [0], [1: ^d|1:4])}, 
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.230.247.3@tcp:12345:33:903"], [0], [0])}, 
 {0x73| ((^s|18:3), @M0_CST_DTM0, [1: "10.230.247.3@tcp:12345:33:903"], [1: "origin:in-volatile"], [0])},                  
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.230.247.3@tcp:12345:33:903"], [0], [0])}, 
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.230.247.3@tcp:12345:33:903"], [0], [0])}, 
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.230.247.3@tcp:12345:33:903"], [0], [0])}, 
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.230.247.3@tcp:12345:33:903"], [0], [1: ^d|20:3])}, 
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.230.247.3@tcp:12345:33:903"], [0], [0])}, 
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.230.247.3@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.230.247.3@tcp:12345:34:1"], [0], [0])}, 
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.230.247.3@tcp:12345:33:100"], [0], [0])},

 {0x73| ((^s|18:6), @M0_CST_DTM0, [1: "10.230.247.3@tcp:12345:34:1"], [1: "origin:in-volatile"], [0])}, 
 {0x73| ((^s|18:7), @M0_CST_DTM0, [1: "10.230.247.3@tcp:12345:33:100"], [1: "origin:in-volatile"], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 1074790400, 3, 4, "/dev/loop1")}, 
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])}, 
 {0x6a| ((^j|1:1), ^k|1:1, [0])}, 
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 1074790400, 3, 4, "/dev/loop2")}, 
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])}, 
 {0x6a| ((^j|1:2), ^k|1:2, [0])}, 
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 1074790400, 3, 4, "/dev/loop3")}, 
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])}, 
 {0x6a| ((^j|1:3), ^k|1:3, [0])}, 
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 1074790400, 3, 4, "/dev/loop4")}, 
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])}, 
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [8: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (2, 1,
                                            1, 4,
                                            [5: 0, 0, 0, 0, 1],
                                            [1: ^j|1:5])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:5), ^S|1:6, [1: ^j|1:6])},
  {0x6a| ((^j|1:6), ^a|1:6, [1: ^j|1:7])},
  {0x6a| ((^j|1:7), ^e|1:7, [1: ^j|1:8])},
  {0x6a| ((^j|1:8), ^c|1:8, [4: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4])} , 
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])}, 
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:5])})}, 
{0x6a| ((^j|2:5), ^S|1:6, [1: ^j|2:6])}, 
{0x6a| ((^j|2:6), ^a|1:6, [1: ^j|2:7])}, 
{0x6a| ((^j|2:7), ^e|1:7, [1: ^j|2:8])}, 
{0x6a| ((^j|2:8), ^c|1:8, [4: ^j|2:1, ^j|2:2, ^j|2:3, ^j|2:4])}, 
{0x6a| ((^j|2:1), ^k|1:1, [0])}, 
 {0x6a| ((^j|2:2), ^k|1:2, [0])}, 
 {0x6a| ((^j|2:3), ^k|1:3, [0])}, 
 {0x6a| ((^j|2:4), ^k|1:4, [0])}  , 
{0x64| ((^d|20:0), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")}, 
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])}, 
{0x6a| ((^j|20:100), ^k|20:0, [0])}, 
{0x64| ((^d|20:1), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")}, 
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])}, 
{0x6a| ((^j|20:101), ^k|20:1, [0])}, 
{0x64| ((^d|20:2), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")}, 
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])}, 
{0x6a| ((^j|20:102), ^k|20:2, [0])}, 
{0x64| ((^d|20:3), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")}, 
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])}, 
{0x6a| ((^j|20:103), ^k|20:3, [0])}, 
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])}, 
{0x76| ((^v|1:20), {0| (1, 1, 1,
                                                    4,
                                                    [5: 0, 0, 0, 0, 1],
                                                    [1: ^j|20:1])})}, 
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])}, 
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])}, 
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])}, 
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/root/sandbox.st-73motr-io-small-disks/confd && exec /root/motr/motr_test_github_workdir/workdir/src/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                 -w 4 -m 65536               -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.247.3@tcp:12345:35:1           -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0mkfs.log
motr[147066]:  9a00  ERROR  [dtm0/log.c:136:m0_dtm0_log_open]  <! rc=-2
cd /var/motr/root/sandbox.st-73motr-io-small-disks/confd && exec /root/motr/motr_test_github_workdir/workdir/src/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.230.247.3@tcp:12345:33:100 -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0d.log
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ha && exec /root/motr/motr_test_github_workdir/workdir/src/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.247.3@tcp:12345:35:1 -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0mkfs.log
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[147279]:  e5e0  ERROR  [dtm0/log.c:136:m0_dtm0_log_open]  <! rc=-2
cd /var/motr/root/sandbox.st-73motr-io-small-disks/mds1 && exec             /root/motr/motr_test_github_workdir/workdir/src/utils/mkfs/m0mkfs -F -T ad          -D db -S stobs -A linuxstob:addb-stobs               -w 4 -m 65536               -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:35:800            -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0mkfs.log
motr[147696]:  6be0   WARN  [stob/ad.c:285:stob_ad_domain_cfg_create_parse]  The default container size 134217728000 is used for this device, but it may differ from the actual size.
motr[147696]:  6220  ERROR  [dtm0/log.c:136:m0_dtm0_log_open]  <! rc=-2
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ios1 && exec /root/motr/motr_test_github_workdir/workdir/src/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0mkfs.log
motr[147904]:  2fd0   WARN  [stob/ad.c:285:stob_ad_domain_cfg_create_parse]  The default container size 134217728000 is used for this device, but it may differ from the actual size.
motr[147904]:  2610  ERROR  [dtm0/log.c:136:m0_dtm0_log_open]  <! rc=-2
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ios2 && exec /root/motr/motr_test_github_workdir/workdir/src/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0mkfs.log
motr[148123]:  6cf0   WARN  [stob/ad.c:285:stob_ad_domain_cfg_create_parse]  The default container size 134217728000 is used for this device, but it may differ from the actual size.
motr[148123]:  6330  ERROR  [dtm0/log.c:136:m0_dtm0_log_open]  <! rc=-2
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ios3 && exec /root/motr/motr_test_github_workdir/workdir/src/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0mkfs.log
motr[148332]:  ef50   WARN  [stob/ad.c:285:stob_ad_domain_cfg_create_parse]  The default container size 134217728000 is used for this device, but it may differ from the actual size.
motr[148332]:  e590  ERROR  [dtm0/log.c:136:m0_dtm0_log_open]  <! rc=-2
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ios4 && exec /root/motr/motr_test_github_workdir/workdir/src/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0mkfs.log
motr[148555]:   450   WARN  [stob/ad.c:285:stob_ad_domain_cfg_create_parse]  The default container size 134217728000 is used for this device, but it may differ from the actual size.
motr[148555]:  fa90  ERROR  [dtm0/log.c:136:m0_dtm0_log_open]  <! rc=-2
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ha && exec /root/motr/motr_test_github_workdir/workdir/src/motr/m0d -D db -S stobs -A linuxstob:addb-stobs                -w 4 -m 65536               -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.247.3@tcp:12345:34:1               -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.230.247.3@tcp:12345:34:1 |& tee -a m0d.log
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/root/sandbox.st-73motr-io-small-disks/mds1 && exec /root/motr/motr_test_github_workdir/workdir/src/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.230.247.3@tcp:12345:34:1 -c /var/motr/root/sandbox.st-73motr-io-small-disks/confd/conf.xc |& tee -a m0d.log
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ios1 && exec /root/motr/motr_test_github_workdir/workdir/src/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.230.247.3@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ios2 && exec /root/motr/motr_test_github_workdir/workdir/src/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.230.247.3@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ios3 && exec /root/motr/motr_test_github_workdir/workdir/src/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.230.247.3@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/root/sandbox.st-73motr-io-small-disks/ios4 && exec /root/motr/motr_test_github_workdir/workdir/src/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.247.3@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.230.247.3@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
Startin motr_io_small_disks testing ...
creating object 100000:100000 bs=1048576 * c=128
creating object 100000:100001 bs=1048576 * c=128
creating object 100000:100002 bs=1048576 * c=128
creating object 100000:100003 bs=1048576 * c=128
creating object 100000:100004 bs=1048576 * c=128
creating object 100000:100005 bs=1048576 * c=128
creating object 100000:100006 bs=1048576 * c=128
creating object 100000:100007 bs=1048576 * c=128
creating object 100000:100008 bs=1048576 * c=128
creating object 100000:100009 bs=1048576 * c=128
creating object 100000:1000010 bs=1048576 * c=128
creating object 100000:1000011 bs=1048576 * c=128
creating object 100000:1000012 bs=1048576 * c=128
creating object 100000:1000013 bs=1048576 * c=128
creating object 100000:1000014 bs=1048576 * c=128
creating object 100000:1000015 bs=1048576 * c=128
creating object 100000:1000016 bs=1048576 * c=128
creating object 100000:1000017 bs=1048576 * c=128
creating object 100000:1000018 bs=1048576 * c=128
creating object 100000:1000019 bs=1048576 * c=128
creating object 100000:1000020 bs=1048576 * c=128
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c070f88 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c04bff8 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c057ee8 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c058a08 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c044808 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c04e5a8 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c04d708 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c049a08 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c0442c8 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c04d9c8 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c04c2d8 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c070f88 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
motr[149209]:  64b0  ERROR  [balloc/balloc.c:3029:balloc_alloc]  <! rc=-28
motr[149209]:  7700  ERROR  [stob/io.c:183:m0_stob_io_prepare]  io=0x7f434c04bff8 <200000300100000:1000020> FAILED rc=-28
motr[149209]:  77f0  ERROR  [ioservice/io_foms.c:1842:io_launch]  stob_io_launch failed: rc=-28
Writing object 100000:1000020 failed: 28
We got 28 as expected
stopping /root/motr/motr_test_github_workdir/workdir/src/motr/m0d processes...
=== pids of services: 149209 149204 149198 149194 148984 148770 147275 ===
Shutting down services one by one. mdservice is the last.
----- 149209 stopping--------lt-m0d: got signal 1
motr[149209]:  1580  ERROR  [conf/rconfc.c:1200:rconfc_fail]  rconfc: 0x7ffd13849058, state M0_RCS_IDLE failed with -22
----- 149209 stopped --------
----- 149204 stopping--------lt-m0d: got signal 1
motr[149204]:  bdd0  ERROR  [conf/rconfc.c:1200:rconfc_fail]  rconfc: 0x7ffda79c38a8, state M0_RCS_IDLE failed with -22
----- 149204 stopped --------
----- 149198 stopping--------lt-m0d: got signal 1
motr[149198]:  8d80  ERROR  [conf/rconfc.c:1200:rconfc_fail]  rconfc: 0x7ffe456b0858, state M0_RCS_IDLE failed with -22
----- 149198 stopped --------
----- 149194 stopping--------lt-m0d: got signal 1
motr[149194]:  f6c0  ERROR  [conf/rconfc.c:1200:rconfc_fail]  rconfc: 0x7fff26e87198, state M0_RCS_IDLE failed with -22
----- 149194 stopped --------
----- 148984 stopping--------lt-m0d: got signal 1
----- 148984 stopped --------
----- 148770 stopping--------lt-m0d: got signal 1
motr[148770]:  c960  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc0c404a020 dest_ep=10.230.247.3@tcp:12345:33:903 ri_session=0x2f1a1a0 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[148770]:  c960  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc0c405e8c0, item 0x7fc0c404a020[38] failed with ri_error=-110
motr[148770]:  c960  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc0c4056fb0 dest_ep=10.230.247.3@tcp:12345:33:902 ri_session=0x2e77380 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[148770]:  c960  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc0c4044ae0, item 0x7fc0c4056fb0[38] failed with ri_error=-110
motr[148770]:  c960  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc0c4058020 dest_ep=10.230.247.3@tcp:12345:33:902 ri_session=0x2eef120 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[148770]:  c960  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc0c405ef20, item 0x7fc0c4058020[38] failed with ri_error=-110
motr[148770]:  c960  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7fc0c4052e40 dest_ep=10.230.247.3@tcp:12345:33:902 ri_session=0x2f04480 ri_nr_sent_max=1 ri_deadline=0 ri_nr_sent=1
motr[148770]:  c960  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7fc0c405be40, item 0x7fc0c4052e40[38] failed with ri_error=-110
motr[148770]:  4860   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.247.3@tcp:12345:33:800
motr[148770]:  4860   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.247.3@tcp:12345:33:903
----- 148770 stopped --------
----- 147275 stopping--------lt-m0d: got signal 1
----- 147275 stopped --------
Motr services stopped.
motr_io_small_disks: test status: SUCCESS
stale[bot] commented 1 year ago

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @nkommuri @mehjoshi @huanghua78 for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

shailesh-vaidya commented 7 months ago

Closing as an obsolete