Seagate / cortx-motr

CORTX Motr is a distributed object and key-value storage system targeting mass capacity storage configurations. It's the core component of CORTX storage system.
https://github.com/Seagate/cortx
Apache License 2.0
59 stars 142 forks source link

CORTX-33057: Error handing for asserts. #2097

Closed AbhishekSahaSeagate closed 2 years ago

AbhishekSahaSeagate commented 2 years ago

motr/setup.c: M0_ASSERT(rctx->rc_mdstore.md_dom != NULL); This Assert was changed to check for NULL md_dom and return -ENOENT if check failed instead of raising a panic.

Signed-off-by: Abhishek Saha abhishek.saha@seagate.com

Problem Statement

Design

Coding

Checklist for Author

Testing

Checklist for Author

Impact Analysis

Checklist for Author/Reviewer/GateKeeper

Review Checklist

Checklist for Author

Documentation

Checklist for Author

AbhishekSahaSeagate commented 2 years ago

tested with these changes on m0d-ut:cs-single-service UT

diff --git a/motr/setup.c b/motr/setup.c
index f0c69f5..6a43417 100644
--- a/motr/setup.c
+++ b/motr/setup.c
@@ -1672,8 +1672,14 @@ static int cs_storage_setup(struct m0_motr *cctx)
                }
        }

+       M0_LOG(M0_ALWAYS, "ABK: STORAGE SETUP");
+
+       if (M0_FI_ENABLED("no_cob_dom"))
+               rctx->rc_mdstore.md_dom = NULL;
+
        if (rctx->rc_mdstore.md_dom == NULL) {
                rc = -ENOENT;
+               M0_LOG(M0_ALWAYS, "ABK: FI ENABLED");
                M0_ERR_INFO(rc, "Cob domain not found for root cob");
                goto cleanup_addb2;
        }
diff --git a/motr/ut/cs_ut_main.c b/motr/ut/cs_ut_main.c
index 62957e4..db45441 100644
--- a/motr/ut/cs_ut_main.c
+++ b/motr/ut/cs_ut_main.c
@@ -299,6 +299,8 @@ static int cs_ut_test_helper_success(struct cl_ctx *cctx, size_t cctx_nr,
        };

        rc = m0_rpc_server_start(&sctx);
+       if (rc == -ENOENT)
+               return rc;
        M0_UT_ASSERT(rc == 0);
        for (i = 0; i < cctx_nr; ++i) {
                rc = cs_ut_client_init(&cctx[i], cl_ep_addrs[i],
@@ -356,6 +358,7 @@ static void test_cs_ut_service_one(void)
 {
        struct cl_ctx cctx[1] = {};

+       m0_fi_enable_once("cs_storage_setup", "no_cob_dom");
        cs_ut_test_helper_success(cctx, ARRAY_SIZE(cctx), cs_ut_service_one_cmd,
                                  ARRAY_SIZE(cs_ut_service_one_cmd));
 }
[root@ssc-vm-g2-rhev4-1290 cortx-motr]#

When FI is enabled, the motr startup returns with error in an expected manner.

----- run_ut -t m0d-ut:cs-single-service -----
START Iteration: 1 out of 1
m0d-ut
  cs-single-service  motr[51122]:  57f0 ALWAYS  [motr/setup.c:1675:cs_storage_setup]  ABK: STORAGE SETUP
motr[51122]:  57f0 ALWAYS  [motr/setup.c:1682:cs_storage_setup]  ABK: FI ENABLED
                             1.46 sec   84 MiB
  [ time: 1.46 sec, mem: 84 MiB, leaked: 7 MiB ]

Time: 1.46 sec, Mem: 84 MiB, Leaked: 7 MiB, Asserts: 0
Unit tests status: SUCCESS
END   Iteration: 1 out of 1
yatin-mahajan commented 2 years ago

have triggered CI-CD for this PR https://eos-jenkins.colo.seagate.com/job/Cortx-PR-Build/job/Motr/1646/

rkothiya commented 2 years ago

Jenkins CI Result : Motr#1646

Motr Test Summary

Test ResultCountInfo
:x:Failed2
:file_folder:

04motr-single-node/49motr-rpc-cancel
01motr-single-node/00userspace-tests

:checkered_flag:Skipped32
:file_folder:

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/04initscripts
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation
04motr-single-node/34sns-repair-1n-1f
04motr-single-node/08spiel-sns-repair-quiesce
04motr-single-node/28sys-kvs-kernel
04motr-single-node/11m0t1fs-rconfc-fail
04motr-single-node/08spiel-sns-repair
04motr-single-node/19sns-repair-abort
04motr-single-node/22sns-repair-ios-fail
05motr-single-node/18sns-repair-quiesce
05motr-single-node/12fwait
05motr-single-node/16sns-repair-multi
05motr-single-node/07mount-fail
05motr-single-node/15sns-repair-single
05motr-single-node/23sns-abort-quiesce
05motr-single-node/17sns-repair-concurrent-io
05motr-single-node/07mount
05motr-single-node/07mount-multiple
05motr-single-node/12fsync

:heavy_check_mark:Passed43
:file_folder:

01motr-single-node/43m0crate
01motr-single-node/05confgen
01motr-single-node/06hagen
01motr-single-node/52motr-singlenode-sanity
01motr-single-node/01net
01motr-single-node/01kernel-tests
01motr-single-node/03console
01motr-single-node/02rpcping
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/72spiel-sns-motr-repair-quiesce
03motr-single-node/08spiel-multi-confd
03motr-single-node/69sns-repair-motr-quiesce
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/70sns-failure-after-repair-quiesce
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/71spiel-sns-motr-repair
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
04motr-single-node/73motr-io-small-disks
04motr-single-node/48motr-raid0-io
04motr-single-node/74motr-di-corruption-detection
04motr-single-node/25m0kv
04motr-single-node/44motr-rm-lock-cc-io
04motr-single-node/45motr-rmw
05motr-single-node/23dix-repair-m0repair
05motr-single-node/43motr-sync-replication
05motr-single-node/42motr-utils
05motr-single-node/45motr-sns-repair-N-1
05motr-single-node/40motr-dgmode
05motr-single-node/23dix-repair-quiesce-m0repair
05motr-single-node/23spiel-dix-repair-quiesce
05motr-single-node/44motr-sns-repair
05motr-single-node/23spiel-dix-repair

Total77:link:

CppCheck Summary

   Cppcheck: No new warnings found :thumbsup: