sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
201 stars 728 forks source link

[warmboot][dualtor] `event_counters` remains undefined if configuring mux mode fails #10857

Open zjswhhh opened 11 months ago

zjswhhh commented 11 months ago

https://github.com/sonic-net/sonic-mgmt/blob/51a5cf3f9794a3d1bd36049c38f7690004606408/tests/common/fixtures/advanced_reboot.py#L620-L637

Noticed that if config mux mode cmd fails (which is not rare), __setupRebootOper will exit out.

14/10/2023 05:25:26 advanced_reboot.runRebootTest            L0555 ERROR  | Exception caught while running advanced-reboot test on ptf: 
Traceback (most recent call last):
  File "/azp/_work/37/s/tests/common/fixtures/advanced_reboot.py", line 534, in runRebootTest
    event_counters = self.__setupRebootOper(rebootOper)
  File "/azp/_work/37/s/tests/common/fixtures/advanced_reboot.py", line 619, in __setupRebootOper
    'dut_mux_status': copy.deepcopy(show_muxcable_status(self.duthost)),
  File "/azp/_work/37/s/tests/common/dualtor/dual_tor_utils.py", line 1218, in show_muxcable_status
    output = json.loads(duthost.shell(command)["stdout"])
  File "/azp/_work/37/s/tests/common/devices/multi_asic.py", line 112, in _run_on_asics
    return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)
  File "/azp/_work/37/s/tests/common/devices/base.py", line 89, in _run
    raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res)
RunAnsibleModuleFail: run module shell failed, Ansible Results =>
{"changed": true, "cmd": "show muxcable status --json", "delta": "0:00:01.295710", "end": "2023-10-14 05:25:26.069724", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2023-10-14 05:25:24.774014", "stderr": "", "stderr_lines": [], "stdout": "could not retrieve key state value for port Ethernet8 inside table MUX_CABLE_TABLE", "stdout_lines": ["could not retrieve key state value for port Ethernet8 inside table MUX_CABLE_TABLE"]}

Some variables won't be initiated, which causes issue in later tests.

UnboundLocalError: local variable 'event_counters' referenced before assignment
zjswhhh commented 11 months ago

And since mux ports are not marked as manual, warmboot can fail for this. This might be why I can manually warmboot successfully but tests always fail.