search

sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
200 stars 732 forks source link

SNMP spytest failure due to SNMP service failure #2857

Open tbgowda opened 3 years ago

tbgowda commented 3 years ago

Description

SNMP spytest test cases fail due to SNMP service failure. SNMP service exceeds the restart limit.

Steps to reproduce the issue:

  1. Have a two router setup with Sonic
  2. Run the spytest test case: system/test_snmp.py::test_ft_snmp_link_down_trap

Describe the results you received: 2021-01-21 01:19:47,122 T0000: INFO Restarting the 'snmp' service , post configuring snmp community. 2021-01-21 01:19:47,123 T0000: INFO ####### Restarting snmp service ######## 2021-01-21 01:19:47,323 T0000: INFO [D1-SD1] SCMD: sudo service snmp restart 2021-01-21 01:19:58,538 T0000: INFO [D1-SD1] Job for snmp.service failed. . . 2021-01-21 01:20:07,401 T0047: INFO [D1-SD1] Jan 21 01:20:05.547450 sonic ERR systemd[1]: Failed to start SNMP container.

Describe the results you expected: Not to have the script fail while restarting the SNMP container. There are two possibilities: a. The test case is incorrectly written b. Sonic software is restarting the SNMP docker more times than expected.

Additional information you deem important (e.g. issue happens only occasionally):

SNMP service has a limit of 3 starts within 1200 seconds (20 m). The script restarts the SNMP multiple times:

  1. When the config is applied
  2. Enable service timers
  3. Before the start of test (after configuring SNMP)
  4. When configuring : snmptrap modify
  5. After the test (after removing SNMP) (1) and (2) do not contribute towards the limit. (3), (4) and (5) occur within 20m and (5) fails. (4) is expected. The sonic code restarts SNMP docker when snmptrap is modified. It seems like things are working as expected but it is not clear as to why the test is written in this way

SNMP Failure logs at the end of the test: Jan 21 01:19:57 sonic systemd[1]: snmp.service: Start request repeated too quickly. Jan 21 01:19:57 sonic systemd[1]: Failed to start SNMP container. Jan 21 01:19:57 sonic systemd[1]: snmp.service: Unit entered failed state. Jan 21 01:19:57 sonic systemd[1]: snmp.service: Failed with result 'start-limit-hit'. Jan 21 01:20:05 sonic systemd[1]: snmp.service: Start request repeated too quickly. Jan 21 01:20:05 sonic systemd[1]: Failed to start SNMP container. Jan 21 01:20:05 sonic systemd[1]: snmp.service: Failed with result 'start-limit-hit'. Jan 21 01:20:48 sonic systemd[1]: snmp.service: Start request repeated too quickly. Jan 21 01:20:48 sonic systemd[1]: Failed to start SNMP container. Jan 21 01:20:48 sonic systemd[1]: snmp.service: Failed with result 'start-limit-hit'.

ramakristipati commented 3 years ago

Is there any command to increase the SNMP service limit so that this can be avoided