sonic-net / sonic-platform-daemons

Platform module daemons for SONiC
Other
23 stars 152 forks source link

[ycabled][active-active] Fix in gRPC channel callback logic by creating swsscommon table within the context #509

Closed vdahiya12 closed 3 months ago

vdahiya12 commented 3 months ago

This PR Introduces a an initialization mechanism within the wait_for_state_change function to ensure that the YcableChannelStateTableHelper instance is created only once per port. Also it Adds a check using hasattr to see if table_helper is already an attribute of wait_for_state_change. YcableChannelStateTableHelper instances are keyed by port.

This helps ensure that channel object will have a seperate instance of swsscommon table and they will not interfere with normal functioning of state specific threads.

Description

MSFT ADO

28724244

Motivation and Context

How Has This Been Tested?

UTs and deploying changes on Libra Testbeds as well as pilot devices This script was used to test and after each run the output was matched to be as expected. The fwd_state table is continuously written when link goes down and comes up unknown and active respectively.

admin@svcstr-7050-acs-1:~$ cat link_off.sh 
#!/bin/bash

declare -a runs=100

for ((i = 1; i <= 5000; i++));
do 
    sleep 1
    echo "*******shutting down the link*****************"
    sudo config interface shutdown Ethernet12
    sleep 1
    show mux status Ethernet12
    show mux hw mux Ethernet12
    redis-cli -n 0 hgetall "FORWARDING_STATE_RESPONSE:Ethernet12"
    sleep 1
    echo "***********starting up****************"
    sudo config interface start Ethernet12
    sleep 1
    show mux status Ethernet12
    show mux hw mux Ethernet12
    redis-cli -n 0 hgetall "FORWARDING_STATE_RESPONSE:Ethernet12"
    sleep 100
    echo "**********after long sleep************"
    show mux status Ethernet12
    show mux hw mux Ethernet12
    redis-cli -n 0 hgetall "FORWARDING_STATE_RESPONSE:Ethernet12"

done

example output for one iteration

*******shutting down the link*****************
PORT        STATUS    SERVER_STATUS    HEALTH     HWSTATUS      LAST_SWITCHOVER_TIME
----------  --------  ---------------  ---------  ------------  ----------------------
Ethernet12  standby   unknown          unhealthy  inconsistent
Port        Direction    Presence    PeerDirection    ConnectivityState
----------  -----------  ----------  ---------------  -------------------
Ethernet12  active       True        active           READY
response
unknown
response_peer
unknown
***********starting up****************
PORT        STATUS    SERVER_STATUS    HEALTH    HWSTATUS    LAST_SWITCHOVER_TIME
----------  --------  ---------------  --------  ----------  ---------------------------
Ethernet12  active    active           healthy   consistent  2024-Jul-15 23:51:32.477858
Port        Direction    Presence    PeerDirection    ConnectivityState
----------  -----------  ----------  ---------------  -------------------
Ethernet12  active       True        active           READY
response
active
response_peer
active

we can also check in syslog how the grpc connectivity state change syslog is posted with each link/up down

Jul 15 23:53:30.621827 svcstr-7050-acs-1 NOTICE pmon#CCmisApi: gRPC port Ethernet12 state changed to IDLE tid Thread-31
Jul 15 23:53:32.457226 svcstr-7050-acs-1 NOTICE pmon#CCmisApi: gRPC port Ethernet12 state changed to CONNECTING tid Thread-32
Jul 15 23:53:32.459680 svcstr-7050-acs-1 NOTICE pmon#CCmisApi: gRPC port Ethernet12 state changed to READY tid Thread-33

Additional Information (Optional)

another way to check thread function is correct is through logging, as seen fwd table is created in different threads

Jul  8 06:22:10.780902 svcstr-7050-acs-2 NOTICE pmon#CCmisApi:  created fwd table id Thread-2
Jul  8 06:22:10.806261 svcstr-7050-acs-2 NOTICE pmon#CCmisApi:  created fwd table id Thread-5
Jul  8 06:22:10.810397 svcstr-7050-acs-2 NOTICE pmon#CCmisApi: created fwd table id Thread-8
mssonicbld commented 3 months ago

Cherry-pick PR to 202311: https://github.com/sonic-net/sonic-platform-daemons/pull/516

mssonicbld commented 3 months ago

Cherry-pick PR to 202305: https://github.com/sonic-net/sonic-platform-daemons/pull/517

mssonicbld commented 3 months ago

Cherry-pick PR to 202405: https://github.com/sonic-net/sonic-platform-daemons/pull/518