search

sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
200 stars 732 forks source link

multi-asic:qos-sai: Test fails if we provide "-d" option with second DUT #8356

Open rraghav-cisco opened 1 year ago

rraghav-cisco commented 1 year ago

After the multi-asic support is added in qos-sai, we can't run the test with a single DUT, unless it is the first DUT itself. I have the following setup in testbed.csv:

sfd-vt2,sfd-vt2,t2,docker-ptf,ptf_vt2,1.72.33.204/16,fc0b::1/64,server_1,VM0100,sfd-vt2-lc0;sfd-vt2-lc1;sfd-vt2-lc2;sfd-vt2-sup,lab,True,Tests SFD T2

If I run without "-d" option, it works: ./run_tests.sh -n sfd-vt2 -O -u -r -m individual -t t2,any -l debug -p /run_logs/qos-rerun/ -e '--capture=no --skip_sanity --qos_swap_syncd=False --disable_loganalyzer' -c 'qos/test_qos_sai.py'

If I run with "-d sfd-vt2-lc0" it works, the single-asic mode is working: ./run_tests.sh -n sfd-vt2 -d sfd-vt2-lc0 -O -u -r -m individual -t t2,any -l debug -p /run_logs/qos-rerun/ -e '--capture=no --skip_sanity --qos_swap_syncd=False --disable_loganalyzer' -c 'qos/test_qos_sai.py'

However, if I run with "-d sfd-vt2-lc1", the single asic mode is failing: ./run_tests.sh -n sfd-vt2 -d sfd-vt2-lc1 -O -u -r -m individual -t t2,any -l debug -p /run_logs/qos-rerun/ -e '--capture=no --skip_sanity --qos_swap_syncd=False --disable_loganalyzer' -c 'qos/test_qos_sai.py'

The PTF command fails for all of them:

        if (res.is_failed or 'exception' in res) and not module_ignore_errors:
>           raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res)
E           RunAnsibleModuleFail: run module shell failed, Ansible Results =>
E           {"changed": true, "cmd": ["/root/env-python3/bin/ptf", "--test-dir", "saitests/py3", "sai_qos_tests.QSharedWatermarkTest", "--platform-dir", "ptftests", "--platform", "remote", "-t", "downlink_port_ips=[];dst_p
ort_2_vlan=None;sonic_asic_type=u'cisco-8000';dut_password=u'cisco123';pkts_num_fill_min=0;src_port_vlan=None;uplink_port_names=[];cell_size=384;src_server='1.72.33.206:9092';uplink_port_ips=[];pkts_num_margin=3072;src_asi
c_index=0;router_mac=u'E8:D3:22:30:22:1A';sonic_version=u'detached.4492-dirty-20230509.053744';downlink_port_names=[];pkts_num_leak_out=0;uplink_port_ids=[];downlink_port_ids=[];platform_asic=u'cisco-8000';dst_port_2_ip=u'
10.0.0.73';dst_port_3_ip=u'10.0.0.77';src_port_id=34;dst_port_ip=u'10.0.0.65';ecn=1;dst_port_3_vlan=None;dscp=8;dst_port_3_id=38;dst_asic_index=0;dst_port_2_id=36;dut_username=u'cisco';dst_port_id=32;port_map_file='/root/p
tf_test_port_map.json';dst_dut_index=0;hwsku=u'Cisco-88-LC0-36FH-M-O36';queue=0;src_dut_index=0;src_port_ip=u'10.0.0.69';pkts_num_trig_drp=16000;dst_port_vlan=None", "--qlen", "10000", "--disable-ipv6", "--disable-vxlan",
"--disable-geneve", "--disable-erspan", "--disable-mpls", "--disable-nvgre", "--log-file", "/tmp/sai_qos_tests.QSharedWatermarkTest.log", "--test-case-timeout", "1200"], "delta": "0:00:28.382151", "end": "2023-05-18 05:20:
29.331594", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2023-05-18 05:20:00.949443", "stderr": "/root/env-python3/bin/ptf:19: DeprecationWarning: the imp module is deprecated in favour of importlib; s
ee the module's documentation for alternative uses\n  import imp\nsai_qos_tests.QSharedWatermarkTest ... router_mac: E8:D3:22:30:22:1A\ndst_port_id: 32, src_port_id: 34, src_port_vlan: None\nactual dst_port_id: 33\nERROR\n
\n======================================================================\nERROR: sai_qos_tests.QSharedWatermarkTest\n----------------------------------------------------------------------\nTraceback (most recent call last)
:\n  File \"saitests/py3/sai_qos_tests.py\", line 3960, in runTest\n    recv_counters_base, _ = sai_thrift_read_port_counters(self.src_client, asic_type, port_list['src'][src_port_id])\nKeyError: 34\n\n--------------------
--------------------------------------------------\nRan 1 test in 25.893s\n\nFAILED (errors=1)", "stderr_lines": ["/root/env-python3/bin/ptf:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see
the module's documentation for alternative uses", "  import imp", "sai_qos_tests.QSharedWatermarkTest ... router_mac: E8:D3:22:30:22:1A", "dst_port_id: 32, src_port_id: 34, src_port_vlan: None", "actual dst_port_id: 33", "
ERROR", "", "======================================================================", "ERROR: sai_qos_tests.QSharedWatermarkTest", "----------------------------------------------------------------------", "Traceback (most
recent call last):", "  File \"saitests/py3/sai_qos_tests.py\", line 3960, in runTest", "    recv_counters_base, _ = sai_thrift_read_port_counters(self.src_client, asic_type, port_list['src'][src_port_id])", "KeyError: 34"
, "", "----------------------------------------------------------------------", "Ran 1 test in 25.893s", "", "FAILED (errors=1)"], "stdout": "Using packet manipulation module: ptf.packet_scapy\n\n**************************
****************\nATTENTION: SOME TESTS DID NOT PASS!!!\n\nThe following tests errored:\nQSharedWatermarkTest\n\n******************************************", "stdout_lines": ["Using packet manipulation module: ptf.packet_s
capy", "", "******************************************", "ATTENTION: SOME TESTS DID NOT PASS!!!", "", "The following tests errored:", "QSharedWatermarkTest", "", "******************************************"]}

complex_args = {'argv': ['/root/env-python3/bin/ptf', '--test-dir', 'saitests/py3', 'sai_qos_tests.QSharedWatermarkTest', '--platform-dir', 'ptftests', ...], 'chdir': '/root'}
filename   = '/masic/tests/qos/qos_sai_base.py'

As you can see, the src_dut_id=0 and src_dut_id=0 is the problem. The port_map file(/root/ptf_test_port_map.json) in the PTF is showing the correct mapping. It has port 33 mapped in dut index 1: "33": {"asic_idx": 0, "target_dest_mac": "e8:d3:22:30:22:1a", "target_dut": [1], "target_src_mac": ["e8:d3:22:30:22:1a"], "dut_port": "Ethernet8"}

I believe the src_dut_id=1 and dst_dut_id=1 would be required here.

@abdosi , @vmittal-msft , Pls check.

rraghav-cisco commented 1 year ago

I tried swapping the order of DUTs in the testbed.csv, but this time the port numbers are swapped. Due to this the PTF doesn't receive the packets, and the test fails.


E           RunAnsibleModuleFail: run module shell failed, Ansible Results =>
E           {"changed": true, "cmd": ["/root/env-python3/bin/ptf", "--test-dir", "saitests/py3", "sai_qos_tests.PFCXonTest", "--platform-dir", "ptftests", "--platform", "remote", "-t", "downlink_port_ips=[];dst_port_2_vlan=None;sonic_asic_type=u'cisco-8000';dut_password=u'cisco123';packet_size=1350;src_port_vlan=None;uplink_port_names=[];src_server='1.72.33.206:9092';uplink_port_ips=[];src_asic_index=0;router_mac=u'E8:D3:22:30:22:1A';sonic_version=u'detached.4492-dirty-20230509.053744';downlink_port_names=[];pkts_num_dismiss_pfc=2;pg=4;uplink_port_ids=[];pkts_num_leak_out=0;downlink_port_ids=[];platform_asic=u'cisco-8000';dst_port_2_ip=u'10.0.0.73';dst_port_3_ip=u'10.0.0.77';src_port_id=2;dst_port_ip=u'10.0.0.65';ecn=1;dst_port_3_vlan=None;dscp=4;dst_port_3_id=6;dst_asic_index=0;pkts_num_trig_pfc=3414;dst_port_2_id=4;buffer_max_size=u'0';dut_username=u'cisco';dst_port_id=0;port_map_file='/root/ptf_test_port_map.json';dst_dut_index=0;hwsku=u'Cisco-88-LC0-36FH-M-O36';src_dut_index=0;src_port_ip=u'10.0.0.69';dst_port_vlan=None", "--qlen", "10000", "--disable-ipv6", "--disable-vxlan", "--disable-geneve", "--disable-erspan", "--disable-mpls", "--disable-nvgre", "--log-file", "/tmp/sai_qos_tests.PFCXonTest.log", "--test-case-timeout", "1200"], "delta": "0:00:31.274357", "end": "2023-05-18 13:13:25.793444", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2023-05-18 13:12:54.519087", "stderr": "/root/env-python3/bin/ptf:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n  import imp\nsai_qos_tests.PFCXonTest ... dst_port_id:0, src_port_id:2\nFAIL\n\n======================================================================\nFAIL: sai_qos_tests.PFCXonTest\n----------------------------------------------------------------------\nTraceback (most recent call last):\n  File \"saitests/py3/sai_qos_tests.py\", line 1767, in runTest\n    src_port_id, pkt_dst_mac, dst_port_ip, src_port_ip, dst_port_id, src_port_vlan\n  File \"saitests/py3/sai_qos_tests.py\", line 1664, in get_rx_port\n    self, 0, src_port_id, pkt_dst_mac, dst_port_ip, src_port_ip, src_vlan\n  File \"saitests/py3/sai_qos_tests.py\", line 312, in get_rx_port\n    result.port, result.format()))\nAssertionError: Expected packet was not received. Received on port:None ========== EXPECTED ==========\nMask:\n\npacket status: OK\npacket:\n0000  E8 D3 22 30 22 1A BE 48 6F F9 28 02 08 00 45 01  ..\"0\"..Ho.(...E.\n0010  00 22 BA BE 00 00 40 00 AB 97 0A 00 00 45 0A 00  .\"....@......E..\n0020  00 41 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D  .A..............\n\npacket's mask:\n0000  00 00 00 00 00 00 00 00 00 00 00 00 FF FF FF FF  ................\n0010  00 00 FF FF FF FF 00 FF 00 00 FF FF FF FF FF FF  ................\n0020  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ................\n\n========== RECEIVED ==========\n2624 total packets. Displaying most recent 3 packets:\n------------------------------\n0000  0A 38 24 DC F1 4E E8 D3 22 30 24 6A 08 00 45 C0  .8$..N..\"0$j..E.\n0010  00 34 52 82 40 00 01 06 12 82 0A 00 00 00 0A 00  .4R.@...........\n0020  00 01 00 B3 86 2F 3E 50 B3 53 17 D5 4B EC 80 10  ...../>P.S..K...\n0030  02 80 1B A8 00 00 01 01 08 0A 01 13 23 09 92 2D  ............#..-\n0040  B2 03                                            ..\n------------------------------\n0000  4E D5 00 25 0A A3 E8 D3 22 30 24 6B 86 DD 6C 02  N..%....\"0$k..l.\n0010  B3 41 00 33 06 01 FC 00 00 00 00 00 00 00 00 00  .A.3............\n0020  00 00 00 00 00 41 FC 00 00 00 00 00 00 00 00 00  .....A..........\n0030  00 00 00 00 00 42 B1 20 00 B3 0C 90 3B 6F 0B E7  .....B. ....;o..\n0040  29 D3 80 18 00 6F E7 F6 00 00 01 01 08 0A 90 A2  )....o..........\n0050  1D CA 72 07 41 A4 FF FF FF FF FF FF FF FF FF FF  ..r.A...........\n0060  FF FF FF FF FF FF 00 13 04                       .........\n------------------------------\n0000  F2 98 80 D1 87 D2 E8 D3 22 30 24 6B 08 00 45 C0  ........\"0$k..E.\n0010  00 34 7E DA 40 00 01 06 E5 CD 0A 00 00 2E 0A 00  .4~.@...........\n0020  00 2F 9F 76 00 B3 60 11 97 FF 4F 5F 82 A8 80 10  ./.v..`...O_....\n0030  00 6F FA F5 00 00 01 01 08 0A 1E 01 CB DF 33 AC  .o............3.\n0040  DF 2C                                            .,\n==============================\n\n\n----------------------------------------------------------------------\nRan 1 test in 28.229s\n\nFAILED (failures=1)", "stderr_lines": ["/root/env-python3/bin/ptf:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses", "  import imp", "sai_qos_tests.PFCXonTest ... dst_port_id:0, src_port_id:2", "FAIL", "", "======================================================================", "FAIL: sai_qos_tests.PFCXonTest", "----------------------------------------------------------------------", "Traceback (most recent call last):", "  File \"saitests/py3/sai_qos_tests.py\", line 1767, in runTest", "    src_port_id, pkt_dst_mac, dst_port_ip, src_port_ip, dst_port_id, src_port_vlan", "  File \"saitests/py3/sai_qos_tests.py\", line 1664, in get_rx_port", "    self, 0, src_port_id, pkt_dst_mac, dst_port_ip, src_port_ip, src_vlan", "  File \"saitests/py3/sai_qos_tests.py\", line 312, in get_rx_port", "    result.port, result.format()))", "AssertionError: Expected packet was not received. Received on port:None ========== EXPECTED ==========", "Mask:", "", "packet status: OK", "packet:", "0000  E8 D3 22 30 22 1A BE 48 6F F9 28 02 08 00 45 01  ..\"0\"..Ho.(...E.", "0010  00 22 BA BE 00 00 40 00 AB 97 0A 00 00 45 0A 00  .\"....@......E..", "0020  00 41 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D  .A..............", "", "packet's mask:", "0000  00 00 00 00 00 00 00 00 00 00 00 00 FF FF FF FF  ................", "0010  00 00 FF FF FF FF 00 FF 00 00 FF FF FF FF FF FF  ................", "0020  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ................", "", "========== RECEIVED ==========", "2624 total packets. Displaying most recent 3 packets:", "------------------------------", "0000  0A 38 24 DC F1 4E E8 D3 22 30 24 6A 08 00 45 C0  .8$..N..\"0$j..E.", "0010  00 34 52 82 40 00 01 06 12 82 0A 00 00 00 0A 00  .4R.@...........", "0020  00 01 00 B3 86 2F 3E 50 B3 53 17 D5 4B EC 80 10  ...../>P.S..K...", "0030  02 80 1B A8 00 00 01 01 08 0A 01 13 23 09 92 2D  ............#..-", "0040  B2 03                                            ..", "------------------------------", "0000  4E D5 00 25 0A A3 E8 D3 22 30 24 6B 86 DD 6C 02  N..%....\"0$k..l.", "0010  B3 41 00 33 06 01 FC 00 00 00 00 00 00 00 00 00  .A.3............", "0020  00 00 00 00 00 41 FC 00 00 00 00 00 00 00 00 00  .....A..........", "0030  00 00 00 00 00 42 B1 20 00 B3 0C 90 3B 6F 0B E7  .....B. ....;o..", "0040  29 D3 80 18 00 6F E7 F6 00 00 01 01 08 0A 90 A2  )....o..........", "0050  1D CA 72 07 41 A4 FF FF FF FF FF FF FF FF FF FF  ..r.A...........", "0060  FF FF FF FF FF FF 00 13 04                       .........", "------------------------------", "0000  F2 98 80 D1 87 D2 E8 D3 22 30 24 6B 08 00 45 C0  ........\"0$k..E.", "0010  00 34 7E DA 40 00 01 06 E5 CD 0A 00 00 2E 0A 00  .4~.@...........", "0020  00 2F 9F 76 00 B3 60 11 97 FF 4F 5F 82 A8 80 10  ./.v..`...O_....", "0030  00 6F FA F5 00 00 01 01 08 0A 1E 01 CB DF 33 AC  .o............3.", "0040  DF 2C                                            .,", "==============================", "", "", "----------------------------------------------------------------------", "Ran 1 test in 28.229s", "", "FAILED (failures=1)"], "stdout": "Using packet manipulation module: ptf.packet_scapy\n\n******************************************\nATTENTION: SOME TESTS DID NOT PASS!!!\n\nThe following tests failed:\nPFCXonTest\n\n******************************************", "stdout_lines": ["Using packet manipulation module: ptf.packet_scapy", "", "******************************************", "ATTENTION: SOME TESTS DID NOT PASS!!!", "", "The following tests failed:", "PFCXonTest", "", "******************************************"]}