search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
736 stars 1.42k forks source link

test_decap[ttl=pipe, dscp=pipe] was consistently failing #8749

Closed bingwang-ms closed 3 years ago

bingwang-ms commented 3 years ago

Description

test_decap[ttl=pipe, dscp=pipe] failed on both Broadcom and Mellanox platform. Is ttl=pipe, dscp=pipe supported?

E               "stdout_lines": [
E                   "", 
E                   "IPv4inIPv4:", 
E                   "Did not receive expected packet on any of ports [0, 1, 4, 5, 16, 17, 20, 21] for device 0.", 
E                   "========== EXPECTED ==========", 
E                   "Mask:", 
E                   "0000   00 11 22 33 44 55 1C 34  DA 1D E4 00 08 00 45 00   ..\"3DU.4......E.", 
E                   "0010   00 64 00 01 00 00 01 06  B6 92 01 01 01 01 01 00   .d..............", 
E                   "0020   00 00 04 D2 00 50 00 00  00 00 00 00 00 00 50 02   .....P........P.", 
E                   "0030   20 00 1D FC 00 00 00 01  02 03 04 05 06 07 08 09    ...............", 
E                   "0040   0A 0B 0C 0D 0E 0F 10 11  12 13 14 15 16 17 18 19   ................", 
E                   "0050   1A 1B 1C 1D 1E 1F 20 21  22 23 24 25 26 27 28 29   ...... !\"#$%&'()", 
E                   "0060   2A 2B 2C 2D 2E 2F 30 31  32 33 34 35 36 37 38 39   *+,-./0123456789", 
E                   "0070   3A 3B                                              :;", 
E                   "mask = 00 00 00 00 00 00 00 00  00 00 00 00 ff ff ff ff", 
E                   "0010   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0020   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0030   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0040   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0050   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0060   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0070   ff ff  ", 
E                   "", 
E                   "========== RECEIVED ==========", 
E                   "3 total packets. Displaying most recent 3 packets:", 
E                   "------------------------------", 
E                   "0000   52 54 00 82 1D 7A 1C 34  DA 1D E4 00 08 00 45 C0   RT...z.4......E.", 
E                   "0010   00 47 2F 84 40 00 01 06  35 6D 0A 00 00 00 0A 00   .G/.@...5m......", 
E                   "0020   00 01 00 B3 EB EE B3 71  CF EE 83 6F 36 D8 80 18   .......q...o6...", 
E                   "0030   01 BA 8B 05 00 00 01 01  08 0A 9F 9D F7 B9 57 77   ..............Ww", 
E                   "0040   B8 B6 FF FF FF FF FF FF  FF FF FF FF FF FF FF FF   ................", 
E                   "0050   FF FF 00 13 04                                     .....", 
E                   "------------------------------", 
E                   "0000   52 54 00 4B B9 56 1C 34  DA 1D E4 00 08 00 45 C0   RT.K.V.4......E.", 
E                   "0010   00 47 AC 59 40 00 01 06  B8 7F 0A 00 00 0C 0A 00   .G.Y@...........", 
E                   "0020   00 0D 00 B3 9A 1A 05 04  C7 2E 10 7A FD FD 80 18   ...........z....", 
E                   "0030   01 BA B9 FD 00 00 01 01  08 0A E8 06 F9 B1 57 78   ..............Wx", 
E                   "0040   F4 15 FF FF FF FF FF FF  FF FF FF FF FF FF FF FF   ................", 
E                   "0050   FF FF 00 13 04                                     .....", 
E                   "------------------------------", 
E                   "0000   52 54 00 4B B9 56 1C 34  DA 1D E4 00 08 00 45 40   RT.K.V.4......E@", 
E                   "0010   00 64 00 01 00 00 01 06  B6 52 01 01 01 01 01 00   .d.......R......", 
E                   "0020   00 00 04 D2 00 50 00 00  00 00 00 00 00 00 50 02   .....P........P.", 
E                   "0030   20 00 1D FC 00 00 00 01  02 03 04 05 06 07 08 09    ...............", 
E                   "0040   0A 0B 0C 0D 0E 0F 10 11  12 13 14 15 16 17 18 19   ................", 
E                   "0050   1A 1B 1C 1D 1E 1F 20 21  22 23 24 25 26 27 28 29   ...... !\"#$%&'()", 
E                   "0060   2A 2B 2C 2D 2E 2F 30 31  32 33 34 35 36 37 38 39   *+,-./0123456789", 
E                   "0070   3A 3B                                              :;", 
E                   "==============================", 
E                   "", 
E                   "", 
E                   "IPv6inIPv4:", 
E                   "Did not receive expected packet on any of ports [0, 1, 4, 5, 16, 17, 20, 21] for device 0.", 
E                   "========== EXPECTED ==========", 
E                   "Mask:", 
E                   "0000   00 11 22 33 44 55 1C 34  DA 1D E4 00 86 DD 60 40   ..\"3DU.4......`@", 
E                   "0010   00 00 00 2E 06 02 00 01  00 00 00 00 00 00 00 00   ................", 
E                   "0020   00 00 00 00 00 01 00 00  00 00 00 00 00 00 00 00   ................", 
E                   "0030   00 00 00 00 00 02 04 D2  00 50 00 00 00 00 00 00   .........P......", 
E                   "0040   00 00 50 02 20 00 13 2C  00 00 44 44 44 44 44 44   ..P. ..,..DDDDDD", 
E                   "0050   44 44 44 44 44 44 44 44  44 44 44 44 44 44 44 44   DDDDDDDDDDDDDDDD", 
E                   "0060   44 44 44 44                                        DDDD", 
E                   "mask = 00 00 00 00 00 00 00 00  00 00 00 00 ff ff ff ff", 
E                   "0010   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0020   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0030   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0040   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0050   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0060   ff ff ff ff  ", 
E                   "", 
E                   "========== RECEIVED ==========", 
E                   "2 total packets. Displaying most recent 2 packets:", 
E                   "------------------------------", 
E                   "0000   52 54 00 4B B9 56 1C 34  DA 1D E4 00 86 DD 64 40   RT.K.V.4......d@", 
E                   "0010   00 00 00 2E 06 02 00 01  00 00 00 00 00 00 00 00   ................", 
E                   "0020   00 00 00 00 00 01 00 00  00 00 00 00 00 00 00 00   ................", 
E                   "0030   00 00 00 00 00 02 04 D2  00 50 00 00 00 00 00 00   .........P......", 
E                   "0040   00 00 50 02 20 00 13 2C  00 00 44 44 44 44 44 44   ..P. ..,..DDDDDD", 
E                   "0050   44 44 44 44 44 44 44 44  44 44 44 44 44 44 44 44   DDDDDDDDDDDDDDDD", 
E                   "0060   44 44 44 44                                        DDDD", 
E                   "------------------------------", 
E                   "0000   52 54 00 C3 FE 06 1C 34  DA 1D E4 00 08 00 45 C0   RT.....4......E.", 
E                   "0010   00 47 8F CA 40 00 01 06  D5 1E 0A 00 00 04 0A 00   .G..@...........", 
E                   "0020   00 05 00 B3 AB FC 46 23  AF 03 40 D1 FA 02 80 18   ......F#..@.....", 
E                   "0030   01 BA F7 4B 00 00 01 01  08 0A 14 EF 38 F8 57 7B   ...K........8.W{", 
E                   "0040   E3 73 FF FF FF FF FF FF  FF FF FF FF FF FF FF FF   .s..............", 
E                   "0050   FF FF 00 13 04                                     .....", 
E                   "==============================", 
E                   "", 
E                   "", 
E                   "IPv4inIPv6:", 
E                   "Did not receive expected packet on any of ports [0, 1, 4, 5, 16, 17, 20, 21] for device 0.", 
E                   "========== EXPECTED ==========", 
E                   "Mask:", 
E                   "0000   00 11 22 33 44 55 1C 34  DA 1D E4 00 08 00 45 08   ..\"3DU.4......E.", 
E                   "0010   00 64 00 01 00 00 03 06  B4 8A 01 01 01 01 01 00   .d..............", 
E                   "0020   00 00 04 D2 00 50 00 00  00 00 00 00 00 00 50 02   .....P........P.", 
E                   "0030   20 00 1D FC 00 00 00 01  02 03 04 05 06 07 08 09    ...............", 
E                   "0040   0A 0B 0C 0D 0E 0F 10 11  12 13 14 15 16 17 18 19   ................", 
E                   "0050   1A 1B 1C 1D 1E 1F 20 21  22 23 24 25 26 27 28 29   ...... !\"#$%&'()", 
E                   "0060   2A 2B 2C 2D 2E 2F 30 31  32 33 34 35 36 37 38 39   *+,-./0123456789", 
E                   "0070   3A 3B                                              :;", 
E                   "mask = 00 00 00 00 00 00 00 00  00 00 00 00 ff ff ff ff", 
E                   "0010   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0020   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0030   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0040   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0050   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0060   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0070   ff ff  ", 
E                   "", 
E                   "========== RECEIVED ==========", 
E                   "1 total packets. Displaying most recent 1 packets:", 
E                   "------------------------------", 
E                   "0000   52 54 00 82 1D 7A 1C 34  DA 1D E4 00 08 00 45 48   RT...z.4......EH", 
E                   "0010   00 64 00 01 00 00 03 06  B4 4A 01 01 01 01 01 00   .d.......J......", 
E                   "0020   00 00 04 D2 00 50 00 00  00 00 00 00 00 00 50 02   .....P........P.", 
E                   "0030   20 00 1D FC 00 00 00 01  02 03 04 05 06 07 08 09    ...............", 
E                   "0040   0A 0B 0C 0D 0E 0F 10 11  12 13 14 15 16 17 18 19   ................", 
E                   "0050   1A 1B 1C 1D 1E 1F 20 21  22 23 24 25 26 27 28 29   ...... !\"#$%&'()", 
E                   "0060   2A 2B 2C 2D 2E 2F 30 31  32 33 34 35 36 37 38 39   *+,-./0123456789", 
E                   "0070   3A 3B                                              :;", 
E                   "==============================", 
E                   "", 
E                   "", 
E                   "IPv6inIPv6:", 
E                   "Did not receive expected packet on any of ports [0, 1, 4, 5, 16, 17, 20, 21] for device 0.", 
E                   "========== EXPECTED ==========", 
E                   "Mask:", 
E                   "0000   00 11 22 33 44 55 1C 34  DA 1D E4 00 86 DD 60 C0   ..\"3DU.4......`.", 
E                   "0010   00 00 00 2E 06 04 00 01  00 00 00 00 00 00 00 00   ................", 
E                   "0020   00 00 00 00 00 01 00 00  00 00 00 00 00 00 00 00   ................", 
E                   "0030   00 00 00 00 00 02 04 D2  00 50 00 00 00 00 00 00   .........P......", 
E                   "0040   00 00 50 02 20 00 13 2C  00 00 44 44 44 44 44 44   ..P. ..,..DDDDDD", 
E                   "0050   44 44 44 44 44 44 44 44  44 44 44 44 44 44 44 44   DDDDDDDDDDDDDDDD", 
E                   "0060   44 44 44 44                                        DDDD", 
E                   "mask = 00 00 00 00 00 00 00 00  00 00 00 00 ff ff ff ff", 
E                   "0010   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0020   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0030   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0040   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0050   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff", 
E                   "0060   ff ff ff ff  ", 
E                   "", 
E                   "========== RECEIVED ==========", 
E                   "1 total packets. Displaying most recent 1 packets:", 
E                   "------------------------------", 
E                   "0000   52 54 00 E6 E9 37 1C 34  DA 1D E4 00 86 DD 64 C0   RT...7.4......d.", 
E                   "0010   00 00 00 2E 06 04 00 01  00 00 00 00 00 00 00 00   ................", 
E                   "0020   00 00 00 00 00 01 00 00  00 00 00 00 00 00 00 00   ................", 
E                   "0030   00 00 00 00 00 02 04 D2  00 50 00 00 00 00 00 00   .........P......", 
E                   "0040   00 00 50 02 20 00 13 2C  00 00 44 44 44 44 44 44   ..P. ..,..DDDDDD", 
E                   "0050   44 44 44 44 44 44 44 44  44 44 44 44 44 44 44 44   DDDDDDDDDDDDDDDD", 
E                   "0060   44 44 44 44                                        DDDD", 
E                   "==============================", 
E                   "", 
E                   "", 
E                   "Summary:", 
E                   "IPv6inIPv4: Failed", 
E                   "IPv4inIPv4: Failed", 
E                   "IPv6inIPv6: Failed", 
E                   "IPv4inIPv6: Failed"
E               ]
E           }

Steps to reproduce the issue:

  1. Run test_decap

Describe the results you received:

Describe the results you expected:

Output of show version:

SONiC Software Version: SONiC.20201231.25
Distribution: Debian 10.10
Kernel: 4.19.0-12-2-amd64
Build commit: 8b5a00ba22
Build date: Thu Sep  9 12:36:50 UTC 2021
Built by: AzDevOps@sonic-int-build-workers-0003MB

Platform: x86_64-mlnx_msn4600c-r0
HwSKU: ACS-MSN4600C
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2023X22076
Uptime: 08:12:51 up  3:31,  1 user,  load average: 1.72, 1.51, 1.20

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

Blueve commented 3 years ago

This test can pass on Nokia 7215 platform (ARM) with parameter ttl=pipe, dscp=pipe

neethajohn commented 3 years ago

Was debugging this on 2700 and see that the dscp field is different in the received packet. Look at the highlighted field in the sent packet and the 3rd received packet.

E "IPv4inIPv4:", E "Did not receive expected packet on any of ports [0, 1, 4, 5, 16, 17, 20, 21] for device 0.", E "========== EXPECTED ==========", E "Mask:", E "0000 00 11 22 33 44 55 1C 34 DA 1D E4 00 08 00 45 00 ..\"3DU.4......E.", E "0010 00 64 00 01 00 00 01 06 B6 92 01 01 01 01 01 00 .d..............", E "0020 00 00 04 D2 00 50 00 00 00 00 00 00 00 00 50 02 .....P........P.", E "0030 20 00 1D FC 00 00 00 01 02 03 04 05 06 07 08 09 ...............", E "0040 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 ................", E "0050 1A 1B 1C 1D 1E 1F 20 21 22 23 24 25 26 27 28 29 ...... !\"#$%&'()", E "0060 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 +,-./0123456789", E "0070 3A 3B :;", E "mask = 00 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff", E "0010 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff", E "0020 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff", E "0030 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff", E "0040 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff", E "0050 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff", E "0060 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff", E "0070 ff ff ", E "", E "========== RECEIVED ==========", E "3 total packets. Displaying most recent 3 packets:", E "------------------------------", E "0000 52 54 00 82 1D 7A 1C 34 DA 1D E4 00 08 00 45 C0 RT...z.4......E.", E "0010 00 47 2F 84 40 00 01 06 35 6D 0A 00 00 00 0A 00 .G/.@...5m......", E "0020 00 01 00 B3 EB EE B3 71 CF EE 83 6F 36 D8 80 18 .......q...o6...", E "0030 01 BA 8B 05 00 00 01 01 08 0A 9F 9D F7 B9 57 77 ..............Ww", E "0040 B8 B6 FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................", E "0050 FF FF 00 13 04 .....", E "------------------------------", E "0000 52 54 00 4B B9 56 1C 34 DA 1D E4 00 08 00 45 C0 RT.K.V.4......E.", E "0010 00 47 AC 59 40 00 01 06 B8 7F 0A 00 00 0C 0A 00 .G.Y@...........", E "0020 00 0D 00 B3 9A 1A 05 04 C7 2E 10 7A FD FD 80 18 ...........z....", E "0030 01 BA B9 FD 00 00 01 01 08 0A E8 06 F9 B1 57 78 ..............Wx", E "0040 F4 15 FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................", E "0050 FF FF 00 13 04 .....", E "------------------------------", E "0000 52 54 00 4B B9 56 1C 34 DA 1D E4 00 08 00 45 40 RT.K.V.4......E@", E "0010 00 64 00 01 00 00 01 06 B6 52 01 01 01 01 01 00 .d.......R......", E "0020 00 00 04 D2 00 50 00 00 00 00 00 00 00 00 50 02 .....P........P.", E "0030 20 00 1D FC 00 00 00 01 02 03 04 05 06 07 08 09 ...............", E "0040 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 ................", E "0050 1A 1B 1C 1D 1E 1F 20 21 22 23 24 25 26 27 28 29 ...... !\"#$%&'()", E "0060 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 +,-./0123456789", E "0070 3A 3B :;", E "==============================",

neethajohn commented 3 years ago

https://github.com/Azure/sonic-mgmt/pull/4278 was the fix needed for Mellanox