Fix arp/test_unknown_mac by disabling arp_update

justin-wong-ce commented 3 months ago

Description of PR

The DUT will occasionally (~5 mins) send a echo request through IPv6 to the PTF container. When the PTF container replies, it populates the fdb table. This is bad because the test expects the fdb table to be empty.

~Temporarily disabling IPv6 on the PTF container during the test will eliminate the problem.~ Temporarily disabling arp_update on the DUT during the test will eliminate the problem.

Summary: Fixes #

Type of change

[ ] Bug fix
[ ] Testbed and Framework(new/improvement)
[x] Test case(new/improvement)

Back port request

[ ] 202012
[ ] 202205
[ ] 202305
[x] 202311
[x] 202405

Approach

What is the motivation for this PR?

Test was flaky. Monitoring tcpdump on the PTF container and the DUT's fdb table shows IPv6 echo is causing mac addresses to be learned on the DUT even though it was previously flushed.

How did you do it?

Disable IPv6.

How did you verify/test it?

Disabling IPv6 will allow the test to consistently past. If IPv6 is re-enabled during the test and packets are sent, it immediately fails.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld commented 3 months ago

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:

trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/arp/test_unknown_mac.py:45:1: E302 expected 2 blank lines, found 1 tests/arp/test_unknown_mac.py:47:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:54:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:55:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:56:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:58:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:60:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:61:4: E111 indentation is not a multiple of 4 tests/arp/test_unknown_mac.py:63:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped ... [truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
Ensure that the pre-commit package is installed:
```
sudo pip install pre-commit
```
Go to repository root folder
Install the pre-commit hooks:
```
pre-commit install
```
Use pre-commit to check staged file:
```
pre-commit
```

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

justin-wong-ce commented 3 months ago

Can we simply ignore the fdb entries from the IPv6 echo replies?

I don't believe there is a simple way to just ignore it. The echo reply would cause the DUT to learn all the mac addresses used in the test (even the "fake" ones that are generated). There is also no telltale sign from show mac that will indicate that a particular entry is from the echo reply.

Even if there is a way to differentiate ipv6 echo replied fdb entries, I don't believe it will be a good idea to filter or add logic to check, as the goal of the test is to test the behaviour when the mac address is unknown. The test will be the most accurate and functionally correct if the DUT indeed does not know the mac at all.

justin-wong-ce commented 3 months ago

Can we simply ignore the fdb entries from the IPv6 echo replies?

Actually, will change it so that we disable IPv6 on the DUT side instead of disabling it on the PTF container. Disabling on PTF may cause some previously populated items to be lost.

StormLiangMS commented 3 months ago

@lolyu could you help to review?

mssonicbld commented 3 months ago

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:

trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/arp/test_unknown_mac.py:44:1: E302 expected 2 blank lines, found 1 tests/arp/test_unknown_mac.py:71:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
Ensure that the pre-commit package is installed:
```
sudo pip install pre-commit
```
Go to repository root folder
Install the pre-commit hooks:
```
pre-commit install
```
Use pre-commit to check staged file:
```
pre-commit
```

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

justin-wong-ce commented 3 months ago

Changed to disable IPv6 from DUT instead of PTF. This method of disabling PTF is same as the method used here: https://github.com/sonic-net/sonic-mgmt/blob/master/tests/qos/qos_sai_base.py#L1815-L1830

lolyu commented 2 months ago

Maybe we can disable the arp refresh by disabling the arp_update:

# docker exec -it swss supervisorctl stop arp_update
arp_update: stopped
# docker exec -it swss supervisorctl start arp_update
arp_update: started

justin-wong-ce commented 2 months ago

Maybe we can disable the arp refresh by disabling the arp_update:
# docker exec -it swss supervisorctl stop arp_update
arp_update: stopped
# docker exec -it swss supervisorctl start arp_update
arp_update: started

Will use this method instead, thanks.

justin-wong-ce commented 2 months ago

Changed the fix method to use the arp_update command instead of disabling ipv6

mssonicbld commented 2 months ago

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:

trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/arp/test_unknown_mac.py:44:1: E302 expected 2 blank lines, found 1 tests/arp/test_unknown_mac.py:53:121: E501 line too long (121 > 120 characters) tests/arp/test_unknown_mac.py:57:121: E501 line too long (122 > 120 characters) tests/arp/test_unknown_mac.py:59:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
Ensure that the pre-commit package is installed:
```
sudo pip install pre-commit
```
Go to repository root folder
Install the pre-commit hooks:
```
pre-commit install
```
Use pre-commit to check staged file:
```
pre-commit
```

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld commented 2 months ago

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:

trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/arp/test_unknown_mac.py:44:1: E302 expected 2 blank lines, found 1 tests/arp/test_unknown_mac.py:53:121: E501 line too long (121 > 120 characters) tests/arp/test_unknown_mac.py:57:121: E501 line too long (122 > 120 characters) tests/arp/test_unknown_mac.py:59:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
Ensure that the pre-commit package is installed:
```
sudo pip install pre-commit
```
Go to repository root folder
Install the pre-commit hooks:
```
pre-commit install
```
Use pre-commit to check staged file:
```
pre-commit
```

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

ZhaohuiS commented 2 months ago

@lolyu could you please review it again?

mssonicbld commented 1 month ago

Cherry-pick PR to 202405: https://github.com/sonic-net/sonic-mgmt/pull/14121

mssonicbld commented 1 month ago

Cherry-pick PR to 202311: https://github.com/sonic-net/sonic-mgmt/pull/14213

sonic-net / sonic-mgmt