unstable tests

[RomanBelozerov]

Tempesta - ff0f02145172b92794ef3cd086a12af737caa793 Tempesta-test - ff0860655187b159cd981399961f9f1345e09b58

Any config:

• [ ] ws tests receive warning, but can be successful.
• [ ] t_frang.test_http_resp_code_block - AssertionError: 0 != 1 : Expected nums of warnings in journalctl: 1, but got 0
• [ ] tls.test_tls_integrity.ManyClientsH2 with body greater than 64 KB receives incorrect hash.

  FAIL: test_various_req_resp_sizes (tls.test_tls_integrity.ManyClientsH2)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
  File "/home/jenkins/workspace/_PR_tempesta-test_PR-509/tls/test_tls_integrity.py", line 251, in test_various_req_resp_sizes
  self.common_check(65536, 65536)
  File "/home/jenkins/workspace/_PR_tempesta-test_PR-509/tls/test_tls_integrity.py", line 328, in common_check
  self.assertTrue(hash1 == hash2, "Bad response checksum")
  AssertionError: False is not true : Bad response checksum

• [x] sessions.test_h2_sticky_scheduler.H2StickySchedulerTestCase.test_h2_cookie_scheduler - doesn't receive a warning in dmesg sometimes.

Local config:

• [ ] t_long_body.test_long_response.LongBodyInResponse - these tests do not work correctly after DeproxyServer reworking. Now tests work for me with 100 MB body. Maybe Nginx should be used for these tests.

Local config with TCP seg 1:

• [x] ~http2_general.test_h2_hpack.TestFramePayloadLength.test_large_frame_payload_length~ fixed in #2021

FAIL: test_large_frame_payload_length (http2_general.test_h2_hpack.TestFramePayloadLength)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/_PR_tempesta_PR-1981/tempesta-test/http2_general/test_h2_hpack.py", line 792, in test_large_frame_payload_length
    self.assertIn(ErrorCodes.PROTOCOL_ERROR, client.error_codes)
AssertionError: <ErrorCodes.PROTOCOL_ERROR: 1> not found in [<class 'ssl.SSLWantReadError'>]

• [ ] http2_general/test_h2_streams.TestH2Stream.test_max_concurrent_stream - test fail when runnig all tests in http2_general -T 1 http2_general. I cannot reproduce fail with -R option. client.error_codes when test fail - [<class 'ssl.SSLWantReadError'>, <class 'ssl.SSLWantReadError'>]

  Traceback (most recent call last):
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/http2_general/test_h2_streams.py", line 38, in test_max_concurrent_stream
  self.assertTrue(client.wait_for_reset_stream(stream_id=client.stream_id - 2))
  AssertionError: False is not true

Remote config:

• [ ] access_log - remote tests sometimes freeze on CI. Last test - access_log.test_access_log_h2.Response404Test.test_tempesta.
• [ ] t_frang.test_connection_rate_burst.FrangTcpVsBoth

  FAIL: test_rate (t_frang.test_connection_rate_burst.FrangTcpVsBoth)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
  File "/home/jenkins/workspace/Run_remote_tests/tempesta-test/t_frang/test_connection_rate_burst.py", line 351, in test_rate
  self.check_connections(
  File "/home/jenkins/workspace/Run_remote_tests/tempesta-test/t_frang/frang_test_case.py", line 112, in check_connections
  warns_occured = self.assertFrangWarning(warning, resets_expected)
  File "/home/jenkins/workspace/Run_remote_tests/tempesta-test/t_frang/frang_test_case.py", line 132, in assertFrangWarning
  self.assertTrue(self.klog.find(warning, cond=dmesg.amount_equals(expected)), expected)
  AssertionError: False is not true : 1

Please remove these tests after fix from tests_disabled*.json files and from tests_retry.

[voodam]

Also see these: https://github.com/tempesta-tech/tempesta/issues/1968 https://github.com/tempesta-tech/tempesta-test/issues/467

[voodam]

./run_tests.py -T 1:

FAIL: test_large_frame_payload_length (http2_general.test_h2_hpack.TestFramePayloadLength)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/jenkins/workspace/_PR_tempesta_PR-1981/tempesta-test/http2_general/test_h2_hpack.py", line 792, in test_large_frame_payload_length

    self.assertIn(ErrorCodes.PROTOCOL_ERROR, client.error_codes)

AssertionError: <ErrorCodes.PROTOCOL_ERROR: 1> not found in [<class 'ssl.SSLWantReadError'>]

Flaky test on TCP segmentation (pretty much, about half of tries). I tried to find the place where it was introduced, but didn't find, probably it was enabied by:

commit cac7c14f8b3e93b4be7ad0a872ec2d079a984820 (HEAD)
Author: EvgeniiMekhanik <em@tempesta-tech.com>
Date:   Mon Jul 24 10:48:05 2023 +0300

    Enabled tests blocked by PR 1926

[EvgeniiMekhanik]

test_block_action_error_reply_with_conn_close_garbage (http2_general.test_h2_block_action.BlockActionH2ReplyFramesAfterShutdownWithCustomErrorPageSmallWindow) is also unstable with TCP segmentation

[EvgeniiMekhanik]

tls.test_tls_cert.WrkTestsMultipleVhosts.test_wrk is also unstable with TCP segmentation

======================================================================
ERROR: test_wrk (tls.test_tls_cert.WrkTestsMultipleVhosts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/helpers/remote.py", line 107, in run_cmd
    assert p.returncode == 0, "Cmd: '%s' return code is not 0 (%d)." % (
AssertionError: Cmd: '/root/tempesta/scripts/tempesta.sh --reload' return code is not 0 (1).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/tls/test_tls_cert.py", line 875, in test_wrk
    self.config_changer(10)
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/tls/test_tls_cert.py", line 861, in config_changer
    self.get_tempesta().reload()
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/helpers/control.py", line 334, in reload
    self._do_run(f"{self.srcdir}/scripts/tempesta.sh --reload")
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/helpers/control.py", line 346, in _do_run
    self.node.run_cmd(cmd, timeout=30, env=env, err_msg=(self.err_msg % "start"))
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/helpers/remote.py", line 121, in run_cmd
    raise CmdError(
helpers.remote.CmdError: ('Сmd /root/tempesta/scripts/tempesta.sh --reload exited with return code 1', b'...compile html templates for JS challenge\nRunning live reconfiguration of Tempesta...\nERROR: cannot reconfigure Tempesta FW (sysctl message: \n0), please check dmesg\n', b'', 1)

----------------------------------------------------------------------