search

celestiaorg / knuu

Integration Test Framework
Apache License 2.0
39 stars 34 forks source link

migrate chaos mesh to the knuu #459

Closed mojtaba-esk closed 1 week ago

mojtaba-esk commented 3 months ago

Closes #451

it is in draft mode because the tests are flaky, need to find a way to assure the tests always.

The tests are passing when run one by one, but when run altogether some of them fail. The attempts done to mitigate this behavior:

  1. Running in sequence
  2. Running in individual namespaces (each test has its own knuu object)
  3. Added a function to wait until the Chaos rule is applied
  4. Added some delay time to make sure the Chaos traffic shape rule is applied & effective
  5. Increased the size of the bigfile to download to make the elapsed time measurement more accurate

Even tried to run them separately in different terminals, they also fail when they are running at the same time.

coderabbitai[bot] commented 3 months ago

[!IMPORTANT]

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

[!TIP]

Early access features: enabled We are currently testing the following features in early access: - **OpenAI `gpt-4o` model for code reviews and chat**: OpenAI claims that this model is better at understanding and generating code than the previous models. We seek your feedback over the next few weeks before making it generally available. Note: - You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file. - Please join our [Discord Community](https://discord.com/invite/GsXnASn26c) to provide feedback and report issues. - OSS projects are currently opted into early access features by default.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share - [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)
Tips ### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.
MSevey commented 3 months ago

Can you add it to the github workflow matrix so we can see the failures in CI? https://github.com/celestiaorg/knuu/blob/8d1b00fcae71de633bf5236bbec903d4b859303d/.github/workflows/knuu_testing.yml#L14

Can you share what the failures are that you are seeing locally?

mojtaba-esk commented 3 months ago

Can you add it to the github workflow matrix so we can see the failures in CI?

done

Can you share what the failures are that you are seeing locally?

Sure. Here is when I run it all together with isolated namespaces sequentially:

=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestBandwidth
time="2024-06-26T10:41:51+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:41:51+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104151-940
    chaosmesh_test.go:140: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:145: Time taken for wget before bandwidth: 27.345988762s   after bandwidth: 12.980991868s
    chaosmesh_test.go:147: 
            Error Trace:    /home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:147
            Error:          "12.980991868s" is not greater than "27.345988762s"
            Test:           TestRunSuite/TestBandwidth
            Messages:       Time after bandwidth is not longer than time before bandwidth
=== RUN   TestRunSuite/TestCorrupt
time="2024-06-26T10:42:56+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:42:56+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104256-035
    chaosmesh_test.go:108: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:113: Time taken for wget before corrupt: 29.818175866s     after corrupt: 14.994630351s
    chaosmesh_test.go:115: 
            Error Trace:    /home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:115
            Error:          "14.994630351s" is not greater than "29.818175866s"
            Test:           TestRunSuite/TestCorrupt
            Messages:       Time after corrupt is not longer than time before corrupt
=== RUN   TestRunSuite/TestDelay
time="2024-06-26T10:44:04+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:44:04+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104404-580
    chaosmesh_test.go:27: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:32: Time taken for wget before delay: 26.889980033s    after delay: 26.170361577s
    chaosmesh_test.go:34: 
            Error Trace:    /home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:34
            Error:          "26.170361577s" is not greater than "26.889980033s"
            Test:           TestRunSuite/TestDelay
            Messages:       Time after delay is not longer than time before delay
=== RUN   TestRunSuite/TestDuplicate
time="2024-06-26T10:45:19+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:45:19+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104519-042
    chaosmesh_test.go:81: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:86: Time taken for wget before duplicate: 27.027887443s    after duplicate: 26.465900458s
    chaosmesh_test.go:88: 
            Error Trace:    /home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:88
            Error:          "26.465900458s" is not greater than "27.027887443s"
            Test:           TestRunSuite/TestDuplicate
            Messages:       Time after duplicate is not longer than time before duplicate
=== RUN   TestRunSuite/TestLoss
time="2024-06-26T10:46:42+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:46:42+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-104642-121
    chaosmesh_test.go:54: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:59: Time taken for wget before loss: 13.95433273s      after loss: 14.305350963s
--- FAIL: TestRunSuite (343.31s)
    --- FAIL: TestRunSuite/TestBandwidth (64.09s)
    --- FAIL: TestRunSuite/TestCorrupt (68.55s)
    --- FAIL: TestRunSuite/TestDelay (74.46s)
    --- FAIL: TestRunSuite/TestDuplicate (83.08s)
    --- PASS: TestRunSuite/TestLoss (53.13s)
FAIL
FAIL    github.com/celestiaorg/knuu/e2e/chaosmesh   343.448s
FAIL

and when we run them one by one, they usually pass:

go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestBandwidth
go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestCorrupt
go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestDelay
go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestDuplicate
go test -v ./e2e/chaosmesh/ -timeout 60m -count=1 --run TestRunSuite/TestLoss
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestBandwidth
time="2024-06-26T10:54:40+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:54:40+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105440-946
    chaosmesh_test.go:140: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:145: Time taken for wget before bandwidth: 13.117046776s   after bandwidth: 27.103344713s
--- PASS: TestRunSuite (65.91s)
    --- PASS: TestRunSuite/TestBandwidth (65.91s)
PASS
ok      github.com/celestiaorg/knuu/e2e/chaosmesh   66.045s
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestCorrupt
time="2024-06-26T10:55:49+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:55:49+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105549-726
    chaosmesh_test.go:108: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:113: Time taken for wget before corrupt: 26.214538065s     after corrupt: 26.921705398s
--- PASS: TestRunSuite (77.70s)
    --- PASS: TestRunSuite/TestCorrupt (77.70s)
PASS
ok      github.com/celestiaorg/knuu/e2e/chaosmesh   77.845s
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestDelay
time="2024-06-26T10:57:10+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:57:10+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105710-157
    chaosmesh_test.go:27: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:32: Time taken for wget before delay: 28.403200834s    after delay: 29.653790723s
--- PASS: TestRunSuite (83.11s)
    --- PASS: TestRunSuite/TestDelay (83.11s)
PASS
ok      github.com/celestiaorg/knuu/e2e/chaosmesh   83.239s
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestDuplicate
time="2024-06-26T10:58:36+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:58:36+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105836-117
    chaosmesh_test.go:81: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:86: Time taken for wget before duplicate: 14.752317089s    after duplicate: 13.546388261s
    chaosmesh_test.go:88: 
            Error Trace:    /home/moji/celestia/knuu/e2e/chaosmesh/chaosmesh_test.go:88
            Error:          "13.546388261s" is not greater than "14.752317089s"
            Test:           TestRunSuite/TestDuplicate
            Messages:       Time after duplicate is not longer than time before duplicate
--- FAIL: TestRunSuite (52.02s)
    --- FAIL: TestRunSuite/TestDuplicate (52.02s)
FAIL
FAIL    github.com/celestiaorg/knuu/e2e/chaosmesh   52.155s
FAIL
=== RUN   TestRunSuite
=== RUN   TestRunSuite/TestLoss
time="2024-06-26T10:59:30+02:00" level=info msg="The .env file does not exist, continuing without loading environment variables."
time="2024-06-26T10:59:30+02:00" level=info msg="LOG_LEVEL: info" file="log/logger.go:34"
    suite_setup_test.go:67: Scope: 20240626-105930-866
    chaosmesh_test.go:54: Waiting 10s for chaos mesh to settle
    chaosmesh_test.go:59: Time taken for wget before loss: 14.224850495s     after loss: 26.397818292s
--- PASS: TestRunSuite (64.47s)
    --- PASS: TestRunSuite/TestLoss (64.47s)
PASS
ok      github.com/celestiaorg/knuu/e2e/chaosmesh   64.594s
smuu commented 1 week ago

Can we close this PR?

mojtaba-esk commented 1 week ago

Can we close this PR?

Well if we decide not to use Chaos mesh in knuu, then we can close it.

smuu commented 1 week ago

Can we close this PR?

Well if we decide not to use Chaos mesh in knuu, then we can close it.

The teams do not use knuu with network shaping and it seems they will not in the near future.