Fix e2e self-hosted test failures in macOS

shubham1172 commented 1 year ago

Expected Behavior

E2E self-hosted tests should not be flaky on macOS.

Actual Behavior

E2E self-hosted tests are flaky on macOS. Example runs from last few days

Steps to Reproduce the Problem

Trigger a test.

Release Note

RELEASE NOTE: FIX Flakiness in E2E self-hosted tests

shubham1172 commented 1 year ago

Summary (past 7 days):
Successes: 30
Failures: 52
    Success rate: 36.59%

Job-wise summary:
Job: Run Self-Hosted E2E tests in darwin_amd64_slim
    failure: 25
    success: 24
    cancelled: 3
Job: Run Self-Hosted E2E tests in darwin_amd64_complete
    cancelled: 30
    failure: 22
Job: Run Self-Hosted E2E tests in linux_amd64_slim
    success: 46
    cancelled: 5
    failure: 1
Job: Run Self-Hosted E2E tests in linux_amd64_complete
    success: 42
    cancelled: 7
    failure: 3
Job: Run Self-Hosted E2E tests in windows_amd64_slim
    cancelled: 17
    success: 34
    failure: 1

shubham1172 commented 1 year ago

darwin_amd64_slim

The error comes from TestRunWithTemplateFile tests with message

"\ntermination signal received: shutting down\n" does not contain "XYZ"

Able to run the test suite locally on my Mac M1 consistently (with slim installation).

darwin_amd64_complete

Failures are coming due to podman installation being stuck on the github action.

Machine init complete
To start your machine run:

    podman machine start

Starting machine "podman-machine-default"
Waiting for VM ...
Error: The action has timed out.

Fix proposal: [X] Downgrade to podman 4.4.4 and check, failures seem to have increased from podman 4.5.0 [ ] Or, Remove this from scheduled e2e tests until podman installation on mac is fixed.

After PR #1279, average podman installation time is 4-5 mins, however it still times out occasionally (much better than 4.5.0)

dapr / cli