MetaMask / metamask-extension

:globe_with_meridians: :electric_plug: The MetaMask browser extension enables browsing Ethereum blockchain enabled websites
https://metamask.io
Other
11.95k stars 4.89k forks source link

Fix "Test Snap Metrics test snap update failed metric" flaky tests #26579

Open hjetpoluru opened 1 month ago

hjetpoluru commented 1 month ago

Latest CI link

Test scenario - Test Snap Metrics test snap update failed metric

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/98396/workflows/be9ffb55-3ba2-4b60-9a36-92a18a5f2511/jobs/3662206/tests

TimeoutError: Waiting until element is enabled
Wait timed out after 10166ms
  (Ran on CircleCI Node 8 of 24, Job test-e2e-firefox)
    at /home/circleci/project/node_modules/selenium-webdriver/lib/webdriver.js:929:17
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/96587/workflows/b7beffde-c70c-4ee8-9f22-3d840f6e8a3f/jobs/3595672/tests

Error: listen EADDRINUSE: address already in use 127.0.0.1:8545.
  (Ran on CircleCI Node 1 of 10, Job test-e2e-firefox-flask)
    at listenSocket (webpack://Ganache/core/lib/src/server.js:193:33)
    at async Promise.allSettled (index 1)
    at Ganache.start (test/e2e/seeder/ganache.ts:23:5)
    at async withFixtures (test/e2e/helpers.js:102:7)
    at async Context.<anonymous> (test/e2e/snaps/test-snap-metrics.spec.js:802:5)
chloeYue commented 1 week ago

This test is still flaky and needs investigation. CI failure: https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/102084/workflows/5eabf94e-d53c-4896-964e-be046bbd4e40/jobs/3802642

bowensanders commented 1 week ago

Note: I attempted about a hundred times to recreate this particular transient failure (locally, in VMs and in Github Codespaces). I was unsuccessful.

The failure point seemed to be waiting for an alert popup (which already had a 1 second delay before attempting to close it) which, in the CI failure above just shows a spinner at the point the alert should pop up.

27519 adds an additional second to this delay because there does not seem to be another answer.