test: fix failing e2e tests because of marketing consent sheet

cortisiko commented 3 months ago

Description

The E2E tests are failing because of the enhance experience bottom sheet that appears in some of the tests. Furthermore, there is also a policy alert which also affects the e2e tests. The purpose of this PR is to stabilize the E2E tests so they handle the bottom sheet and the policy alert.

Keep in mind this bug also affects the e2e tests and a few try catches were added to deal with the bug.

Passing smoke run: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/7457565d-c8e4-4fd6-9df6-1cd6a1d76861

Related issues

Fixes:

Manual testing steps

Go to this page...

Screenshots/Recordings

Before

After

Pre-merge author checklist

[ ] I’ve followed MetaMask Contributor Docs and MetaMask Mobile Coding Standards.
[ ] I've completed the PR template to the best of my ability
[ ] I’ve included tests if applicable
[ ] I’ve documented my code using JSDoc format if applicable
[ ] I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

[ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
[ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

github-actions[bot] commented 3 months ago

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

cortisiko commented 3 months ago

Failing unit test: https://github.com/MetaMask/metamask-mobile/actions/runs/9558878763/job/26348334724?pr=10021

FAIL app/components/UI/OptinMetrics/index.test.tsx
  ● OptinMetrics › render matches snapshot

    expect(received).toMatchSnapshot()

    Snapshot name: `OptinMetrics render matches snapshot 1`

    - Snapshot  - 1
    + Received  + 0

    @@ -649,11 +649,10 @@
                                          "flexDirection": "row",
                                          "height": 24,
                                          "opacity": 1,
                                        }
                                      }
    -                                 testID="marketing-consent-opt-in-checkbox"
                                    >
                                      <View
                                        accessibilityRole="checkbox"
                                        style={
                                          {

      48 |       { state: {} },
      49 |     );
    > 50 |     expect(toJSON()).toMatchSnapshot();
         |                      ^
      51 |   });
      52 |
      53 |   describe('sets traits and sends metric event on confirm', () => {

      at Object.toMatchSnapshot (app/components/UI/OptinMetrics/index.test.tsx:50:22)

Snapshot Summary
 › 1 snapshot failed from 1 test suite. Inspect your code changes or run `yarn run test:unit -u` to update them.

yup I am currently looking into this

legobeat commented 3 months ago

Failing audit:ci addressed separately in #10018

github-actions[bot] commented 3 months ago

Bitrise

✅✅✅ pr_smoke_e2e_pipeline passed on Bitrise! ✅✅✅

Commit hash: e56955bb98cee032a5e6b95093a19b4ed4db93cc Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/7457565d-c8e4-4fd6-9df6-1cd6a1d76861

[!NOTE]

You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

github-actions[bot] commented 3 months ago

Bitrise

✅✅✅ pr_smoke_e2e_pipeline passed on Bitrise! ✅✅✅

Commit hash: a25dde57dead50e21d72051279900f79daa9328e Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/314e3a21-1cdc-4874-b2bb-04ca498b3e86

[!NOTE]

You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

github-actions[bot] commented 3 months ago

Bitrise

✅✅✅ pr_smoke_e2e_pipeline passed on Bitrise! ✅✅✅

Commit hash: 93a6e17e755c90e339aa1ceb3f4a5eb327fd5403 Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/27ece309-9a94-4bd4-b4ce-bfbde7e4307a

[!NOTE]

You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

cortisiko commented 3 months ago

Great coverage of these bottom sheets. However, too many incompressible sleeps added to me to feel comfortable. At least reduce them. This many sleeps either indicate a wrong way to test or a really unstable testing environment or a really bad initial implementation. Which one is it? Do we have an issue created for this to be fixed?

Side comment, you nicely logged some issues in try/catch that makes it easy to understand what failed while running e2e. Why no doing it all the time instead of empty comment lines in catches?

Hey Nico so an issue was created and the product team is aware. I made mention of the issue in the ticket description! The sleeps in the tests are in place to combat any latency on CI. Without some of the explicit delays, the tests run successfully locally but are flakey on CI.

github-actions[bot] commented 3 months ago