Fix multi-select in gazebo tests run with vite

suejung-sentry commented 6 days ago

Describe the bug There seem to be flaky tests related to the multi-select component when running yarn test:vite. From what I can tell, within the test, we render the multi-select component, click to trigger it to open, then look for the text expected in one of the options of the open multi-select component dropdown. BUT some of the time, the component isn't opening by the time we are looking for the text. The awaits in the test don't seem to be awaiting.

Example of a multi select component

The files I've seen this happen in are:

src/ui/FileViewer/ToggleHeader/Title/Title.test.tsx > TitleFlags > when selecting a flag > updates the location params TestingLibraryElementError: Unable to find an element with the text: flag-1. This could be because the text is broken up by multiple elements. In this case, you can provide a function for your text matcher to make your matcher more flexible.

(error is really long - posted at the bottom of this)

src/pages/RepoPage/CoverageTab/OverviewTab/subroute/ComponentsMultiSelect/ComponentsMultiSelect.test.tsx > ComponentsMultiSelect > when selecting a component > updates the location params Error: expect(element).toBeInTheDocument()

element could not be found in the document
❯ src/pages/RepoPage/CoverageTab/OverviewTab/subroute/ComponentsMultiSelect/ComponentsMultiSelect.test.tsx:198:26
196| 
197|       const component1 = await screen.findByText('component-1')
198|       expect(component1).toBeInTheDocument()
   |                          ^
199|       await user.click(component1)

To Reproduce Steps to reproduce the behavior:

run yarn test:vite src/ui/FileViewer/ToggleHeader/Title/Title.test.tsx a handful of times and see it fail some of the time
same with yarn test:vite src/pages/RepoPage/CoverageTab/OverviewTab/subroute/ComponentsMultiSelect/ComponentsMultiSelect.test.tsx

Expected behavior A clear and concise description of what you expected to happen. Expect it to always pass

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

desktop, vscode

Additional context Add any other context about the problem here.

Here's the really long error message referenced in the description


Ignored nodes: comments, script, style
<body>
  <div>
    <div
      class="w-full sm:w-60"
    >
      <div
        class="relative"
      >
        <div>
          <button
            aria-controls="downshift-0-menu"
            aria-expanded="false"
            aria-label="Filter by flags"
            class="flex justify-between items-center w-full border border-ds-gray-tertiary rounded bg-ds-container text-left px-3 h-8 disabled:text-ds-gray-quaternary disabled:bg-ds-gray-primary disabled:border-ds-gray-tertiary focus:outline-1 whitespace-nowrap"
            data-marketing="fileviwer-filter-by-flags"
            id="downshift-0-toggle-button"
            tabindex="-1"
          >
            <span
              class="flex items-center gap-2"
            >
              <svg
                class="w-6 h-6"
                data-icon=""
                data-testid=""
                fill="currentColor"
                viewBox="0 0 20 20"
                xmlns="http://www.w3.org/2000/svg"
              >
                <path
                  clip-rule="evenodd"
                  d="M3 6a3 3 0 013-3h10a1 1 0 01.8 1.6L14.25 8l2.55 3.4A1 1 0 0116 13H6a1 1 0 00-1 1v3a1 1 0 11-2 0V6z"
                  fill-rule="evenodd"
                />
              </svg>
              All flags
            </span>
            <svg
              class="w-6 h-6"
              data-icon=""
              data-testid=""
              fill="currentColor"
              viewBox="0 0 20 20"
              xmlns="http://www.w3.org/2000/svg"
            >
              <path
                clip-rule="evenodd"
                d="M5.293 7.293a1 1 0 011.414 0L10 10.586l3.293-3.293a1 1 0 111.414 1.414l-4 4a1 1 0 01-1.414 0l-4-4a1 1 0 010-1.414z"
                fill-rule="evenodd"
              />
            </svg>
          </button>
          <div
            class="absolute inset-x-0 z-50"
          >
            <div
              class="hidden"
            >
              <div>
                <label
                  class="mb-2 block font-semibold sr-only"
                  for="text-input6"
                >
                  Search for Flags
                </label>
                <div
                  class="relative"
                >
                  <div
                    class="absolute left-2 flex h-full items-center text-ds-gray-quaternary"
                  >
                    <svg
                      class="w-4 h-4"
                      data-icon="search"
                      data-testid="search"
                      fill="none"
                      stroke="currentColor"
                      viewBox="0 0 24 24"
                      xmlns="http://www.w3.org/2000/svg"
                    >
                      <path
                        d="M21 21l-6-6m2-5a7 7 0 11-14 0 7 7 0 0114 0z"
                        stroke-linecap="round"
                        stroke-linejoin="round"
                        stroke-width="2"
                      />
                    </svg>
                  </div>
                  <input
                    aria-activedescendant=""
                    aria-autocomplete="list"
                    aria-controls="downshift-0-menu"
                    aria-expanded="false"
                    aria-labelledby="downshift-0-label"
                    autocomplete="off"
                    class="block h-8 w-full border-ds-gray-tertiary bg-ds-container px-3 text-sm disabled:border-ds-gray-tertiary disabled:bg-ds-gray-primary disabled:text-ds-gray-quaternary border-t border-r border-l focus:border rounded-tl rounded-tr pl-7"
                    data-marketing="multiselect-search"
                    id="downshift-0-input"
                    placeholder="Search for Flags"
                    role="combobox"
                    type="text"
                    value=""
                  />
                </div>
              </div>
            </div>
          </div>
        </div>
        <ul
          aria-label="Filter by flags"
          aria-labelledby="downshift-0-label"
          class="overflow-hidden rounded-bl rounded-br bg-ds-container border-ds-gray-tertiary absolute w-full z-40 max-h-80 min-w-fit top-16"
          id="downshift-0-menu"
          role="listbox"
        />
      </div>
    </div>
  </div>
  <div
    aria-live="polite"
    aria-relevant="additions text"
    id="a11y-status-message"
    role="status"
    style="border: 0px; height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; width: 1px;"
  />
</body>

Ignored nodes: comments, script, style
<body>
  <div>
    <div
      class="w-full sm:w-60"
    >
      <div
        class="relative"
      >
        <div>
          <button
            aria-controls="downshift-0-menu"
            aria-expanded="false"
            aria-label="Filter by flags"
            class="flex justify-between items-center w-full border border-ds-gray-tertiary rounded bg-ds-container text-left px-3 h-8 disabled:text-ds-gray-quaternary disabled:bg-ds-gray-primary disabled:border-ds-gray-tertiary focus:outline-1 whitespace-nowrap"
            data-marketing="fileviwer-filter-by-flags"
            id="downshift-0-toggle-button"
            tabindex="-1"
          >
            <span
              class="flex items-center gap-2"
            >
              <svg
                class="w-6 h-6"
                data-icon=""
                data-testid=""
                fill="currentColor"
                viewBox="0 0 20 20"
                xmlns="http://www.w3.org/2000/svg"
              >
                <path
                  clip-rule="evenodd"
                  d="M3 6a3 3 0 013-3h10a1 1 0 01.8 1.6L14.25 8l2.55 3.4A1 1 0 0116 13H6a1 1 0 00-1 1v3a1 1 0 11-2 0V6z"
                  fill-rule="evenodd"
                />
              </svg>
              All flags
            </span>
            <svg
              class="w-6 h-6"
              data-icon=""
              data-testid=""
              fill="currentColor"
              viewBox="0 0 20 20"
              xmlns="http://www.w3.org/2000/svg"
            >
              <path
                clip-rule="evenodd"
                d="M5.293 7.293a1 1 0 011.414 0L10 10.586l3.293-3.293a1 1 0 111.414 1.414l-4 4a1 1 0 01-1.414 0l-4-4a1 1 0 010-1.414z"
                fill-rule="evenodd"
              />
            </svg>
          </button>
          <div
            class="absolute inset-x-0 z-50"
          >
            <div
              class="hidden"
            >
              <div>
                <label
                  class="mb-2 block font-semibold sr-only"
                  for="text-input6"
                >
                  Search for Flags
                </label>
                <div
                  class="relative"
                >
                  <div
                    class="absolute left-2 flex h-full items-center text-ds-gray-quaternary"
                  >
                    <svg
                      class="w-4 h-4"
                      data-icon="search"
                      data-testid="search"
                      fill="none"
                      stroke="currentColor"
                      viewBox="0 0 24 24"
                      xmlns="http://www.w3.org/2000/svg"
                    >
                      <path
                        d="M21 21l-6-6m2-5a7 7 0 11-14 0 7 7 0 0114 0z"
                        stroke-linecap="round"
                        stroke-linejoin="round"
                        stroke-width="2"
                      />
                    </svg>
                  </div>
                  <input
                    aria-activedescendant=""
                    aria-autocomplete="list"
                    aria-controls="downshift-0-menu"
                    aria-expanded="false"
                    aria-labelledby="downshift-0-label"
                    autocomplete="off"
                    class="block h-8 w-full border-ds-gray-tertiary bg-ds-container px-3 text-sm disabled:border-ds-gray-tertiary disabled:bg-ds-gray-primary disabled:text-ds-gray-quaternary border-t border-r border-l focus:border rounded-tl rounded-tr pl-7"
                    data-marketing="multiselect-search"
                    id="downshift-0-input"
                    placeholder="Search for Flags"
                    role="combobox"
                    type="text"
                    value=""
                  />
                </div>
              </div>
            </div>
          </div>
        </div>
        <ul
          aria-label="Filter by flags"
          aria-labelledby="downshift-0-label"
          class="overflow-hidden rounded-bl rounded-br bg-ds-container border-ds-gray-tertiary absolute w-full z-40 max-h-80 min-w-fit top-16"
          id="downshift-0-menu"
          role="listbox"
        />
      </div>
    </div>
  </div>
  <div
    aria-live="polite"
    aria-relevant="additions text"
    id="a11y-status-message"
    role="status"
    style="border: 0px; height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; width: 1px;"
  />
</body>
 ❯ waitForWrapper node_modules/@testing-library/dom/dist/wait-for.js:163:27
 ❯ node_modules/@testing-library/dom/dist/query-helpers.js:86:33
 ❯ src/ui/FileViewer/ToggleHeader/Title/Title.test.tsx:255:34
    253|       await user.click(select)
    254| 
    255|       const flag1 = await screen.findByText('flag-1')
       |                                  ^
    256|       expect(flag1).toBeInTheDocument()
    257|       await user.click(flag1)

suejung-sentry commented 6 days ago

so far me and @calvin-codecov have seen this happen

calvin-codecov commented 5 days ago

My failure was

 FAIL  src/pages/RepoPage/CoverageTab/OverviewTab/subroute/ComponentsMultiSelect/ComponentsMultiSelect.test.tsx > ComponentsMultiSelect > when searching for a component > falls back to substring search when bad regex is provided
TestingLibraryElementError: Unable to find an element with the text: component-1. This could be because the text is broken up by multiple elements. In this case, you can provide a function for your text matcher to make your matcher more flexible.

Ignored nodes: comments, script, style
<body>
  <div
    aria-live="polite"
    aria-relevant="additions text"
    id="a11y-status-message"
    role="status"
    style="border: 0px; height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; width: 1px;"
  />
  <div />
</body>

Ignored nodes: comments, script, style
<body>
  <div
    aria-live="polite"
    aria-relevant="additions text"
    id="a11y-status-message"
    role="status"
    style="border: 0px; height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; width: 1px;"
  />
  <div />
</body>
 ❯ waitForWrapper node_modules/@testing-library/dom/dist/wait-for.js:163:27
 ❯ node_modules/@testing-library/dom/dist/query-helpers.js:86:33
 ❯ src/pages/RepoPage/CoverageTab/OverviewTab/subroute/ComponentsMultiSelect/ComponentsMultiSelect.test.tsx:271:39
    269|       await user.click(select)
    270|
    271|       const component1 = await screen.findByText('component-1')
       |                                       ^
    272|       expect(component1).toBeInTheDocument()
    273|       const component2 = await screen.findByText('component-2')

but I fixed it with https://github.com/codecov/gazebo/pull/3363/commits/5aa73e58bb594faf9b2c63662873498ec1444de6

codecov / engineering-team

Fix multi-select in gazebo tests run with vite #2637