sourcegraph / cody

Type less, code more: Cody is an AI code assistant that uses advanced search and codebase context to help you write and fix code.
https://cody.dev
Apache License 2.0
2.6k stars 278 forks source link

CI: disable symf in agent integration tests #4591

Closed olafurpg closed 3 months ago

olafurpg commented 3 months ago

Previously, symf was enabled in agent integration tests. We have observed quite a lot of unstable test failures that seem to be related to symf so this PR disabled symf entirely in the agent integration tests to see if it stablizes the CI.

Test plan

Non-flaky green CI.

olafurpg commented 3 months ago

Retrying an unrelated e2e flake


      56 | export async function expectAuthenticated(page: Page) {
      57 |     await focusSidebar(page)
    > 58 |     await expect(page.getByText('Chat alongside your code, attach files,')).toBeVisible()
         |                                                                             ^
      59 | }
      60 |
      61 | // Selector for the Explorer button in the sidebar that would match on Mac and Linux
olafurpg commented 3 months ago

Seemed like this PR improves the stability of the non-e2e tests. I'm retrying all jobs to see if they non-e2e tests pass again CleanShot 2024-06-17 at 19 56 20@2x

olafurpg commented 3 months ago

All tests passed against except this one here

Error:   1) troubleshooting-authConnection.test.ts:32:3 › allow retrying on connection issues ─────────────

    Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
    Error: expect(locator).toBeVisible()

    Locator: getByText('Chat alongside your code, attach files,')
    Expected: visible
    Received: <element(s) not found>
    Call log:
      - expect.toBeVisible with timeout 5000ms
      - waiting for getByText('Chat alongside your code, attach files,')

       at common.ts:58

      56 | export async function expectAuthenticated(page: Page) {
      57 |     await focusSidebar(page)
    > 58 |     await expect(page.getByText('Chat alongside your code, attach files,')).toBeVisible()
         |                                                                             ^
      59 | }

Retrying all jobs to as the final test to see if non-e2e tests pass again.

olafurpg commented 3 months ago

All non-e2e tests passed three times in a row (I ran "retry all" twice), so I think we have a strong signal that this PR stablizes the CI.

olafurpg commented 3 months ago

The third time I'm seeing the same e2e test failure

Error:   1) troubleshooting-authConnection.test.ts:32:3 › allow retrying on connection issues ─────────────

    Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
    Error: expect(locator).toBeVisible()

    Locator: getByText('Chat alongside your code, attach files,')
    Expected: visible
    Received: <element(s) not found>
    Call log:
      - expect.toBeVisible with timeout 5000ms
      - waiting for getByText('Chat alongside your code, attach files,')

       at common.ts:58

      56 | export async function expectAuthenticated(page: Page) {
      57 |     await focusSidebar(page)
    > 58 |     await expect(page.getByText('Chat alongside your code, attach files,')).toBeVisible()
         |                                                                             ^
      59 | }
      60 |
      61 | // Selector for the Explorer button in the sidebar that would match on Mac and Linux