Prepare Cody web library API for Sourcegraph Cody Web client

[vovakulikov]

Original PR https://github.com/sourcegraph/cody/pull/4227

Closes SRCH-573 (Main issue for cody agent and vscode extension logic change)

This PR prepares VSCode Cody extension and agent logic for future use in Cody Web. It also updates cody/web package and its demo for testing purposes. After we merge this PR we should publish cody/web package to npm and use it in the sourcegraph repository as a common npm lib package.

Noticeable changes

• Since the agent will be running inside Web Worker for Cody Web, this PR adds support for custom storage (local storage doesn't work in Web Worker but IndexDB does). Because of this change, I had to "promisify" the whole API around localStorage service abstraction. Most changes files (more than have) have diff because of this change
• File and Symbol resolution can now work with remote repositories. Since there are no local files in Cody Web I added support for remote resources in context items (mentions resolutions), also common context also works with remote repositories.
• Some changes in agent API (nothing big more like extend changes to support Cody web client)
• Big update in cody/web client, basically it got close to Sourcegraph demo client usage and supports chat history, initial context, open context support, and mentions.

Todo before merge

• [x] Support custom local storage API
• [x] Solve the problem with Cody's web-style isolation
• [x] Fix unit tests
• [x] Fix integration tests
• [ ] Update Cody agent Kotlin bindings
• [x] Fix Cody agent unit tests
• [x] Fix e2e tests (most tests I got fixed, but here is a list with left tests that are failing)
  • [x] chat-atFile.test.ts:347:3 › @-mention symbol in chat
  • [x] command-core.test.ts:53:3 › Explain Command & Smell Command & Chat from Command Menu
  • [x] command-edit.test.ts:126:1 › edit (fixup) input - range selection
  • [x] initial-context.test.ts:35:18 › initial context - file

How to test it locally

To run cody web demo go to ./web and run pnpm dev To run vscode extension command use standard flow (this PR doesn't change anything about them) To run full Sourcegraph Cody Web demo see instructions in Sourcegraph PR here

Test plan

• Check that the Cody web demo and Cody web package build work properly
• Check manually that the VSCode Cody extension work properly

[github-actions[bot]]

‼️ Hey @sourcegraph/cody-security, please review this PR carefully as it introduces the usage of an unsafe_ function or abuses PromptString.

[olafurpg]

Also, can you update the PR description to link to the Linear issue/project?

[olafurpg]

Also, please tag the Cody Agent GH team for future PRs related to the agent. There is a lot of important context to why things are implemented the way they are. Just having a green CI and a stamp from somebody in Cody Web should not be enough to get a large PR like this merged.

[vovakulikov]

@olafurpg

Also, please tag the Cody Agent GH team for future PRs related to the agent

What do you mean? I tagged Cody's agent team in this PR, is there any other team rather than this one?

[sqs]

Symbols and chat history do not need to be in the initial version that ships to dotcom. Ideally they would make it in the July 10 release, but if they need to slip, I would still want to get everything else shipped on time rather than delay the whole thing.

[vovakulikov]

@sqs I greened CI today, unit, integrations and e2e tests are finally passing, so now I know that these changes are actually possible and don't break core logic (but I haven't checked other agent's consumers but Cody web), We had a conversation today with @olafurpg and even these changes seem ok in terms of tests but there are still concerns about corners cases. I will split this PR into smaller ones today/tomorrow to make it a bit safer to merge

While I'm doing this, on the other side we want to ship the new Cody Web as soon as possible on dotcom, @peterguy suggested that we actually could already ship this to dot com with publishing the updated Cody agent from the brunch, I already did this in sourcegraph PR here https://github.com/sourcegraph/sourcegraph/pull/62792

With this PR in sg repo, the new Cody web client will be available only for signed-in users on Dotcom, so no risks for any non-web agent's clients and we already use (used) the cody-web package which exists only in the npm, so I think this is a low-risk good compromise at this point, while I'm splitting and merging big changes in Cody agent.

cc @peterguy

[vovakulikov]

I'm going to close this one since changes from these PR will be delivered separately