bug: Cody incorrectly states that it doesn't have the required context, then still answers the question

[chillatom]

Version

multiple

Describe the bug

Slack background

Users report that Cody consistently responds with "Unfortunately I don't have access to..." even when Cody does in fact have access to the file and it is even explicitly included in context. Cody still will respond.

Anecdotal feedback that this occurs more prevalently when selecting remote / multiple repos

This happens across IDEs & Web.

Expected behavior

If Cody does have access to a file and successfully uses it as context, it should not respond with "unfortunately".

Additional context

No response

[dhdiemer]

IDE: VS Code Version: 1.6.1 Account Type: Enterprise Repo Configuration: Local + one remote repo, question is asked of remote repo Response:

`Unfortunately there is not much information in the code snippets and documentation provided about how authentication specifically works for Cody.

Some relevant information I could gather:

There are various authentication methods supported in Sourcegraph such as SAML, OAuth, OpenID Connect, etc. as mentioned in the Sourcegraph authentication documentation. These could potentially be leveraged by Cody as well.

The Cody VS Code extension code refers to some authentication status checking, including checking if the user is logged in. This implies some authentication mechanism is used.

There are references to an "authenticated user" in some of the TypeScript code, suggesting authenticated users are represented in a certain way.

One code snippet mentions "Authentication Proxy" as an authentication system option, which may be relevant for Cody.

But there are no clear details provided on the exact end-to-end authentication flow or methods used specifically for Cody itself. More context would be needed to fully understand Cody's authentication implementation.`

[ajaynz]

Happened to me on demo.sourcegraph.com during a customer demo today using web chat .

[hopkinscl]

Not pasting the screenshots directly as some show customers' code, but I've heard several reports of this behavior. Some specific examples are viewable in this Slack thread

[chillatom]

I think we have a few states being discussed here that need to be parsed

• Cody says "unfortunately..." but actually has the relevant information it needs to answer the question and does so. Saying "unfortunately" degrades confidence in the answer

• Cody says "unfortunately..." and was probably correct in hedging. @ajaynz the example you linked probably falls into this category, since Cody is only seeing 16 files, it doesn't necessarily know the whole tech stack. Answer this type of questions perfectly likely requires some more complex reasoning on the backend.

• Cody actually fails to attend to the context pulled in, or fails to retrieve the context selected or referenced. @hopkinscl has some examples that feel like they fit this

[chillatom]

More issues noted in this slack post

Cody Web

[chillatom]

Another user issue on web

[dhdiemer]

issue in VS code, specifically around multi repo:

1. Only access to one repo In this example, Cody only has access to sourcegraph/sourcegraph and is able to give a fairly decent answer about how auth works for a different repo. In this example, sourcegraph/sourcegraph was the local context.

2. Access to two repos Here, we've added sourcegraph/cody as additional context. You can see cody references files from sourcegraph/cody but still says there is not enough information. Sourcegraph/sourcegraph was local context and sourcegraph/cody was remote context

3. Sourcegraph/cody as local context with sourcegraph/cody as the local context, we get a very detailed response to the question.

4. sourcegraph/cody as local context with sourcegraph/sourcegraph as remote repo Finally, we see what happens when we add remote context to cody here as well. Cody was previously able to answer the question when we only had one repo and adding a second context source gave us, unfortunately again.

[chillatom]

Notes from @cconcannon

• Most users encounter this during part of the trial and ask
• Most prevalent on web (very easy to repro on web)
• Not seen as much in VS Code, but it does happen
• Seems to have started around 5.3 release or slightly before
• Chris is testing on demo instance with web front end

[cconcannon]

Another example using multi-repo context on demo.sourcegraph.com (sourcegraph/cody and sourcegraph/sourcegraph):

tell me about the interfaces between sourcegraph and cody

Read 16 files
Searched entire codebase for relevant files
[doc/notebooks/index.md](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/doc/notebooks/index.md)
[doc/dev/background-information/backgroundroutine.md](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/doc/dev/background-information/backgroundroutine.md)
[doc/cody/overview/cody-with-sourcegraph.md](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/doc/cody/overview/cody-with-sourcegraph.md)
[vscode/CHANGELOG.md](https://demo.sourcegraph.com/github.com/sourcegraph/cody@2e09386706222343a92adf833b983b920fed0c19/-/blob/vscode/CHANGELOG.md)
[mockgen.test.yaml](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/mockgen.test.yaml)
[mockgen.temp.yaml](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/mockgen.temp.yaml)
[internal/codeintel/autoindexing/internal/background/dependencies/iface.go](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/internal/codeintel/autoindexing/internal/background/dependencies/iface.go)
[client/browser/src/browser-extension/web-extension-api/types.ts](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/client/browser/src/browser-extension/web-extension-api/types.ts)
[internal/codeintel/autoindexing/internal/background/scheduler/iface.go](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/internal/codeintel/autoindexing/internal/background/scheduler/iface.go)
[internal/search/job/job.go](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/internal/search/job/job.go)
[internal/scim/user_create_test.go](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/internal/scim/user_create_test.go)
[internal/audit/audit_test.go](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/internal/audit/audit_test.go)
[internal/insights/scheduler/scheduler.go](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/internal/insights/scheduler/scheduler.go)
[client/browser/src/shared/repo/index.tsx](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/client/browser/src/shared/repo/index.tsx)
[internal/codeintel/resolvers/codenav.go](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/internal/codeintel/resolvers/codenav.go)
[internal/codeintel/dependencies/internal/background/iface.go](https://demo.sourcegraph.com/github.com/sourcegraph/sourcegraph@b432e1daaeeb17489f34180a8617d9aca972cb52/-/blob/internal/codeintel/dependencies/internal/background/iface.go)

Unfortunately I do not have enough context to fully describe the interfaces between Sourcegraph and Cody.

Based on the repositories I have access to, here is what I can say:

Cody is an AI assistant that works inside the Sourcegraph code search and navigation platform. The main interface for users to interact with Cody is through the web chat interface in Sourcegraph.

Some key things I know about the Cody and Sourcegraph integration:

    Cody is embedded directly in the Sourcegraph web application and accessible through the "Cody > Web Chat" menu. This allows users to chat with Cody to get help with their code.

    Cody has access to view and read files from repositories that a user has access to in Sourcegraph. This allows Cody to answer questions about code snippets and files directly.

    There is likely additional integration to allow Cody to perform actions like generating code snippets, explaining code, etc. But without more context into the exact integration code I cannot describe the technical interfaces enabling those actions.

To fully describe the integration interfaces between Cody and Sourcegraph, I would need access to additional repository context focused specifically on the Cody integration code within the Sourcegraph codebase. Let me know if you can provide access to additional repos and I'm happy to further describe the integration architecture.

---

[dhdiemer]

@chillatom - i am fairly confident that I can re create this consistently in VS Code by including multiple repos for context

[kalanchan]

@dhdiemer can you show me an example of what you're asking and the repos you selected in VS Code? I want to reproduce it on my end and investigate a couple things to help us fix this

[sqs]

@jtibshirani Let's look at this tomorrow and repro it e2e.

[alexfogg]

Link to customer thread, where the issue is also happening on JetBrains.