search

continuedev / continue

⏩ Continue is the leading open-source AI code assistant. You can connect any models and any context to build custom autocomplete and chat experiences inside VS Code and JetBrains
https://docs.continue.dev/
Apache License 2.0
18.05k stars 1.42k forks source link

Json files using the @ RAG routine are being evaluated prior to being presented to the model. #1392

Open SynFinAck opened 4 months ago

SynFinAck commented 4 months ago

Before submitting your bug report

Relevant environment info

- OS: Windows 11
- Continue:0.8.36
- IDE: Version: 1.89.1 (system setup)
Commit: dc96b837cf6bb4af9cd736aa3af08cf8279f7685
Date: 2024-05-07T05:13:33.891Z
Electron: 28.2.8
ElectronBuildId: 27744544
Chromium: 120.0.6099.291
Node.js: 18.18.2
V8: 12.0.267.19-electron.0
OS: Windows_NT x64 10.0.22631

Description

Json files using the @ RAG routine are being evaluated prior to being presented to the model.

When you use the @ symbol to select a file to present to the model, if you chose a json file the text will not be sent to the model only '[object Object]' is sent.

To reproduce

  1. Open a Json file
  2. In Continue with any model use the @ symbol in chat and select the json file you are trying to reference.
  3. Ask the model to display exactly what it receives from you inside a code barrier. If the model is not hallucinating it should result in a similar answer to the following: 
    It seems I've encountered an unusual situation with your provided config.json file. The output is actually a JSON object, which means the [object Object] representation you see inside the code barrier is the default output of most languages when working with this data structure type.

Log output

No response

sestinj commented 4 months ago

@SynFinAck you're right! Looks like we're serializing/deserializing in not the right way

SynFinAck commented 4 months ago

Cool glad It wasn't just me :-)

Hey, on another note, I have some Ideas for Continuedev that I think might help the models follow the conversation better.
I have noticed some behavior that might have to do with stale RAG calls to a particular file. It appears that the model doesn't care to update that file definition if it has changed since you passed it to it. It keeps a cached version of the document or code or whatever, but only sometimes, as other times, it updates perfectly. I wanted to see if we might be able to edit the model's responses to help it infer correctly even after it messes up. You could easily correct one of its code suggestions to make sure it follows that format from now on, etc. I was also hoping it might be possible to introduce another window or prompt that will display the current model's thought process in a summary format. This could appear in a small window in a horizontal tree format similar to the chat log; every time the model changes its intention, it will get updated with an associated summary of intentions. I hope we can redirect the model via this window to an old thought process or even a new one if we monitor and/or modify the summary the model makes to describe its thought process. I am not sure if that makes sense really, let me know if you are curious or have questions?

sestinj commented 4 months ago

@SynFinAck I think these are good ideas. The first one I happen to have thought through a bit ago, and it seems great until you realize that editing the file means that the range could have shifted, and then we have the extremely difficult problem of matching the old range to a new one that is supposed to be equivalent

I like the second idea a lot, and it will probably come into play when we have more complex agent behaviors going on

SynFinAck commented 4 months ago

I am really glad you like that concept. I was trying to implement my own vscode client prior to finding this one. I also noticed you guys are Hiring. I seriously have tons of ideas how to expand this client as I was trying to make my own. In all seriousness it's a rough market out there and as fast as things seem to be moving in this industry, I believe that it's these types of real interactions that will hopefully get me past the application stage. (Well hopefully). Any chance I can hop onboard and potentially get paid to help 😁. If you know who I might be able to talk about this with I would be very appreciative.
I wasn't going to talk to much here as to keep this for its purposes, so you may notice a LinkedIn chat message from me.