Prompt history/transparency?

[tophee]

This is partly a question for clarification and probably partly a feature request. I'm trying to understand how the AI assistant is intended to be used. After it generated the code, I am asked to accept the code. What exactly is the point with this confirmation step? At first I thought that accepting the code would also run the cell, but that is not the case.

The main effect of accepting the code is that the diff-highlighting disappears, which makes sense and perhaps that is the main idea with the confirmation prompt. In that case, my issue may be mainly with semantics of "accepting" the code, because in my mind, I am not accepting the code before it ran without errors. Yet, if I'm not mistaken, I have to accept the code before I can run it.

But if I accept the code, I no longer have access to the prompt (or did I miss something?), which means if the code fails, I have to rewrite the prompt from scratch. - If it fails with an error, I can ask AI to fix it, but if it runs smoothly but does not produce the intended results, I need to re-prompt.

So, to formulate this as a feature request, I would find it useful if all prompts ever executed on a cell would be preserved and accessible through some prompt-history button. Perhaps the most recent prompt could always be used to pre-populate the "Ask AI" text box?

Final point: transparency. While I'm happy to have pretzel give me good code without me having to think about prompting, I would still find it useful to be able to see what exactly was sent to the API, i.e. model (see #86), temperature, etc but especially the context. Is the entire notebook transmitted as part of the prompt? Only the code or also the cell outputs? It looked like there was some embedding going on, but if this is used to select relevant cells as context, it would be useful to know so that I can make sure to explicitly mention terms from those cells (see e.g. #88).

[prasoon2211]

You raise a good point with wanting to see the prompt again after running the code!

The reason why we want you to see the code before accepting is so that you know that the code isn't going to run something you didn't want to run!

For eg, sometimes dataframes I have in memory took a long time to calculate - it would be bad behaviour if the AI code irreversibly modified the dataframe without my knowledge. So, we let users vet the code before running.

If the code isn't to your liking, you can click on Edit Prompt to modify the prompt. However yes, you're correct - you need to run the code to see if it does what you want. And I 100% agree - there should be a way to see once again see the prompt you ran previously. This is a feature we want to build and it's high on our roadmap!

[prasoon2211]

To your second request on the prompts: That's a fair request. While the code is openly available for users to see how the prompt is generated, I agree this might be a good usecase. I will add a setting that when true, shows the prompt sent to the AI in the javascript console.

For now - we have a custom prompt calculated in this file: https://github.com/pretzelai/pretzelai/blob/main/packages/pretzelai-extension/src/prompt.ts

We currently send the user input along with some code in the notebook that matches the prompt.

[tophee]

The reason why we want you to see the code before accepting is so that you know that the code isn't going to run something you didn't want to run!

But the code is not run immediately after accepting, right? But I still get the point. Here is a thought: I realize that it was not clear to me from the beginning, that the suggested code is not actually pasted into the cell but into a separate "cell". I wonder whether these two levels of content could be made more intuitive by putting the AI response into a modal? If that makes sense, the prompt should probably also be in a modal, though I see that this might limit the users ability to navigate back and forth in the notebook while writing the prompt or checking the response.

But I think it would be better to keep the AI interaction separate from the notebook-cells. The obvious option would be to put it into the sidebar on the right (just like the "contextual help". Having access to the chat history in the sidebar would also solve the problem of losing previous prompts. The downside might be that if it looks too much like a classic chatgpt conversation, it might make users think that the AI is aware of this history, which, of course, it isn't. So, that would have to be considered when designing the interface. Maybe put each prompt and response into a distinct box? And once the code respinse is accepted, the response could be minimized, or folded away somehow so that what is shown is mainly a list of previous prompts.

BTW: it has happened to me multiple times today that I wanted to accept the code, except for a few little changes (I think it was some deletions I wanted to comment out instead of deleting), so it would be great if the response was editable before accepting.

[prasoon2211]

But the code is not run immediately after accepting, right? If you click the "Accept" button, the code goes from the diff editor into the cell and is ran.

Modals are an interesting choice here but - we've tried it in some other tools and it's very clunky. Also, we want each AI prompt box + the generated code to be tied to each cell itself.

The reason why we put the new code into a separate area under the cell is because it's not an ordinary cell - it's a Diff cell. So, when you edit code in an existing cell, you'll actually be able to see the diff of your old code vs new code:

As for the sidebar - that's high the priority list of things for us to work. We're using the Cursor model as inspiration - Cmd+K is for generating code in the notebook cells and the sidebar is for asking questions, generating code etc but again, with context. See this screenshot

[prasoon2211]

EDIT : This has been added in 4.2.2

it has happened to me multiple times today that I wanted to accept the code, except for a few little changes

Right, I've felt this problem myself when testing - here's a solution we're thinking - we'll have two buttons

1. Accept and Run (which is the behaviour of Accept right now) - it accepts the code and runs is in the notebook. Also invoked with the Shift+Enter key (my most common usecase right now - I simply want to run the generated code)

2. Accept button - it only takes the generated code into the Jupyter cell so you can edit it before you run it. Invoked with Enter.

I think this would solve most of these problems

[Emasoft]

Some humble suggestions:

• You should add a "history" button near each cell, so that the user can see the list of all versions of the cell ordered by date, and quickly (using up/down arrow buttons for mobile browsers and keyboard arrows keys for desktop browsers to browse the list and see the cell updating displaying each version).
• Also every cell should have a button to quickly switch between diff view and normal view with a touch or click of the mouse).
• Add a 'Revert to this version' button, so that once a certain previous version is found in the history, just a click will revert the cell to that content.
• You must be always able to revert back to the last version. If you use git under the hood this should be automatic.
• If a cell has unit tests, you should automatically revert the tests to the correspondent version used by the previous cell.