finer-grained control over which cells are included in the prompt

[jeisner]

As long as I'm making random suggestions ...

It would be nice to be able to control which cells are harvested for the prompt. A few cases come to mind:

1. Markdown cells were originally intended for private notes that are invisible to the kernel (typically about the results we just got and/or what we're going to try next). In this spirit, I might want to keep some of my Markdown cells invisible to the LLM.
2. "Truncate tokens" is fine as a default strategy. But maybe I'm not sure what it will give me, or would like it to give me something different.
3. Sometimes I might want to try asking the same question a different way, ignoring the previous answer, just to see if it makes a difference.

[jeisner]

For 1., perhaps there should be 2 different Markdown cell types (public and private), just as you currently have 2 different code cell types (system and user). And they could be colored differently. Another possibility would be to distinguish them with some magic string.

For 2., perhaps a cell magic like %%include -1, -3 would do the job? (See issue #2.) It might also be nice if the LLM output somehow indicated which cells were actually included in the prompt (so that you know what truncate tokens did, and can change it).

For 3., something like %%exclude -1 might do the job.

I'm sure there are good alternative designs as well.

[zkolter]

I'll think about how to address these. A few thoughts:

1. I like the idea of commenting/uncomment cells a lot (both markdown or code cells, which can accomplish what you're after in point 3 too, but see below for some additional thoughts). I'll add this. The one thing to think about is the right UI for it. I have found that notebook extensions are bit annoying, in that, for instance, you can't actually add new types of Markdown cells, only new types of code cells. What I could do is add a general kind of metadata, similar to the "cell model" right now, that indicates whether a cell is commented or not. But the only place I would be able to display this is in the status bar at the bottom, and I hate how unobvious these notes are. For markdown at least I could color it differently when displayed, as you suggested, though, and we could also have a an additional code cell type that was a "ChatLLM Comment" or something. I'll think about the best UX for this, but it's a bit limited by what VSCode supports.

2. Truncation is definitely not about fine grained control, and more about just saving money on API costs. For fine-grained truncation, I'm not sure the best way of intuitively displaying which cells were included, without being unobtrusive when you don't want it. I'll think about this too.

3. In addition to the "comment" solution above (can you tell I'm not really a fan of cell magics? 😅), one quick thing you can do right now is just insert the cell above rather than below. The mechanism here is that each completion will use only the content above the code cell, so you can get several completions for the same question this way (I know this requires non-sequential execution, which is probably the benefit of the cell magic design...)

[jeisner]

one quick thing you can do right now is just insert the cell above rather than below.

Good point. Of course, then the notebook becomes less readable as a record of what was tried and why, but I guess you could move the cell back down after it executes!

I'm not sure the best way of intuitively displaying which cells were included

Particularly important if cells are getting moved around as suggested just above ...

Don't know what VSCode supports -- but from a UI point of view, it would be really nice to do this not with text, but just by softly highlighting any other cells that the cursor cell depends on.

That both previews the dependencies before you execute the cell you're editing (when you could still change them through extension settings or magics) and also documents those dependencies after you execute it.

[zkolter]

Ok, got it. I think we can mostly address this by adding: 1) the ability to comment/uncomment cells, and 2) some stored information about exactly which prompts went into the response.

VSCode definitely doesn't give me the ability to change the UI in subtle ways like that, so the way to make this work might be a bit more of a kludge, but how about something like this. I can add a general setting (because I don't think everyone will want this), to have each cell store the exact prompt history that lead to the output (as cell metadata). Then, I can add a separate command to "spawn a temporary notebook" that includes exactly these prompts.

[jeisner]

Sort of like how a diff can be shown in a read-only VSCode tab?
Yes, that's certainly better than nothing, even though it's a shame to have to switch tabs.

I wonder if the read-only tab could update automatically depending on what is selected in the main notebook.