search

jlewi / foyle

AI For Software Operations
https://foyle.io
Apache License 2.0
75 stars 7 forks source link

Support for non-bash Runme notebooks cells #143

Open sourishkrout opened 3 weeks ago

sourishkrout commented 3 weeks ago

As a precursor to Grafana or other forms of embedded custom renderings, Foyle <> Runme need to support non-bash interactions. In this case, I've used the existing GCE renderer which, in principle, is just a custom renderer.

The problem here is "simple": After correcting the response to my first prompt, the "ground truth" does not appear to be retained by Foyle. I could be wrong but glancing over the logs, my hunch is that "custom renderers" aren't processed the same way vanilla bash is. It for the most part, skips the Kernel's Runner and makes an API call to the Cloud Provider.

Search for stateful.runme/gcp in https://gist.github.com/sourishkrout/e871d5b182ca16bff6eaaa374aaf532d to see how non-bash (custom MIME/media types) are currently processed as if they were bash.

Happy to provide more information. Figured I'd file this issue to put it on your radar, @jlewi, before I forget.

Notebook used: non-bash.md

jlewi commented 3 weeks ago

@sourishkrout I think your right. If IUC correctly renderers work today by creating a code cell with particular type of language and pasting some form of link in it e.g. "https://console.cloud.google.com/compute/instances?project=runme-ci".

How do RunMe renderes work? in particular where does that link get turned into an output cell? Does that happen entirely in the vscode extension or is it hitting the gRPC server?

To collect feedback from RunMe we instrumented the Execution service. In particular its this line. https://github.com/stateful/runme/blob/4ee8c4c3b335ba981172dc8cdb36a8b0db9eebfa/internal/runner/service.go#L232

I suspect if your using a renderer you aren't calling Execute. So we probably need to come up with a different mechanism to log that feedback.

sourishkrout commented 1 week ago

I suspect if your using a renderer you aren't calling Execute.

That is the gist of the problem. More elaborately, the Runner would be called ShellRunner, which excludes any URL/API-based executions for cloud resources. However, I want renderers to become more integral to the Runme Kernel. Let me mull over it. That way, the emission of learning logs could remain exclusively on the kernel server side.

Independently on how to solve this, it'd be good to understand the learning logs/data gap first to work backward.