Move AI Logging Of Execution Into VSCode Frontend and out of RunMe GRPC Server

[jlewi]

Background

Foyle relies on logs of cell execution for feedback and training. Cell execution is currently logged from the RunMe GRPC server and not the vscode frontend. See stateful/runme#585. I think the backend was chosen over the frontend because I know GoLang much better than type script.

Logging from the backend has a number of disadvantages.

1. Complicates user setup
2. Can't collect additional types of feedback.
3. Complicates event processing
4. Requires AIService and Executor to be co-located.

User setup

Right now the user has to configure Foyle with the location of the RunMe logs (docs). This is brittle and friction. Now that learning is working pretty well and critical to Foyle's value proposition I'd like to remove this friction.

Can't collect additional types of feedback.

As noted in Tech Note 008 there are valuable events that we'd like to log Accepting/Rejecting AI suggested cells. These events would need to be logged from the front end and not the RunMe gRPC server.

Complicates event processing

Processing AI events for learning is complicated by the fact that Foyle needs to be aware of log management in RunMe. RunMe can launch multiple instances of the RunMe server each with their own log file. Foyle has to monitor all the log files and doesn't necessarily know when a log file is complete and will no longer have events being written to it.

Requires AI Service and Executor to be co-located

Since Foyle needs access to the RunMe executor logs the services need to be colocated.

Proposal

The proposed solution is to introduce an RPC into the Foyle AI service to LogEvents.

rpc LogEvents(LogEventsRequest) returns (LogEventsResponse) {}

Events can then be logged from the frontend via RPC.

1. We can instrument _doExecute to log executions
2. We can instrument handleOnDidChangeActiveTextEditor to log ghost cell acceptances
3. We can instrument processResponse to log ghost cell deletions

With this change we can turn on logging by default without requiring any additional configuration from users. The only thing they would need to configure would be the Foyle endpoint.

@sourishkrout what do you think? Any suggestions or concerns; particularly about the changes to the frontend?

[sourishkrout]

@sourishkrout what do you think? Any suggestions or concerns; particularly about the changes to the frontend?

Yes, when we initially considered porting Foyle on top of Runme, I also considered a more UI-aware approach first. However, it was a good call to lean into strength to PoC the integration. I'll review https://github.com/stateful/vscode-runme/pull/1589 for feedback.

One terminology I mildly object to is the term "frontend," although it's no objection to the proposal. I find that thinking about an IDE, even if a stripped-down version that could run a backend-less "frontend," minimizes the IDE's role and its importance. While it's a longer-term goal of Runme to have the ability to run as a web extension (in a way that makes sense in browser-only), referring to it as "frontend," factually, it is not "just" a frontend app. My web development knee-jerk is over-indexing on its meaning in the web world. Anyways, just a side note, really.

[jlewi]

One terminology I mildly object to is the term "frontend," although it's no objection to the proposal. Ack. Do you have a preferred terminology to distinguish between running in the vscode extension vs. the grpc server?

[sourishkrout]

One terminology I mildly object to is the term "frontend," although it's no objection to the proposal. Ack. Do you have a preferred terminology to distinguish between running in the vscode extension vs. the grpc server?

Usually use "extension" for the whole concept and UI/UX (notebook, editor, terminal, ...) for literally the chrome that's facing the user.

[jlewi]

• [x] Merge Front end changes stateful/vscode-runme#1589
• [x] Merge backend #222
• [x] Merge RunMe gRPC server cleanup stateful/runme#661
• [ ] Create PR to remove the logging option in RunMe VSCode extension
• [ ] Create PR to remove the logging option in RunMe gRPC server
  • We should probably do this after one version release of the VSCode extension with the aiLogging option removed in the extension so it won't try to set the flag when starting the server

[jlewi]

Frontend should have released https://github.com/stateful/vscode-runme/releases/tag/3.7.5 So now we just need to create a PR to remove the logging option.