jlewi
commented
1 month ago Here's a more detailed look at what we need to do for logging
Gaps In Current RunMe Logging
Here are the current limitations of logging in RunMe for the purposes of supporting Foyle retraining
- Server logging isn't always enabled when using VSCode
- When server logs are enabled, server logs aren't persisted to files; they are echoed to an output terminal in VSCode
- This means they are ephemeral and aren't easily accessible by Foyle for retraining
- Server Logs don't record the actual executed command at info level
- request is logged at debug level here
- However, I don't think it would be logged using the JSON serialization format of the proto which would make deserializing it difficult
- Per StackOverflow it won't be seralized as the proto JSON format
Proposed Changes to RunMe Code Base
Server Changes
-
Server should setup a second logger to always log to a JSON file
- We can use a tee logger as in Foyle to create a logger that sends messages to the console and to the JSON file
-
The logs should be stored in
${CONFIGDIR}/logs -
Add a command line flag to the serve command to enable Foyle logs
-
Add a configDir flag to the serve command
-
Execute Logger should be keyed by known_id and known_name is present
-
We need to log the Execute request at info level in JSON format StackOverflow
- An issue here is how to log it in proto JSON format
- The simplest thing to do is to just add code to serialize the request to JSON and then log the result
- The alternative is to to use the proto plugin go-proto-zap-marshaler to auto generate LogMarshalObject methods for all protos
- That only makes sense if there's going to be widespread logging of protos in JSON format. Since we only need to update a single log message right now I don't think its worth introducing it as part of this change
vscode-runme Changes
I think the main changes we'd need to make to RunMe's vscode extension is to plumb through the server changes
- We should introduce a Foyle experiments flag to enable Foyle
- Update start in runmeServer.ts to pass along the appropriate command line flags if the Foyle experiment is enabled
@sourishkrout Could you PTAL and let me know if this looks good or if you have any suggestions before I get started on implementing the changes?
sourishkrout
Per the discussion in discord; we'd like to experiment with integrating RunMe with Foyle's AI capabilities. Foyle aims to be a DevOps copilot. One of the key problems its tackling is trying to train an AI copilot to be an expert in your infrastructure. Foyle relies on a notebook like experience to collect implicit human feedback which is used to train the AI (blog post). Like RunMe, Foyle is also built using VSCode Notebooks. RunMe is far more mature and has more features than Foyle. So it likely makes sense to reuse RunMe as Foyle's notebook rather than rebuilding a separate stack.
Towards that end, this issue is tracking the minimal implementation to get RunMe to support calling out to Foyle and surfacing the results. The current thinking is they would be two separate extensions to start with and RunMe would use gRPC to communicate with Foyle. A more detailed design is provided in Foyle Tech Note 004.
The first step is designing a shared gRPC service to allow the services to communicate; #573