Log buffering next steps

[havocp]

Here's my suggestion as a first cut.

Short-term goals:

• clients get a reasonable batch of currently-interesting logs on connect, especially errors from boot and whatever the current execution is doing
• don't use TOO much memory
• LogEvents are properly ordered, that is, taskId-tagged events are between the TaskStarted/Finished events, and logs are kept in the order they happened.

A possible way to do it:

• punt on saving to files for now; we can use buffers with a cap on size to ensure nothing entirely insane happens
• at all times save the following buffers of log events, separately with capped size on each buffer:
  • all log events for each taskId which has been started but not finished (drop the whole list of events when the task finishes)
  • all log events without a taskId since the last execution (we reset this each time an execution completes). So on boot before anything executes, it will have the boot logs.
• when a client connects, first send it all no-taskId log events we have; then each time we send TaskStarted (during initial sync), also send it the log buffer for that task
• don't make the first-to-connect client special anymore; logs are saved until tasks ends or until a new execution finishes, so new clients always get everything for the current execution-in-progress

Future elaboration to punt for now, because it requires fancy file-based solution:

• remove caps on log sizes
• save previous execution/task logs
• save logs of already-finished tasks from each execution
• give clients control over which logs to get (e.g. request logs for a task on-demand when user expands that task in the UI)

[jsuereth]

Have you seen our TaskStreams API and how logs are handled there? If not you need to look into it, as we should be UNIFYING the core sbt as we do this, rather than creating duplicate systems.

TaskStreams is designed to store logs in a file stream and restore them on demand, as well as other aspects of data. I have a few experiments to refactor this API to be 100% stream of events, based model, but this is where you should be tackling things, especially if you're talking about saving/resotring previous execution results.

Do you have time for a call, possibly tomorrow or wed on this?

[havocp]

Yep I am around. fwiw I am not talking about a "system" here - I'm talking about a Map[Long,Seq[LogEvent]] private to ReadOnlyServerEngine ;-) that's the point about short term vs. future elaboration. Changing the standard sbt facility is probably an "N weeks" tasks and the proposal I made here is a probably an "N hours" task. If we did the weeks task then we'd drop the Map[Long,Seq[LogEvent]] in favor of a call into the new standard log buffers facility.

[jsuereth]

Yeah, it's not just about log buffers, but I get your point. Another band-aide is ok for now, but I think we may need to clean-up before launch. The logging subsystem of sbt is barely functional in the sbt-server, and things like this are destined to be issues soon.