Fiber

[AThilenius]

Fiber (the streaming text service) needs to be reimplemented. I want to be really careful with this one, it's important to get this one right. It needs to do two important things:

• Archive closed streams to DB
  • This will allow previous streams to be read, even if they have been closed
• Socket IO Streaming
  • Web Sockets to stream data as fast as humanly possible to clients. Socket.io wraps all this crap for us so we don't need to think about it.

The challenge is in storing the stream history. This is essentially a pub-sub system with history, step 4 of this article talks about how to do it with Redis. We will probably use Redis for this because it's such a good fit. But very old runs need to be pulled out of Redis and flushed to the DB and Redis is limited in size by how much memory the machine has.

Note for later: There is a potential race condition in 'pull history', 'observe changes'. To avoid this, we need to observe the stream first AND get a sentinel token from the server of where it is at in the stream (by ID). Only then are we safe to pull history from 0-N where N was given to us by the server.

[AThilenius]

Starting exploration on ways to implement this. Please feel free to add to the conversation.

[AThilenius]

Logical problem; the old model for Fiber doesn't actually work any more because a single container lives "forever", so a stream is never closed. Meaning we have only one, never ending stream to/from the container. I'm then inclined to ditch the idea of persistent streams and think of this more as an 'SSH' sort of thing. However, keeping the last N entries would be nice incase the person has a long running job and closes their laptop while it finishes.

I then propose:

• Each user is given exactly one bi-directional stream. Later we can add namespaces for real-time communication for other components apart from just console
• A "push" into the stream from either direction does two things
  • Emits the message immediately over Socket.IO. This will only get to the client if they are actively connected.
  • Saves message to the user to a Redis cache. Only stores the last k10 messages (could be a problem with very large / very tiny messages)
• The user can push a "Give me the last N entires" request into the stream and that will result in the changes being streamed (as if they were happening one after another) to the client. This is limited by the last k10 entires. Nothing else will be streamed until these have finished (eliminating race conditions)

Depends on #22

[skycoop]

This all makes sense to me.

[AThilenius]

Design Choices.

• Socket.IO for all real-time communication to clients
• Redis for cross-process communication, that is, communication from the Billet Manager to it's Hosts.
  • socket.io-redis and socket.io-emitter make this very easy
• Redis for storing the last k10 entries, using the JavaScript API directly

[AThilenius]

We'll this all got thrown out the window with tty.js