Finally we'll be able to let a server running all the time, collecting celery events, independent of any connected clients! This will allow to use Clearly as an actual flower substitute.
The software architecture is quite different now. We have a threaded EventListener that captures raw celery events, storing tasks and workers in the default LRU memory, and converting to an immutable format before passing up to the next phase via a Queue.
Then there's the StreamingDispatcher, another threaded processor, that maintains connected interested parties and shares events with them. For each new event, it tests whether a client would like to see it, and if yes it generates the missing gaps in states and sends them to the client Queue.
Finally there's the ClearlyServerServicer, a gRPC server in a ThreadPoolExecutor, that accepts connections and redirects to: the streaming dispatcher if a realtime capture was requested, or to the listener memory for already persisted events. Of course there is the new ClearlyClient, which does not use threads anymore or any server resources like broker or celery app, instead it uses a stub to connect to the server host:port via gRPC.
The code is way better, way more testable, and now all modules has 100% code coverage!
There's also a few small bugs fixed, and better tools (makefile) to support the project.
Finally we'll be able to let a server running all the time, collecting celery events, independent of any connected clients! This will allow to use
Clearlyas an actual flower substitute.The software architecture is quite different now. We have a threaded
EventListenerthat captures raw celery events, storing tasks and workers in the default LRU memory, and converting to an immutable format before passing up to the next phase via a Queue.Then there's the
StreamingDispatcher, another threaded processor, that maintains connected interested parties and shares events with them. For each new event, it tests whether a client would like to see it, and if yes it generates the missing gaps in states and sends them to the client Queue.Finally there's the
ClearlyServerServicer, a gRPC server in aThreadPoolExecutor, that accepts connections and redirects to: the streaming dispatcher if a realtime capture was requested, or to the listener memory for already persisted events. Of course there is the newClearlyClient, which does not use threads anymore or any server resources like broker or celery app, instead it uses a stub to connect to the server host:port via gRPC.The code is way better, way more testable, and now all modules has 100% code coverage! There's also a few small bugs fixed, and better tools (makefile) to support the project.
Achievement unlocked: 100% CODE COVERAGE