PITCH: ACE Framework message monitoring

[samgriek]

Proposal for a RabbitMQ Logging Subscriber with Browser-Based UI

Objective:

Develop a robust RabbitMQ logging subscriber that records messages from all topics and provides a browser-based user interface (UI) for real-time log display. The UI will also support viewing archived log files and offer a search functionality to filter logs based on user input.

Logging Subscriber:

Exchange Type: The subscriber will bind to the RabbitMQ topic exchange using a # binding key to ensure all topic messages are captured.

Queue Properties: The logging queue will be durable and non-exclusive to ensure its persistence across broker restarts and accessibility across multiple connections.

Message Handling: Messages will be acknowledged post successful logging to ensure no loss of messages. Exception handling will be robust to handle any processing anomalies without interrupting the subscription.

Web-Based User Interface (UI):

Real-Time Display: As logs are written by the subscriber, they will be streamed to the UI, allowing real-time visibility of RabbitMQ messages.

(FUTURE)  Log Archive Viewing: Users will have the option to select archived log files. Once selected, the UI will display logs from the chosen archive, allowing historical message viewing.

Search Functionality: A search input box will be provided, enabling users to filter currently displayed logs based on the entered text. This will aid in quick troubleshooting and log analysis.

Technical Components & Libraries:

Backend:

Language: Python.

RabbitMQ Client: pika for message subscription.

Web Framework: Flask for serving the WebSocket.

Frontend: TDB - with FE dev resource

Framework: Svelte.dev + Vite

Real-Time Updates: WebSockets (e.g., Flask-SocketIO) for streaming logs to the UI.

UI Components: Material UI and Svelte components.

Tailwind for styling

4. Deployment & Scalability:

The solution will be designed to run locally on consumer-grade hardware. The application's modular nature will allow for future enhancements, including possible cloud deployments, distributed logging.

[lancecarlson]

@samgriek this is looking good! Have you thought about using a database for persistence? Postgres maybe? If the user refreshes the browser, how do we keep the incoming messages?

[samgriek]

Yes, actually I have. After discussing with the team, we didn't believe it was in scope for this feature. This is because we don't have have direct control over the AMQP implementation.

However, RabbitMQ outputs log messages in JSON without any intervention. These logs could be loaded to an RDBMS using a technology like LogStash. In the future feature, the user should be able to search logs for key words and timeframes. However, the Web UI plays a small part in this as it just provides the UI and APIs for working with the logs. When the AMQP system is set up, LogStash can be integrated and a suitable persistence selected. The current proposal is limited to displaying the real-time logs for monitoring purposes.

[lancecarlson]

@samgriek sounds good! Less scope is good!

[daveshap]

Hmm, this seems like it belongs in the architecture discussion board, no?

[thehunmonkgroup]

We're not using the PITCH structure anymore, closing.