Currently, UserALE bundles logs into a buffer and flushes the buffer on a regular interval or in response to certain event triggers before POSTing them to the REST API url. This works well in most circumstances, but adds unnecessary network overhead. Also if requests are dropped we have no back up mechanism.
Solution
Add support for websocket connections if the backend REST API supports them. Upon first connection to the back-end, we would check to see if web sockets are supported, if so, we would upgrade the connection.
This achieves two things:
Minimizes network overhead
Adds redundancy
We would still store logs in a client-side memory buffer and flush them over the socket. However, the socket reduces network overhead. And if the connection ever gets dropped before the buffer has been flushed, then we can concurrently work to re-establish the connect while continuing to store event logs in the buffer. Upon reconnect, you flush all logs to relieve back-pressue. In this way, no logs are ever dropped. As an escape hatch, if the buffer exceeds a certain memory threshold, then you can treat it like a FIFO queue and start dumping old logs to avoid a memory leak.
rc10house
commented
3 weeks ago
Problem
Currently, UserALE bundles logs into a buffer and flushes the buffer on a regular interval or in response to certain event triggers before POSTing them to the REST API url. This works well in most circumstances, but adds unnecessary network overhead. Also if requests are dropped we have no back up mechanism.
Solution
Add support for websocket connections if the backend REST API supports them. Upon first connection to the back-end, we would check to see if web sockets are supported, if so, we would upgrade the connection.
This achieves two things:
We would still store logs in a client-side memory buffer and flush them over the socket. However, the socket reduces network overhead. And if the connection ever gets dropped before the buffer has been flushed, then we can concurrently work to re-establish the connect while continuing to store event logs in the buffer. Upon reconnect, you flush all logs to relieve back-pressue. In this way, no logs are ever dropped. As an escape hatch, if the buffer exceeds a certain memory threshold, then you can treat it like a FIFO queue and start dumping old logs to avoid a memory leak.