search

HarryR / logpipe

Converts web log files between multiple formats
BSD 4-Clause "Original" or "Old" License
4 stars 0 forks source link

Publish and Subscribe with Redis #6

Open HarryR opened 8 years ago

HarryR commented 8 years ago

Redis is widely supported and quite fast as a message queue.

To allow for decoupled processing and easy integration with other systems it would be a good idea to enable logpipe to PUBLISH and SUBSCRIBE to queues in Redis.

This requires the event loop in #4 to be implemented so that I/O to and from remote sockets doesn't block and so that pipelining can be used to make publishing very fast.

HarryR commented 8 years ago

For general usefulness and integration with other programs this is the most useful feature at the moment.

squid.logfile_daemon means you can attach it directly to Squid, print.clfjson provides fast and safe encoding to JSON, and redis.pub means logpipe can offload processing to more specialist daemons (e.g. to collect statistics).