search

Soostone / katip

A structured logging framework for Haskell
205 stars 69 forks source link

Add flushing controls #4

Open MichaelXavier opened 8 years ago

MichaelXavier commented 8 years ago

A good point was raised that we don't do any internal buffering of logs in handle scribe. We may want to provide the user with controls such as how many (if any) messages to buffer, when to fsync, etc. Our elasticsearch scribe actually does some buffering with a bounded queue but this is just a memory usage tactic. Scribes still eagerly send every message as soon as they get it. There may be a heuristic there where its preferable to wait and use the bulk indexing API in elasticsearch.

ozataman commented 8 years ago

Seems like it should be on the backends to do this since I doubt a single buffering strategy will be well suited for all future backends.

MichaelXavier commented 8 years ago

Yeah, katip-elasticsearch sets a good precedence of this: gathers all configuration into a type, exports only the type and lenses to its fields and a default constructor for maximum backwards compatibility when new fields are added. The next time we have to add configuration to the handle scribe (or add a new scribe) we should pull the band-aid off and gather all of its configuration arguments into a type.