search

Logflare / logflare_logger_backend

Logger backend to send Elixir logs to Logflare.
MIT License
88 stars 17 forks source link

Remove cachex dep #34

Closed josevalim closed 4 years ago

josevalim commented 4 years ago

It seems the cachex usage was very simply and that an agent could suffice, so I gave it a try. I may be missing something here though, so feel free to close this.

josevalim commented 4 years ago

Another approach is to remove the BatchCache altogether and simply keep the state inside the HTTPBackend. This way you don't need extra processes and stuff. :)

chasers commented 4 years ago

Yeah we were doing this to minimize ram as much as possible by not stashing stuff in state. I guess this shouldn't get crazy since Logger events get evicted if too many come in too fast.

josevalim commented 4 years ago

If cachex uses ETS, a copy is still made every time you read and write to it, so it should actually take more RAM than the state I think. There would be no copying only if keeping it in the Logger Backend state OR if using persistent_term.

chasers commented 4 years ago

Roger ... okay the only other concern was throughput. We were worried about the genserver becoming a bottleneck. Maybe because the upstream logger gen server runs everything through it, this would not be a concern?

josevalim commented 4 years ago

Right, I don't think it will be a botlleneck but a concern in all of these solutions is that sending the log messages to a separate process or ETS will occur extra copying, both when writing and when reading.

So ideally we would get rid of the process altogether and keep it within the backend state itself. We could still have a separate module to deal with the state and the state could be opaque to the backend, but it wouldn't use another process.

chasers commented 4 years ago

Alrighty, let's do it! 🙏