quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
8.25k stars 337 forks source link

Remove "actor are stopping when they detect there are no more messages" functionality #574

Open fulmicoton opened 3 years ago

fulmicoton commented 3 years ago

Right now Actor stop as soon as they detect no one will ever send them a message.

This is done by checking the refcount over their mailbox. Once it reaches 1, it means only their copy (in the actor context) exists and no new message will come.

While this sounds neat, it implies a lot of papercuts to enforce.

For instance,

The suggestion is to remove this functionality and force the chain of actor to send a quit command downstream.

fulmicoton commented 3 years ago

Relying on explicit messages would also have the benefit of not identifying the death of upstream actors as a success and make the pipeline supervision simpler.

Right now a successful pipeline means all actors exited with a success. We could change that to the publisher exited with a success (which is not valid semantic today because the packagers panicking would lead to that too).

That would solve the problem of how do we stop the GC considering it keeps running and prevents reaching the "all actors exited successfully" condition.

fulmicoton commented 2 years ago

Moving to Quickwit 0.5. Also we still need ot check whether this is a good idea or not.