Open dfwilbanks395 opened 1 year ago
One set of errors to fix:
Error in replication, will retry after timeout error=Error during RPC (replication_offsets): Could not find domain 2.0.0 on worker timeout_sec=1
These occur when restarting a ReadySet instance because the replicator and domains initialize in different tasks, so the replicator may look for a domain before it is ready.
Ideas:
Deal with log cleanup from a top-down perspective. Look at a typical run, see the WARNINGS and ERRORS and make sure that those are correctly leveled.
https://readyset-workspace.slack.com/archives/C01HF1WTP6Y/p1695663991991359 this is probably the highest ROI log tweak atm.
There are warn!() and error!() statements throughout our code that convey a severity that we don't intend. There are also likely some cases where we emit logs more aggressively than we should. We should spend a couple days identifying and fixing as many of these cases as possible.