search

readysettech / readyset

Readyset is a MySQL and Postgres wire-compatible caching layer that sits in front of existing databases to speed up queries and horizontally scale read throughput. Under the hood, ReadySet caches the results of cached select statements and incrementally updates these results over time as the underlying data changes.
https://readyset.io
Other
4.25k stars 117 forks source link

Clean up ReadySet logs #265

Open dfwilbanks395 opened 1 year ago

dfwilbanks395 commented 1 year ago

There are warn!() and error!() statements throughout our code that convey a severity that we don't intend. There are also likely some cases where we emit logs more aggressively than we should. We should spend a couple days identifying and fixing as many of these cases as possible.

dfwilbanks395 commented 1 year ago

One set of errors to fix:

Error in replication, will retry after timeout error=Error during RPC (replication_offsets): Could not find domain 2.0.0 on worker timeout_sec=1

These occur when restarting a ReadySet instance because the replicator and domains initialize in different tasks, so the replicator may look for a domain before it is ready.

aglover commented 1 year ago

Related: 493

gvsg-rs commented 11 months ago

Ideas:

Deal with log cleanup from a top-down perspective. Look at a typical run, see the WARNINGS and ERRORS and make sure that those are correctly leveled.

lukoktonos commented 11 months ago

https://readyset-workspace.slack.com/archives/C01HF1WTP6Y/p1695663991991359 this is probably the highest ROI log tweak atm.