Send notifications when an object is fetched even if we fail to store it. This prevents a case where a task waiting on a Fetch blocks for longer than strictly necessary (or sometimes indefinitely). Also includes a few related improvements to fetching, including:
retries if we fail to store a fetched object
trigger an active fetch on decide if the payload or VID is missing (rather than waiting for the proactive scanner to discover the missing data)
numerous additional tests
This PR does
Moves notifications from NotifyStorage into a higher level (the Fetcher itself). See 6016137 for further discussion. Deletes NotifyStorage
Moves UpdateAvailabilityData/update from transaction level to data source level, so we can enforce better semantics for notifications. Again see 6016137
Makes update infallible. Instead of failing the entire update when we fail to store one leaf, we retry where appropriate, and eventually move onto the next leaf
Adds retries with exponential backoff when storing fetched objects
Triggers a fetch on decide when data is missing
Adds numerous tests, including regression tests for the deadlock bug and tests for new features (backoff and fetch-on-decide)
Key places to review
Fetching stuff: src/data_source/fetching.rs and src/fetching.rs
Tests for new semantics: src/fetching/provider/query_service.rs
Send notifications when an object is fetched even if we fail to store it. This prevents a case where a task waiting on a
Fetch
blocks for longer than strictly necessary (or sometimes indefinitely). Also includes a few related improvements to fetching, including:This PR does
NotifyStorage
into a higher level (theFetcher
itself). See 6016137 for further discussion. DeletesNotifyStorage
UpdateAvailabilityData
/update
from transaction level to data source level, so we can enforce better semantics for notifications. Again see 6016137update
infallible. Instead of failing the entire update when we fail to store one leaf, we retry where appropriate, and eventually move onto the next leafKey places to review
src/data_source/fetching.rs
andsrc/fetching.rs
src/fetching/provider/query_service.rs