Open teskje opened 3 months ago
I see two general ways to fix this:
PartitionedComputeState
, both of which need to collect and sort errors to find the minimal one.Pick errors in a non-random way instead of randomly. For example, the compute protocol could define that the smallest error, according to the lexicographic order on strings, must be returned.
This seems like the way to go to me!
What version of Materialize are you using?
main
What is the issue?
A general assumption of the compute protocol is that all replicas receive the (semantically) same stream of commands and return the (semantically) same stream of responses. This is currently not the case for dataflow errors returned in
PeekResponse
s,SubscribeResponse
s, andCopyToResponse
s.Reproduction:
The issue is caused by our current policy to return a random error in a peek response, if the peeked collection contains multiple errors. Each replica does so and there is nothing that would ensure that they all pick the same one. When all replicas have the same worker count, compute's determinism ensures (should ensure?) that things work out, but that's not the case when replicas have different sizes.
It's not clear if this non-determinism causes any negative consequences or even correctness issues. Seems like a good idea to fix it in any case.