search

dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes
https://dotmesh.com
Apache License 2.0
539 stars 29 forks source link

#744: guard against popping back into pushPeerState #756

Closed lukemarsden closed 4 years ago

lukemarsden commented 4 years ago

fixed https://github.com/dotmesh-io/dotmesh/issues/744 to some extent

should stop the S3 API returning 503 please try again later when fsm currentState has been bumped back into pushPeerState by the asynchrony in pkg/fsm/fsm_push_peer.go (go func() { ...).

because the updating of push progress is asynchronous wrt returning from pushPeerState function, this at least ensures we don't pop back from active to pushPeerState due to the asynchrony

seen in prod:

time="2019-09-24T12:07:08Z" level=info msg="<transition> ccbf1332-e9da-4e6e-ab3c-e5286aaae685 to active waiting (from pushPeerState running: 0.120 MiB so far, 0.054 MiB/sec, 3.78s ago)"
time="2019-09-24T12:07:08Z" level=info msg="entering active state for ccbf1332-e9da-4e6e-ab3c-e5286aaae685"
time="2019-09-24T12:07:08Z" level=info msg="<transition> ccbf1332-e9da-4e6e-ab3c-e5286aaae685 to pushPeerState running (from active waiting, 0.00s ago)"
time="2019-09-24T12:07:08Z" level=info msg="<transition> ccbf1332-e9da-4e6e-ab3c-e5286aaae685 to pushPeerState running (from pushPeerState running, 0.00s ago)"
time="2019-09-24T12:07:08Z" level=info msg="<transition> ccbf1332-e9da-4e6e-ab3c-e5286aaae685 to pushPeerState running (from pushPeerState running, 0.00s ago)"
time="2019-09-24T12:07:08Z" level=info msg="<transition> ccbf1332-e9da-4e6e-ab3c-e5286aaae685 to pushPeerState running (from pushPeerState running, 0.01s ago)"
time="2019-09-24T12:07:08Z" level=info msg="<transition> ccbf1332-e9da-4e6e-ab3c-e5286aaae685 to pushPeerState running (from pushPeerState running, 0.00s ago)"

however, there's still an issue here where old, stray push updates from a different push could update the state of a current push. that's less likely, and less harmful though.

lukemarsden commented 4 years ago

green pipeline: https://gitlab.dotmesh.com/dotmesh/dotmesh/pipelines/24956