search

dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes
https://dotmesh.com
Apache License 2.0
538 stars 29 forks source link

If initial pull of a dot fails, dotmesh ends up in a weird state where it can't pull again #628

Open lukemarsden opened 5 years ago

lukemarsden commented 5 years ago

User requirement

As a dotmesh user, I want retrying failed pulls to work so that I can get my data.

In particular, one node had a case where it ran out of disk space (probably) during an initial pull of a dot.

Trying to pull again results in:

time="2019-02-14T14:48:59Z" level=info msg="Starting dot pull" dot_name=project-1d80f936-default-workspace dot_namespace=kate4 hub
=cloud.dotscience.net hub_username=kate4 local_name=ds-BLFBRY5NMDQBLHVSZFEFLIW2L5AOWSUGN2KI4OSUMGPCU65HFVJQ transfer_id=972de317-e
6a4-462f-5bc6-cbbae401a6be
time="2019-02-14T14:48:59Z" level=info msg="Watching pull queue" still_pulling=1
time="2019-02-14T14:49:00Z" level=info msg="Transfer status polled" elapsed_ns=0 index=1 message="Response event != finished-{push
,pull} or peer-up-to-date: <Event no-snapshots-of-remote-filesystem: filesystemId: \"fd7e74cc-9449-4734-7b24-fc7e29336484\">" sent
_bytes=0 size_bytes=0 status=error total=1 transfer_id=972de317-e6a4-462f-5bc6-cbbae401a6be
time="2019-02-14T14:49:00Z" level=error msg="Error fetching dots" error="Error performing transfer: Response event != finished-{pu
sh,pull} or peer-up-to-date: <Event no-snapshots-of-remote-filesystem: filesystemId: \"fd7e74cc-9449-4734-7b24-fc7e29336484\">"

One theory is that the state machine got set up, but never ended up with any snaps. It should be able to pull again in this case...

Key acceptance criteria

lukemarsden commented 5 years ago

Looks like the hypothesis is wrong!

root@kate-big-vm:~# dm list
Current remote: admin (use 'dm remote -v' to list and 'dm remote switch' to switch)
  DOT                                                      BRANCH  SERVER            CONTAINERS  SIZE       COMMITS  DIRTY
  ds-BLFBRY5NMDQBLHVSZFEFLIW2L5AOWSUGN2KI4OSUMGPCU65HFVJQ  master  8e62552f501d8da8              3.15 GiB   3        -  
  ds-GJ5MF6GH5A4KBIPP5QQQDCLQGNBHZWUT5C5TTD3SY4IJTO27Y3JA  master  8e62552f501d8da8              3.15 GiB   3        -  
  ds-SJMRSOXPEZUOV6H6B3U7E4QAQI4HBQ4R5AZXELG657DDWBMDY7XQ  master  8e62552f501d8da8              26.00 MiB  5        -  
  ds-UTCOW3UAFH7LEG4MM3LKPYUGHD7BKYM6RFRL3UKNFPCB4TSGXWNQ  master  8e62552f501d8da8              3.33 GiB   7        -