Closed Godley closed 5 years ago
Aight so I've put in the suggestion, I'm hoping this will at least allow us to investigate how this happened in future.
I've then gone back, started a node using the last backup (January 23rd), that gives:
time="2019-02-01T10:32:06Z" level=info msg="Detected my node ID as abd8afff16db76f9 ([172.18.0.7])"
right at the beginning of docker logs dotmesh-server-inner.
Looked in etcd and there's 2 server snapshot entries:
/ # etcdctl ls /dotmesh.io/servers/snapshots
/dotmesh.io/servers/snapshots/79e582b945f41900
/dotmesh.io/servers/snapshots/abd8afff16db76f9
/ # etcdctl ls /dotmesh.io/servers/addresses
/dotmesh.io/servers/addresses/abd8afff16db76f9
zfs says:
zfs get -H guid pool
pool guid 14787429460124929250 -
I then went back to the one before that (early december):
/ # etcdctl ls /dotmesh.io/servers/addresses
/dotmesh.io/servers/addresses/c81987c52d13ede8
/ # etcdctl ls /dotmesh.io/servers/snapshots
/dotmesh.io/servers/snapshots/79e582b945f41900
/dotmesh.io/servers/snapshots/c81987c52d13ede8
zfs:
zfs get -H guid pool
pool guid 12872887580815842038 -
So it seems to change a lot? I'm going to have a look at docker logs dotmesh-server
to check if it's like. Recreating the pool for some reason.
I put in a check which will write the id to a file if it doesn't already exist, and if it does, confirm the contents match what we read from zfs. If they don't it should crash out - commit https://github.com/dotmesh-io/dotmesh/commit/24cb41e39acd5a19d576d16fc6c20ae6ebe6fcde
For now think this will do, followed code paths etc and I can't see anything obviously wrong at either end which would indicate why this is happening. If it happens again, let's dig in.
Dotmesh got into a bad state on dotscience production, wherein it didn't respond to requests (or did with connection refused/timeout). This appears to be down to it attempting to reconcile filesystems which were associated with a different pool...
Not sure how this happened but @rusenask suggestion is to write the ID to a file, so that we're not continually relying on zfs to give us the exact same one every time, and refuse to startup if the ID changed.