dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes
https://dotmesh.com
Apache License 2.0
539 stars 29 forks source link

internal NATS server stopped or client got disconnected #726

Open rusenask opened 4 years ago

rusenask commented 4 years ago

On one dotscience runner I saw errors such as:

time="2019-07-26T09:05:14Z" level=error msg="[globalFsRequestId] failed to subscribe for responses" error="nats: connection closed" filesystem_id=47074cdc-0a17-4f79-a841-1e9be9ce8312 request_id=b9417ad9-047b-4610-bef4-3c2962922941

Couldn't find any more useful logs, need to ensure that if the server for some reason panics/stops we restart that goroutine, same for the client - if it disconnects, we should reconnect it (although I think it has auto reconnect anyways).

Temporary fix:

Restart dotmesh-server-inner container