search

ipfs-shipyard / cube

IPFS Cube will help people deploy and manage their own IPFS pinning services on top of existing cheap hardware, or cloud storage.
32 stars 3 forks source link

Cube doesn't query the state of containers from docker daemon #13

Open lanzafame opened 5 years ago

lanzafame commented 5 years ago

I have hit an issue a few times now, where cube treats its db as the source of truth and therefore any changes to containers, whether they be stopped/removed externally or refreshing the instances page before the containers have spun up results in a mismatch in the state of the db and the reality of the state of the containers.

victorb commented 5 years ago

Yeah, hit the same one a couple of days ago and started thinking about, thanks for opening this issue!

My first instinct would be to add some recover functionality, where Instances are checked by the Monitoring component, that can alert when expected Instances are not running, and either kill them off from the DB or recreate them. One issue is that we don't know if the user killed of the Instances manually, or they died. Judging by the targeted audience of Cube, I think we should assume that they disappeared because of errors, not the user manually intervening.

I'm in the middle of refactoring the Instances component (together with adding DigitalOcean as a provider), as the architecture currently in master doesn't fit well when you have multiple providers and you want X with one, Y with another.

After this, I think we can move forward with adding health checks to Instances where if the DB has a instance but the actual provider doesn't (docker/DigitalOcean atm), we go ahead and recreate it.