improvement to zstor behaviors when some meta backends down

[iwanbk]

Based on #123 discussion, i want to do full check & improvements (if needed) regarding the behavior when some of the meta backends down

• [x] zstor monitor must be started (discussion : #123, PR: #128)
• [x] when the meta backend up, zstor monitor automatically reconnect to that backend (#129)
• [x] zstor retrieve must work if there are >= 2 available meta backends (https://github.com/threefoldtech/0-stor_v2/issues/123#issuecomment-2473189057)
• [x] zstor store must fail if one of the meta backend down (#128)
• [x] reload zstor config using SIGUSR1 must update the meta backends properly #129
• [x] check ReplaceMetaStore behavior #129
• [ ] document these behaviors on the README

[iwanbk]

Hi @LeeSmet What is the plan for ReplaceBackends at https://github.com/threefoldtech/0-stor_v2/blob/b81264c1ed8a1d0a056f95a4189980a8b6f3a2ed/zstor/src/actors/backends.rs#L82

I can see the Handler, but it is not called at all

[LeeSmet]

That's a leftover from old functionality. A long time ago there was functionality to automatically replace backends if they were full/failed. Since the feature was not used and caused a lot of complexity for maintenance, it was removed. The handler is an artifact that wasn't removed properly. If you want it can be removed.

[iwanbk]

The handler is an artifact that wasn't removed properly. If you want it can be removed.

OK, i'll keep it for now as reference.

[iwanbk]

zstor retrieve must work if there are >= 2 available meta backends

tested on both master and #129

[iwanbk]

check ReplaceMetaStore behavior

also fixed on #129 By adding writeable flag in the message

[iwanbk]

reload zstor config using SIGUSR1 must update the meta backends properly

also fixed on #129.

By re-creating the metastore when there is change in the meta zdb backends