[RFC] Temporarily marking storage backends as "temporarily down"

[hexylena]

We need to have a temporary downtime for one backend in our object store configuration that looks something like the following:

<object_store type="distributed" id="primary" order="0">
    <backends>
        <backend id="files1" type="disk" weight="10">
            <files_dir path="/data/1/" />
        </backend>
        <backend id="files2" type="disk" weight="0">
            <files_dir path="/data/2/" />
        </backend>
    </backends>
</object_store>

I'd like to propose a feature whereby we add an additional configuration option to the <backend /> structure, such as offline="true" which behaves as follows:

• datasets are greyed out (like when permissions are wrong)
• the users cannot click "view"
• jobs cannot be launched on these datasets

[bgruening]

Obviously I like this!

[VJalili]

What happens to the shared data persisted on offline storages?

[hexylena]

It is still there, just temporarily inaccessible to all users (and jobs can't be run on it, etc.) The idea is eventually the object store will be re-attached and then the datasets will be available again.

[natefoo]

Could this be done dynamically, possibly with some sort of health check? Unless Galaxy can be made to reload the object store on signal (or... something else, with uWSGI).

[hexylena]

If it could, a healthcheck sounds brilliant, better than forcing a restart to update it. I would not know how to implement it in a dynamic way though. There is I chance I have the ability to implement it in a static config-y way. (In case someone wants to give me some pointers.)