Add throttle/bottleneck control to services

[Baeist]

We need some kind of way to control the flow of assets. Either we can restrict the total amount of assets in the system or we can try and control the total mass of the assets in the system.

Reasons for this has to do with total storage space, erda sync wait times and hpc capabilities.

Be aware of cropping and derivatives create more assets and take up more storage space. It is also possible for assets to enter the system from outside of the integration server. Maybe there need to be some kind of buffer to take all this into account.

[Baeist]

Leaning towards each service counting the assets they send through and keeping track in the service database and then checking the service that comes after them and if there are too many assets there pausing. Would require a max number of assets allowed field in the micro service config.

Some services could be exempted from this like the update_metadata in storage updater since it does not rely on anything else.

Still need to figure out a good way to have this interact with hpc and their queue system. Derivatives also.

[Baeist]

Setting both a max amount and a total max size of assets in the system seems good. Possibly also a max jobs queued, not sure this is needed though.

Considering setting up an api to receive updates when assets enters or leaves the pipeline. Writes to a db.

Throttle service would monitor database for too much/many assets- when threshold is reached would set throttle status for certain services run_status, they would then not start handle assets until throttle service changes their status again. Not sure how easy it should be to go back to normal again. Maybe wait for half capacity before resuming.

Initial points of throttling could be create asset ARS, sync erda, open share, create assets and start jobs on hpc.

[Baeist]

Set up max total asset size check and max asset count being synced so far.

Need to split up the amount of storage space being assigned in total. If not then the initial asset creation will eat up all the space first. This means there has to be designated space for derivatives.

[Baeist]

Have split the storage space into original and derivative space. The check and updating of values is handled by the micro services. A full service that actually changes status and is the sole keeper and updater of this can come later if needed.

[Baeist]

This will act as our way of knowing how much space we have available on ARS/fileproxy for now.

From the throttle db: dev_db> db.throttle.find() [ { _id: 'max_assets_in_flight', value: 0 }, { _id: 'max_sync_asset_count', value: 0 }, { _id: 'total_max_asset_size_mb', value: 0 }, { _id: 'total_max_new_asset_size_mb', value: 0 }, { _id: 'total_max_derivative_size_mb', value: 0 } ]

Max assets in flight is not being used. The others are and the max values are set in the integration servers config files.