Add throttle/bottleneck control to services

[Baeist]

We need some kind of way to control the flow of assets. Either we can restrict the total amount of assets in the system or we can try and control the total mass of the assets in the system.

Reasons for this has to do with total storage space, erda sync wait times and hpc capabilities.

Be aware of cropping and derivatives create more assets and take up more storage space. It is also possible for assets to enter the system from outside of the integration server. Maybe there need to be some kind of buffer to take all this into account.

[Baeist]

Leaning towards each service counting the assets they send through and keeping track in the service database and then checking the service that comes after them and if there are too many assets there pausing. Would require a max number of assets allowed field in the micro service config.

Some services could be exempted from this like the update_metadata in storage updater since it does not rely on anything else.

Still need to figure out a good way to have this interact with hpc and their queue system. Derivatives also.

[Baeist]

Setting both a max amount and a total max size of assets in the system seems good. Possibly also a max jobs queued, not sure this is needed though.

Considering setting up an api to receive updates when assets enters or leaves the pipeline. Writes to a db.

Throttle service would monitor database for too much/many assets- when threshold is reached would set throttle status for certain services run_status, they would then not start handle assets until throttle service changes their status again. Not sure how easy it should be to go back to normal again. Maybe wait for half capacity before resuming.

Initial points of throttling could be create asset ARS, sync erda, open share, create assets and start jobs on hpc.