search

elixir-cloud-aai / proTES

Proxy service for injecting middleware into GA4GH TES requests
Apache License 2.0
7 stars 6 forks source link

Integrate optional task distribution logic middleware #16

Open uniqueg opened 5 years ago

uniqueg commented 5 years ago

Is your feature request related to a problem? Please describe.

One of the main benefits of having TES requests intercepted by proTES is that it allows for the distribution of tasks over a network of TES instances, regardless of whether the workflow engine that emitted the TES request contains such a feature. Through the integration of a task distribution logic, proTES will be able to select the most advantageous TES instance for a given task, according to cost and/or clock time considerations.

Describe the solution you'd like

The proof-of-concept task distribution app TEStribute has recently been developed.

A publicly accessible API service of TEStribute can be integrated by service calls to its single endpoint /rank-services. Refer to TEStribute's documentation for details.

Describe alternatives you've considered

If no publicly accessible TEStribute API service is available, the package can also be imported and used in the following manner (refer to TEStribute documentation for details):

from TEStribute import rank_services

rank_services(...)

Additional context

Note that TEStribute currently works slightly beyond the GA4GH specs, i.e., it makes an assumption of data repository services that is not warranted by their specification and it amends a property in a model of one endpoint and adds another required endpoint to the TES specs, thus making it only with specifically tuned TES implementations for now. For this reason, the use of the task distribution logic middleware must be optional (set a switch and necessary config parameters in the config file.

vaniisgh commented 5 years ago

Would love to try and tackle this (or #37 after this is done) if it's still unassigned or you haven't started work @uniqueg :)

uniqueg commented 5 years ago

Please, go ahead - haven't started! Let me know if you need help

uniqueg commented 4 years ago

The controller handling incoming task requests is in pro_tes/ga4gh/tes/endpoints/create_task.py. This would be the place to inject any middleware. Best is to abstract this a bit by adding a module middleware.py somewhere and then add calling of TEStribute as a function or class...

uniqueg commented 1 year ago

For the time being, we have implemented a (simpler) task distribution logic (which took portions from TEStribute) right in proTES, see https://github.com/elixir-cloud-aai/proTES/pull/119. Integration of TEStribute will be put on hold until further notice (together with #37).