[DISCUSSION] Implement Queue System and Background Task Processing on Galaxy-API

[kshitijrajsharma]

Plan is to implement queue system , Send API request back within seconds and start background task for respective request if request is heavy and takes time ! This will involve better architecture discussion how we can do that in very efficient way , Opening this thread to invite ideas and discussion

[NicolasGrosjean]

Do you have some idea of the waiting time for heavy requests ?

I found this type of workflow with Celery and Redis.

Source (and maybe a good tutorial, I have not tested it)

If the running time is very big, we can imagine send an email to the user when the request has been finished like BBBike extracts do it.

[kshitijrajsharma]

@NicolasGrosjean ,Thank you for your input ! It could be very effective solution. Request time will vary on the basis of payload ,it will be safe to assume process will be cpu intensive and may take half an hour as well . I have used cerely +redis before ! one interesting thing about that article is he recommends flower for monitoring ! I have tested the tutorial he provided , I will try to evaluate the performance as well . We are using dramatiq + redis in one of our other tool , exploring that option as well . May be another option to look into will be rabbitmq + dramatiq ! Email option is effective , IMO i believe it should be setup in the way user want on their UI or their usecase . Right now request coming from export tool will be receiving email done by export tool but request coming to api directly they need to track specific task ! API should be able to provide the status of individual request , job should be trackable with appropriate response to user on different state of task , for eg : if it is finished along with the download url / API response should include all info !

[kshitijrajsharma]

Supporting doc for this discussion : Why I want to implement queue and background task system on galaxy -api ?

• Currently galaxy api does all the processing specially for rawdata based on the user's request and returns the download url back to user , even though it is fast for for the large exports like country exports it will take time , based on the feature request it may take 10 min , 15 min so ,It is CPU intensive process , with the current implementation galaxy api keeps the listening port open for a request until process is finished and gives the response back which will work for the small requests which takes couple of seconds or minute but it is not okay for the large requests , It is not good engineering practice because it will be expensive in terms of network cost , and also api supposed to be returned within minute no matter what
• Second thing is currently there is no heavy usage of galaxy api going on , even though there are since the prodcution has load balancer and distributes load according to cpu usage ram usage of container , To be able to better handle the background task we need to be organized and those distributing tasks should be done through queue , in that way even there is high usage of API API will perform in effective way meanwhile post request will return within second with the link from where respective user can track his export / request . In order to implement this My current concern is :

1) Request should be trackable 2) Background tasks should be on queue basis and user should know either he/she is in queue or not 3 ) if process fails then API should deliver clear message why it got failed and go back to queue and pick another request 4) for each process we launch we should know how much cpu / ram usage that process is consuming and also we should know the worker's state 5) Containers should swan up based on the queue that we will use so that it should be easy on production too

[NicolasGrosjean]

2. Background tasks should be on queue basis and user should know either he/she is in queue or not

If possible, we can try to give the position in the queue if the user's request is not currently running.

I had never heard of Dramatiq, but I see from its website it is simpler than Celery. For others I only know the name, so the developers of other HOT tools using them could give you advice on the best to choose.

[kshitijrajsharma]

I have done some comparison and research around which library to use : Among tens of task queue library available on the internet , I found celery and dramatiq as a strong contestant on the race ! In export tool we are using dramatiq and redis as broker ! For the task queue library I have taken celery and dramatiq in to account and for the broker I have taken RabbitMQ and redis in to account . From my analysis on task queue library I have opted for celery because It is old , matches our requirement , well maintained and has large group of community around it which is active , Latest version of dramatiq was 2 years ago meanwhile celery is being maintained frequently , Also celery will be easy to maintain / find tutorials , also comes with huge documentation ! Dramatiq is very easy to implement and assign task to workers with a single decorater meanwhile celery will require some more efforts to set it up and configure but I am okay with it . Using celery we can have lot of monitoring framework available on the internet , Flower is very popular among them , With dramatiq again there will be very few options with few customization available . For projects which need instant implementation with out thinking about any hard implementation dramatiq will be very good option in no doubt , for us celery will be good option in long run !

Second confusion I had around redis and Rabbitmq , Which is again for our requirement redis will do the job as it is a database that can be used as broker , There will be no complex routing for message queue and redis is also easily managable performs well with in memory message queue , it is lightweight , fast and well suited for our job as we intend to deliver realtime data to user ! This is why I think combination of redis and celery will be good fit for us