Scaling up and automating the resource allocation for global Galaxy training events
Supervisor: Beatriz Serrano-Solano, Gianmauro Cuccuru
For degree: Bachelor
Status: Open
Keywords: Galaxy, training, Python, JavaScript, GitHub workflows
Global context
Galaxy is an open-source, highly modular, flexible, and extensible system that focuses on easily accessible and reproducible research. To use Galaxy, the users just need a web browser, avoiding the drawbacks of downloading and installing software.
The European Galaxy Server provides Training Infrastructure as a Service (TIaaS) for the Galaxy training community. This service provides a special queue in the server so that, during the execution of a workshop, the trainees’ jobs are run preferentially without waiting in the general job queue of the server. The main advantage of this system is that the workshop can run smoothly, avoiding possible waiting execution times depending on the load of the server.
Anyone providing training on using Galaxy is eligible to request the use of this service. The service is free of charge, and there is no need for technical expertise from the trainers’ side to maintain or administrate a Galaxy instance. The trainers have also the possibility of checking the stage in which the trainees are through the dashboard. This feature is GDPR-compliant and particularly valuable for the trainer to get feedback on the understanding of the different steps of the scientific pipeline that is being explained.
Since 2018, close to 10,000 researchers have been trained in more than 200 workshops using TIaaS, spanning all over the globe. A statistical overview of the TIaaS training can be found at https://usegalaxy.eu/tiaas/stats.
Project context
The increasing number of requests is becoming especially noticeable due to the COVID-19 pandemic, in which many of the teaching activities at the university had to move online. This implies not only an increase in the computing infrastructure needs but also an overhead in processing the requests since it still requires human validation.
To use the special TIaaS queue, trainers just need to fill out a form indicating the number of attendees, the start and end dates of the training activity, and what kind of analysis will be run during the workshop. This information is relevant to estimate the computational resources that will ensure a successful performance along with the duration of the training activity. Once the requests are manually approved, the trainers receive an email with a special URL that will give preferential access to the group of trainees and a link to the individual training dashboard.
Objectives of the project
Our aim is to scale this service that we have been developing since 2018 so that we can provide free education to as many researchers as possible. To accomplish this goal, we would need to automatize the processing of requests. The availability of resources and the information provided by the trainer (e.g. workflow, set of tools to be run, number of attendees, duration...) will help to create an automatic pipeline that assigns resources for each training.
Proposed agenda for the project
Estimate the number of virtual machines (VMs) needed for TIaaS requests.
Query the database to retrieve the request submitted by the trainer.
Based on the information provided by the trainer and the training and the availability of resources, estimate the number (and flavour) of VMs needed for the training.
Scaling up and automating the resource allocation for global Galaxy training events
Supervisor: Beatriz Serrano-Solano, Gianmauro Cuccuru For degree: Bachelor Status: Open Keywords: Galaxy, training, Python, JavaScript, GitHub workflows
Global context
Galaxy is an open-source, highly modular, flexible, and extensible system that focuses on easily accessible and reproducible research. To use Galaxy, the users just need a web browser, avoiding the drawbacks of downloading and installing software.
The European Galaxy Server provides Training Infrastructure as a Service (TIaaS) for the Galaxy training community. This service provides a special queue in the server so that, during the execution of a workshop, the trainees’ jobs are run preferentially without waiting in the general job queue of the server. The main advantage of this system is that the workshop can run smoothly, avoiding possible waiting execution times depending on the load of the server.
Anyone providing training on using Galaxy is eligible to request the use of this service. The service is free of charge, and there is no need for technical expertise from the trainers’ side to maintain or administrate a Galaxy instance. The trainers have also the possibility of checking the stage in which the trainees are through the dashboard. This feature is GDPR-compliant and particularly valuable for the trainer to get feedback on the understanding of the different steps of the scientific pipeline that is being explained.
Since 2018, close to 10,000 researchers have been trained in more than 200 workshops using TIaaS, spanning all over the globe. A statistical overview of the TIaaS training can be found at https://usegalaxy.eu/tiaas/stats.
Project context
The increasing number of requests is becoming especially noticeable due to the COVID-19 pandemic, in which many of the teaching activities at the university had to move online. This implies not only an increase in the computing infrastructure needs but also an overhead in processing the requests since it still requires human validation.
To use the special TIaaS queue, trainers just need to fill out a form indicating the number of attendees, the start and end dates of the training activity, and what kind of analysis will be run during the workshop. This information is relevant to estimate the computational resources that will ensure a successful performance along with the duration of the training activity. Once the requests are manually approved, the trainers receive an email with a special URL that will give preferential access to the group of trainees and a link to the individual training dashboard.
Objectives of the project
Our aim is to scale this service that we have been developing since 2018 so that we can provide free education to as many researchers as possible. To accomplish this goal, we would need to automatize the processing of requests. The availability of resources and the information provided by the trainer (e.g. workflow, set of tools to be run, number of attendees, duration...) will help to create an automatic pipeline that assigns resources for each training.
Proposed agenda for the project
Prerequisites
Further reading and useful links