Azure Pipelines is a good solution for basic testing of research software. Key benefits for researchers who have strong scientific programming skills but don't have technical systems knowlegde are that:
It's fully hosted and maintained by Microsoft, so no need for researchers to maintain the service and associated hardware
Configuration happens in-repository and is essentially a programming exercise
It's tightly integrated into the Azure Dev Ops service, centralising testing with team discussion and code hosting
However, larger and in particular operationally used research software projects may find Pipelines limits their ability to test, to the point that they prefer to set up a custom testing solution with the associated costs of human learning and maintenance. A critical issue is that Pipelines doesn't allow use of higher specification workers or workers with non-standard hardware, except in the case where users link their own existing hardware into pipelines. Doing this removes a key benefit of researchers not having to maintain the service and hardware themselves.
For reseachers needing to evaluate Pipelines, develop this theme into a short learning path passing on the key information and considerations as relevant to research software. Provide examples of evaluations and decisions going each way with Devito (pipelines) and Fluidity and/or Firedrake (Jenkins).
Issue addressing the 'no custom hosted workers' limitation is filed as issue #6
Azure Pipelines is a good solution for basic testing of research software. Key benefits for researchers who have strong scientific programming skills but don't have technical systems knowlegde are that:
However, larger and in particular operationally used research software projects may find Pipelines limits their ability to test, to the point that they prefer to set up a custom testing solution with the associated costs of human learning and maintenance. A critical issue is that Pipelines doesn't allow use of higher specification workers or workers with non-standard hardware, except in the case where users link their own existing hardware into pipelines. Doing this removes a key benefit of researchers not having to maintain the service and hardware themselves.
For reseachers needing to evaluate Pipelines, develop this theme into a short learning path passing on the key information and considerations as relevant to research software. Provide examples of evaluations and decisions going each way with Devito (pipelines) and Fluidity and/or Firedrake (Jenkins).
Issue addressing the 'no custom hosted workers' limitation is filed as issue #6