Testing infrastructure with Github Actions using custom-runners on nf-core infra

nextflow-io / nf-nomad

Hashicorp Nomad executor plugin for Nextflow

https://nextflow-io.github.io/nf-nomad/

Apache License 2.0

2 stars 4 forks source link

Testing infrastructure with Github Actions using custom-runners on nf-core infra #3

Open abhi18av opened 1 year ago

abhi18av commented 1 year ago

The goal of this task is to

Use GitHub actions
Setup a test nomad cluster
Use CSI-hostpath volumes

abhi18av commented 8 months ago

Hi @tomiles and @matthdsm , please feel free to add details/subtasks to this task as you see fit. I've marked this as part of v0.1.0

@jagedn and I would focus on the local dev cluster and the mock testing setup for the initial milestones.

abhi18av commented 8 months ago

Adding notes after meeting with @jhaezebr and @matthdsm

Test-Cluster Phase-1: Nomad cluster on Azure

[x] Host Volume / CSI volume setup using Azure FileShare
[x] One Azure VM (Bastian host)
[x] Minimal Nomad ACL setup
[ ] Share the tf codebase on Github.com
[ ] The cluster has 6-node setup ( 3-servers and 3-clients )
[ ] Tweaks for the TF setup regarding nodes
- Bastion Host => Standard_B8ms
- Server => Standard_B2ms
- Clients => Standard_B8ms / B16ms
[x] Bastion host tweaks
- Nextflow
- Java-17 Amazon Correto-LTS (ideally via SDKman)
- Docker
- Conda
[ ]

Test-Cluster Phase-2: Multiple nodes and Real World ACLs

Github runner for the Azure nomad dev-cluster
Iterate on ACL for real world setup https://developer.hashicorp.com/nomad/docs/other-specifications/acl-policy
(LATER) A Nomad TLS certificate
(LATER) Hybrid cluster (mixed client types GPU/CPU)

abhi18av commented 4 months ago

Sync meeting : @jhaezebr , @tomiles , @jagedn , @abhi18av , @matthdsm

Milestone-2 (initiating now)

[x] Jules: Github runner for the Azure nomad dev-cluster
[ ] Jules: Iterate on the minimal ACL policy starting from https://github.com/nextflow-io/nf-nomad/issues/56#issuecomment-2209267588 on az-nomadlab
[ ] Jules: A Nomad TLS certificate
[ ] Tweak the Terraform template for reducing the head nodes and compute nodes. Maintain the bastion host as a stable basis.

Milestone-3

(LATER) Hybrid cluster (mixed client types GPU/CPU)

jagedn commented 2 months ago

for the record

from now we have a GH action able to run pipelines using the Azure cluster, see https://github.com/nextflow-io/nf-nomad/actions/runs/10614308658

jhaezebr commented 2 months ago

for the record

from now we have a GH action able to run pipelines using the Azure cluster, see https://github.com/nextflow-io/nf-nomad/actions/runs/10614308658

I'm looking into setting up scripts so that the runner creates and destroys the nomad cluster each time, so you can test against a nomad cluster