search

nextflow-io / nf-nomad

Hashicorp Nomad executor plugin for Nextflow
https://nextflow-io.github.io/nf-nomad/
Apache License 2.0
2 stars 4 forks source link

Improve job-status query logic and nf-tower support #77

Open abhi18av opened 3 months ago

abhi18av commented 3 months ago

The job query status logic needs to be revamped in order to work well with Tower.

As of now, the jobs (on Tower) are always shown as running and are not progressing to completed status. Despite being actually completed on the cluster itself.

image

Also, as a result, the pipeline execution report (within nf-core pipelines) also doesn't contain any relevant information.

image
abhi18av commented 1 month ago

@jagedn , now that our NomadService class has been refactored via #89 , maybe we should pick this up again?

jagedn commented 3 weeks ago

One question:

I've tried to run a simple pipeline using local executor + tower

nextflow-io/rnaseq-nf --proces.executor=local

and I can see it is created in tower, but the status of the tasks are not updated and the spinner is always running. If I refresh the page them all looks fine

Is this a typical use case of Tower or I'm doing something wrong and Tower is able to show in runtime, without any refresh, the situation of the tasks ?

`

jagedn commented 3 weeks ago

So, I'm working in a revamp of the status logic and now I can see information in tower running ./run-pipeline.sh -c tower/nextflow.config nf-core/fetchngs -profile test -with-tower --outdir $(pwd)/nomad_temp/scratchdir/out

but Tower doesn't refresh the status of the tasks, I need to refresh the page. Not sure if this is an issue of Tower

Running this pipeline and once the pipeline completed I can see :

imagen

imagen

imagen

imagen

imagen

so from my side it seems is working except tower doesnt refresn in realtime the information (also I was debugging and I was able to see the plugin is sending the info to the remote server during the execution of the pipeline)

abhi18av commented 3 weeks ago

so from my side it seems is working except tower doesnt refresn in realtime the information (also I was debugging and I was able to see the plugin is sending the info to the remote server during the execution of the pipeline)

This is great Jorge - and welcome back 😉

The Tower page generally should refresh automatically, but the best experience is when the pipeline is launched through Tower itself. In the monitoring use-case, afaict, there is a bit of time lag before updates.

I think this can conclude the current iteration and we can make the release for v0.3.1.

Let's create a PR and I can also test locally for this.

jagedn commented 3 weeks ago

Still working on it (tests are failing) but you can grab this branch and test in your local

https://github.com/nextflow-io/nf-nomad/tree/tower-nf