search

EBISPOT / gwas-sumstats-harmoniser

GWAS Summary Statistics Data Harmonisation
19 stars 13 forks source link

Setup the nextflow tower #87

Closed jiyue1214 closed 3 months ago

jiyue1214 commented 6 months ago

Karatug is the nextflow tower admin and Yue is a user. We need to test

  1. How could we monitor jobs running on the cluster
  2. Maybe customise a dashboard for it

A review meeting will be with Santiago and C.D. after 4 weeks.

jiyue1214 commented 5 months ago

Yue has tried:

  1. Run the nf pipeline and monitor the job via the nextflow tower (simply add --with-tower in the nextflow tower and set some credentials - documentation)
  2. Set the computing environment for LSF on nextflow tower and tower agent on the codon cluster, which can launch the job from the tower directly.
  3. Set the computing environment for Slurm on nextflow tower but failed (need to set memory and wall time), if needed, can ask C.D.

To discuss: Do we need to launch the job from the nextflow tower? If we do, we need to decide two points:

  1. Running the tower-agent on gwas_lsf forever (SSH link is not available for codon).
  2. Slurm computing environment setting (can get help from cloud group)
jiyue1214 commented 5 months ago

Monitoring the harmonisation job need to generate new release with new docker image (need ps to be installed to collect metrics). It is not urgent , I would suggest to put it in the sprint May,29 - June11.

jiyue1214 commented 4 months ago

A new feature to the harmonisation pipeline: Handle negative 10 p_value

jiyue1214 commented 3 months ago

It is Done. Nextflow pipeline is running on Slurm and can be monitored by the nextflow tower daily. [Uploading... Screenshot 2024-06-11 at 21.47.49.png]