Closed ljwh2 closed 2 months ago
HDF5 files are missing in the private FTP. Asked Kaur if we should run the pipeline on EBI compute cluster.
Asked Kaur for clarification on the Nextflow pipeline.
Gathered *.cc.tsv.gz
files from sumstats and unzipped them.
Submitted tsv2hdf
pipeline as a SLURM job
JOBID PARTITION NAME USER ST TIME NODES NODELIST
7943535 datamover tsv2hdf-eqtl-sumstats-70 gwas_lsf R 2:49 1 codon-dm-07
Update: Failed due to time and memory constraints as they are not specified. This is due to SLURM migration.
Submitted a SLURM job that created HDF5 files. Expect the files in 2 days in this dir: /hps/nobackup/parkinso/spot/gwas/scratch/eqtl-sumstats-70/hdf5
JOBID PARTITION NAME USER ST TIME NODES NODELIST
8342181 datamover tsv2hdf-eqtl-sumstats-70 gwas_lsf R 19:02 1 codon-dm-05
Update. Failed as Wallclock exceeded.
I have resubmitted the SLURM job with expanded time and memory limits (7 days, 32GB). Over two days, it generated 19 files using 4GB each. Estimating roughly two weeks to produce 195 files with the same constraints. I will request an extension next Monday if necessary.
JOBID PARTITION NAME USER ST TIME NODES NODELIST
12602601 datamover tsv2hdf-eqtl-sumstats-70-take_2 gwas_lsf R 0:07 1 codon-dm-07
Update - 14/05. Still running, 17 files are generated so far.
[gwas_lsf@codon-dm-08 goci1306]$ ls -ltr /hps/nobackup/parkinso/spot/gwas/scratch/eqtl-sumstats-70/hdf5/data/
total 11237028
-rw-r--r-- 1 gwas_lsf spot 641870177 May 13 11:48 QTD000564.cc.h5
-rw-r--r-- 1 gwas_lsf spot 1171725085 May 13 14:46 QTD000565.cc.h5
-rw-r--r-- 1 gwas_lsf spot 384931120 May 13 15:05 QTD000566.cc.h5
-rw-r--r-- 1 gwas_lsf spot 651121794 May 13 15:55 QTD000567.cc.h5
-rw-r--r-- 1 gwas_lsf spot 404137316 May 13 16:15 QTD000568.cc.h5
-rw-r--r-- 1 gwas_lsf spot 249738809 May 13 16:24 QTD000569.cc.h5
-rw-r--r-- 1 gwas_lsf spot 635153165 May 13 17:22 QTD000570.cc.h5
-rw-r--r-- 1 gwas_lsf spot 165125845 May 13 17:27 QTD000571.cc.h5
-rw-r--r-- 1 gwas_lsf spot 321946272 May 13 17:42 QTD000572.cc.h5
-rw-r--r-- 1 gwas_lsf spot 237479017 May 13 17:50 QTD000573.cc.h5
-rw-r--r-- 1 gwas_lsf spot 929367552 May 13 19:18 QTD000574.cc.h5
-rw-r--r-- 1 gwas_lsf spot 1229204319 May 13 22:17 QTD000575.cc.h5
-rw-r--r-- 1 gwas_lsf spot 361636592 May 13 22:37 QTD000576.cc.h5
-rw-r--r-- 1 gwas_lsf spot 541988245 May 13 23:17 QTD000577.cc.h5
-rw-r--r-- 1 gwas_lsf spot 284813726 May 13 23:27 QTD000578.cc.h5
-rw-r--r-- 1 gwas_lsf spot 2345812458 May 14 09:31 QTD000579.cc.h5
-rw-r--r-- 1 gwas_lsf spot 950666626 May 14 11:41 QTD000580.cc.h5
132 files done. Asked Codon Team to add another 7 days to the job.
All files done.
Submitted SLURM job 15938725 that generates metadata using https://github.com/eQTL-Catalogue/eQTL-Catalogue-resources/blob/master/data_tables/dataset_metadata_upcoming.tsv
Metadata generation complete.
Copy data and metadata (qtl_metadata.h5
) to the private ftp /otar_sumstats
complete.
Copy data and metadata (qtl_metadata.h5
) to the pub ftp /eqtl
complete.
Submitted batch job 15954227 to copy the .tsv files in sumstats and susie to the pub ftp /nfs/ftp/public/databases/spot/eQTL/
Copy .tsv files complete.
Find the updated docs here: https://www.ebi.ac.uk/seqdb/confluence/display/GOCI/eQTL+Summary+Statistics
Kaur checked the files and announced the release on the eQTL Catalogue website as well: https://www.ebi.ac.uk/eqtl/Release_notes/
✅ Run the
tsv2hdf
Nextflow pipeline -- ✅ Generate HDF5 files -- ✅ Generate metadata file ✅ Copy HDF5 files ✅ Copy metadata file ✅ Copy the.tsv
files to ftp ✅ Test release files (HDF5 & tsv files) ✅ Gitlab release ✅ Update confluence doc https://www.ebi.ac.uk/seqdb/confluence/display/GOCI/eQTL+Summary+Statistics ✅ Review by Kaur