Nextflow

[jdhayhurst]

executed on codon by:

module load nextflow-21.10.6-gcc-9.3.0-tkuemwd
nextflow main.nf --to_harm_folder <path> --inputPath <path>

Review (TODO):

• addition of three additional folders:
  • modules, subworkflows/local and workflows. - are these all necessary?
  • is local subdir required?
  • remove anything from repo that is not used
  • presumably modiles/nf-core is not needed?
• nextflow.config
  • remove all local/server paths
  • add process.container = 'docker://ebispot/gwas-sumstats-harmoniser:latest'
  • add singularity.enabled = true
• documentation/readme - update
• rename harmonnized_ch
• remove all pycache and .pyc files
• What is difference between --to_harm_folder and --inputPath?
• standardise to always use snake_case vs camelCase
• install wget into dockerfile
• NFCORE_GWASCATALOGHARM:GWASCATALOGHARM:prepare_reference:get_tbi_files exits with 0 but gives missing output file - file system latency? Means you have to run and then resume
• hardcoded file name in ./subworkflows/local/quality_control.nf for vcf - depends on someone running it with chr 22 in the config - remove hardcoding
• rename ./modules/local/haronization_log.nf
• rename ftp in the config to something more generic like "destination path"
• qc breaks if chroms from config are not in file
• *.running.log - input file is a hash (possible nextflow byproduct), no reference data and -e at start of some lines
• accession_id.sh fails in the container environment - any reason for this to be bash instead of python?
  • fixable by editing modules/local/ftp_copy.nf change sh to bash
• rename running.log --> harmonisation.log
• 25 harmonisation and 25 strand count jobs are made even if there are not 25 chroms in the config (seems to be dictated by the files in the references dir

[jdhayhurst]

Review (TODO):

• addition of three additional folders:
  • modules, subworkflows/local and workflows. - are these all necessary?
  • is local subdir required?
  • remove anything from repo that is not used
  • presumably modiles/nf-core is not needed?
• nextflow.config
  • [x] remove all local/server paths
  • [x] add process.container = 'docker://ebispot/gwas-sumstats-harmoniser:latest'
  • [x] add singularity.enabled = true
• [ ] documentation/readme - update
• [ ] rename harmonnized_ch
• [ ] remove all pycache and .pyc files
• What is difference between --to_harm_folder and --inputPath?
• [ ] standardise to always use snake_case vs camelCase
• [x] install wget into dockerfile
• [ ] NFCORE_GWASCATALOGHARM:GWASCATALOGHARM:prepare_reference:get_tbi_files exits with 0 but gives missing output file - file system latency? Means you have to run and then resume
• [ ] hardcoded file name in ./subworkflows/local/quality_control.nf for vcf - depends on someone running it with chr 22 in the config - remove hardcoding
• [ ] rename ./modules/local/haronization_log.nf
• [ ] rename ftp in the config to something more generic like "destination path"
• [ ] qc breaks if chroms from config are not in file
• [ ] *.running.log - input file is a hash (possible nextflow byproduct), no reference data and -e at start of some lines
• accession_id.sh fails in the container environment - any reason for this to be bash instead of python?
  • [x] fixable by editing modules/local/ftp_copy.nf change sh to bash
• [ ] rename running.log --> harmonisation.log
• [ ] 25 harmonisation and 25 strand count jobs are made even if there are not 25 chroms in the config (seems to be dictated by the files in the references dir

[jdhayhurst]

@jiyue1214 I've got the map to build working (I committed some changes to read the chromosomes from the NF params). It's breaking for me on this line https://github.com/EBISPOT/gwas-sumstats-harmoniser/blob/278d31ce44242ff655ef98a6614fc1e6f95095a7/modules/local/ten_percent_counts.nf#L18. shuf is not installed in the docker image. There's two approaches 1. install shuf into the docker image or 2. rewrite that either with bash utils that are installed or with python.

[jiyue1214]

@jiyue1214 I've got the map to build working (I committed some changes to read the chromosomes from the NF params). It's breaking for me on this line

https://github.com/EBISPOT/gwas-sumstats-harmoniser/blob/278d31ce44242ff655ef98a6614fc1e6f95095a7/modules/local/ten_percent_counts.nf#L18

. shuf is not installed in the docker image. There's two approaches 1. install shuf into the docker image or 2. rewrite that either with bash utils that are installed or with python.

Hi, @jdhayhurst . I checked the docker image of ebispot/gwas-sumstats-harmoniser:latest and I found shuf had already included in the current docker image. However, pysam package had not yet. It may be the reason you may see the pipeline broken at this line. I created a new docker image, which contains wget and pysam at here which may helps: 'docker://athenaji/gwas_harm_test:latest' .

[jdhayhurst]

@jiyue1214 I haven't been testing with "latest" because that is only built for releases. I've been using a local docker image built from the dockerfile in this branch of the repository, which has wget and pysam installed.