[BUG] FunctionalAnnotation: InterProScan not working on Rackham with original configuration

verku commented 3 years ago

I tried to run the pipeline the same way as I ran it successfully on NAC, but adding the path to the local InterProScan module (/sw/apps/bioinfo/InterProScan/5.30-69.0/rackham) before starting the pipeline with the "uppmax" Nextflow profile, I got the following error:

Command: bin/hmmer/hmmer3/3.1b1/hmmsearch -Z 4488 --cut_tc --cpu 1 -o
/scratch/19129091/r57.uppmax.uu.se_20210312_150617097_ib08//jobTIGRFAM/000000000001_000000000206.raw.out
data/tigrfam/15.0/TIGRFAMs_HMM.LIB
/scratch/19129091/r57.uppmax.uu.se_20210312_150617097_ib08//jobTIGRFAM/000000000001_000000000206.fasta

  Error output from binary:

  Error: File existence/permissions problem in trying to open HMM file data/tigrfam/15.0/TIGRFAMs_HMM.LIB.
  HMM file data/tigrfam/15.0/TIGRFAMs_HMM.LIB not found (nor an .h3m binary of it)

Next, I installed InterProScan locally (testing different versions), and exported the respective location to the PATH instead of the path to the module, but I kept getting similar errors, e.g.

Error: File existence/permissions problem in trying to open HMM file data/sfld/3/sfld.hmm.  
HMM file data/sfld/3/sfld.hmm not found (nor an .h3m binary of it)

Some observations from testing different versions:

Newer Interproscan versions (e.g. 5.46-81.0, 5.51-85.0) have newer database versions installed but when run from within Nextflow, they kept trying to access the older database versions.
Version 5.30-69.0 requires Java 1.8 instead of 11 which is installed in the Nextflow conda environment along with Nextflow.

Finally, I got the impression that Nextflow was trying to run the Interproscan Docker container (version 5.30-69.0) that is provided in config/software_packages.config, although I added other installations to the PATH. I eventually managed to run the pipeline on Rackham by commenting out line 10 in the file config/software_packages.config where the container was specified and by commenting out line 183 in FunctionalAnnotation.nf that was adding the container installation to the PATH, and by starting the pipeline this way:

1) activate the conda environment with Nextflow 2) load the Uppmax module with module load bioinfo-tools InterProScan/5.30-69.0 3) start the pipeline with the "uppmax" profile

This is fine for the purpose of my project, but maybe you have an idea how to get it to run on Rackham with the original configuration.

mahesh-panchal commented 3 years ago

What commands did you use to run it on NAC and Uppmax? Which profiles?

Sorry, there's a few things to unpack here. I'm also not sure of the changes that have been made to Interproscan while I've been away. Jacques is working on conda package for Interproscan (see https://github.com/bioconda/bioconda-recipes/issues/21601), so we never use the docker/singularity profile for the FunctionalAnnotation.nf as the old container doesn't work properly.

Sorry, let's start from how you ran it on Uppmax and NAC, and which files you changed.

mahesh-panchal commented 3 years ago

I think I'm slowly starting to understand.

Using the uppmax profile enables singularity, but the Interproscan container doesn't work properly.

As long as the PATH is correctly set, then line 183 (https://github.com/NBISweden/pipelines-nextflow/blob/0bfecf0e2f4e91a83e7be75db25c8ccbb648e91b/FunctionalAnnotation/FunctionalAnnotation.nf#L183) shouldn't need to be commented out.

At the moment, there is then a need for a custom profile for uppmax to use this particular profile You can add the following to your own config file and pass it to nextflow:

nextflow run -c custom.config -profile custom_uppmax <script>

where custom.config is:

profiles {
    custom_uppmax {
        executor {
            name = 'slurm'
        }
        process {
            scratch = '$SNIC_TMP'
        }
        includeConfig "$baseDir/config/compute_resources.config"
        includeConfig "$baseDir/config/software_packages.config"
        env.PATH='${PATH}:/sw/apps/bioinfo/InterProScan/5.30-69.0/rackham'
    }
}

This uses the conda packages then by default.

I'll make a patch to the config until we can get around to having a proper container available. Thanks!

verku commented 3 years ago

This sounds great, thank you! I will test it later today with my dataset.

On Wed, May 19, 2021 at 9:57 AM Mahesh Binzer-Panchal < @.***> wrote:

I think I'm slowly starting to understand.

Using the uppmax profile enables singularity, but the Interproscan container doesn't work properly.

As long as the PATH is correctly set, then line 183 ( https://github.com/NBISweden/pipelines-nextflow/blob/0bfecf0e2f4e91a83e7be75db25c8ccbb648e91b/FunctionalAnnotation/FunctionalAnnotation.nf#L183) shouldn't need to be commented out.

At the moment, there is then a need for a custom profile for uppmax to use this particular profile You can add the following to your own config file and pass it to nextflow:

nextflow run -c custom.config -profile custom_uppmax Githubissues.
Githubissues is a development platform for aggregating issues.

NBISweden / pipelines-nextflow

[BUG] FunctionalAnnotation: InterProScan not working on Rackham with original configuration #55