search

CCBR / RENEE

A comprehensive quality-control and quantification RNA-seq pipeline
https://CCBR.github.io/RENEE/
MIT License
4 stars 4 forks source link

error in multiqc -- KeyError: 'No metadata except PKG-INFO is available' #51

Closed kelly-sovacool closed 10 months ago

kelly-sovacool commented 1 year ago

Error reported by Krithika & Ying.

snakemake.log

Error executing rule rnaseq_multiqc on cluster (jobid: 324, external: 12899399, jobscript: /vf/users/CCRCCDI/analysis/ccrtegs4/renee_20231120/.snakemake/tmp.39gqdzne/snakejob.rnaseq_multiqc.324.sh). For error details see the cluster log and the log files of the involved rule(s).
Exiting because a job execution failed.

logfiles/slurmfiles/12886741.12899073.rnaseq_multiqc..err:

/// MultiQC 🔍 | v1.12

|           multiqc | MultiQC Version v1.17 now available!
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3023, in _dep_map
      return self.__dep_map
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2817, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3014, in _parsed_pkg_info
    return self._pkg_info
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2817, in __getattr__
    raise AttributeError(attr)
AttributeError: _pkg_info

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/multiqc", line 5, in <module>
    from multiqc.__main__ import multiqc
  File "/usr/local/lib/python3.8/dist-packages/multiqc/__main__.py", line 53, in <module>
    multiqc.run_cli(prog_name="multiqc")
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/multiqc/multiqc.py", line 249, in run_cli
    multiqc_run = run(
  File "/usr/local/lib/python3.8/dist-packages/multiqc/multiqc.py", line 612, in run
    template_mod = config.avail_templates[config.template].load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2444, in load
    self.require(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2467, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 795, in resolve
    new_requirements = dist.requires(req.extras)[::-1]
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2738, in requires
    dm = self._dep_map
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3025, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3034, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3016, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1896, in get_metadata
    raise KeyError("No metadata except PKG-INFO is available")
KeyError: 'No metadata except PKG-INFO is available'
[Tue Nov 21 02:01:48 2023]
kelly-sovacool commented 1 year ago

I'm seeing if the error can be reproduced on the test dataset. Update: I didn't get any error with the test data

renee run --input .tests/*.R?.fastq.gz --output /data/sovacoolkl/renee_test2 --genome hg38_30 --mode slurm

I suspect this could be related to the fact that biowulf upgraded the singularity module to v4.0.1, which may have introduced breaking changes.

kelly-sovacool commented 11 months ago

Update: Krithika got this same error with the test dataset. It seems to be a software environment issue.

slsevilla commented 11 months ago

Second PI running on Biowulf is seeing this error.

Plan to test myself and see if I can replicate since @kelly-sovacool could not.

kelly-sovacool commented 11 months ago

Krithika is testing it now with --singularity-cache /data/CCBR_Pipeliner/SIFS, we'll see if that works.

kelly-sovacool commented 11 months ago

Krithika is testing it now with --singularity-cache /data/CCBR_Pipeliner/SIFS, we'll see if that works.

Failed:

renee: error:
            [6;37;41mFatal: Failed to provided a valid singularity cache![0m
                The provided --singularity-cache already exists on the filesystem with a different owner.
                Singularity strictly enforces that the cache directory is not shared across users.
                Please run /gpfs/gsfs10/users/CCBR_Pipeliner/Pipelines/RENEE/.v2.5.8/renee again with a different --singularity-cache location.

I'm confused about this because it works for me and I'm able to use this shared cache dir location in my nextflow pipelines. Do we need to change the permissions for /data/CCBR_Pipeliner/SIFS to make sure users not in the CCBR_Pipeliner group can use it?

kelly-sovacool commented 11 months ago

alt. potential solution: use different docker container that has been working in nextflow pipelines https://github.com/CCBR/RENEE/commit/6987849a0cfc5742d2d04148b9def4280f42b0e0

Edit: this still failed for Krithika

kelly-sovacool commented 11 months ago

Solutions that we (Krithika) tried:

kopardev commented 11 months ago

Look into the differences between SINGULARITY_LOCALCACHEDIR SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR

kelly-sovacool commented 11 months ago

multiqc worked for Krithika outside of the snakemake workflow.

script:

#!/usr/bin/env bash
#SBATCH --cpus-per-task=2
#SBATCH --mem=20g
#SBATCH --time=00:10:00
#SBATCH --parsable
#SBATCH -J "test-renee"
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --output "test_singularity.log"
#SBATCH --error "test_singularity.log"

workdir=/data/CCRCCDI/analysis/ccrtegs4/renee_test/output_test_sifcache/
#workdir=/data/sovacoolkl/renee_test/
cd $workdir
module load singularity
export SINGULARITY_CACHEDIR=${workdir}/.singularity

cmd="
which python;
echo $PATH;
multiqc --ignore '*/.singularity/*' -f -c ${workdir}/resources/multiqc_config.yaml --interactive --outdir ${workdir}/Reports ${workdir};
echo '=========='
"

echo "echo TEST1; $cmd" |\
    singularity exec -B $workdir docker://nciccbr/ccbr_multiqc_1.15:v1 bash
echo "echo TEST2; $cmd" |\
    singularity exec -e -B $workdir docker://nciccbr/ccbr_multiqc_1.15:v1 bash

output:

[+] Loading singularity  4.0.1  on cn4332 
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:e88f0d2e3e7849afafd4041fd86764a5f53b536aadb461dd3205b95459580653
Copying blob sha256:56e0351b98767487b3c411034be95479ed1710bb6be860db6df0be3a98653027
Copying blob sha256:a51180f2933c21cd973c72aa7fa8188d28c349f7c64670306792b7ca7e742d39
Copying blob sha256:733cf2d680d6f8ed2337737c3697405d0996278069d8b60a1672d5f21278bdcc
Copying blob sha256:25b3077843af896cd92eeadc7d1926f97d3578b224afae4636862519cc3c025b
Copying blob sha256:5f3ff2784635ccf884c461fe5a7ff4fab25389a418726d8bcf63b3f97277f9e6
Copying blob sha256:5421ffbaa795a5726aa4cbbc087320c7a6588bee26a8ca9e1f1636244827f1e8
Copying blob sha256:5de5bc716a2dd322f2e23e25d234d14676e7813dc4e82fc6620f6eb190818600
Copying blob sha256:5c454f3b0b73f2cb9e33421f95afeb397b057b3eb4679bbf94a9968e75b7815b
Copying blob sha256:6e258499a1cb942df31da05b81e1daaef80441d6d4956938b7cc5a16da2b7894
Copying blob sha256:6e258499a1cb942df31da05b81e1daaef80441d6d4956938b7cc5a16da2b7894
Copying blob sha256:835c15da49d683bb52e781fea7230c6d5e8fc9eb47cd8b3e9a96ead937b6fc7f
Copying blob sha256:835c15da49d683bb52e781fea7230c6d5e8fc9eb47cd8b3e9a96ead937b6fc7f
Copying blob sha256:8f3452a0019dc5658d852299ffa6b96f8403ad6d8c058a5e9add57bd004b5781
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:62e9366d14e7e5a7b04e68897fab7ff7f67694bd9fd358ac11c4fb50b91cf8d2
Copying blob sha256:dd8f4fddbd3eee320c75b2d4606339aaed7ae3e0cf1f014a3957941a8a834afe
Copying blob sha256:dd8f4fddbd3eee320c75b2d4606339aaed7ae3e0cf1f014a3957941a8a834afe
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:1d0817e0b5db957a9526eceb13733b65485ec36465621103e99ac2faa66cf6ed
Copying config sha256:b7087ffa6887c3a240cac3a33271335f972db5f9a97c1bc11f8f6b88bff9ba49
Writing manifest to image destination
2023/12/15 14:50:11  info unpack layer: sha256:56e0351b98767487b3c411034be95479ed1710bb6be860db6df0be3a98653027
2023/12/15 14:50:11  info unpack layer: sha256:a51180f2933c21cd973c72aa7fa8188d28c349f7c64670306792b7ca7e742d39
2023/12/15 14:50:11  info unpack layer: sha256:5f3ff2784635ccf884c461fe5a7ff4fab25389a418726d8bcf63b3f97277f9e6
2023/12/15 14:50:11  info unpack layer: sha256:733cf2d680d6f8ed2337737c3697405d0996278069d8b60a1672d5f21278bdcc
2023/12/15 14:50:12  info unpack layer: sha256:25b3077843af896cd92eeadc7d1926f97d3578b224afae4636862519cc3c025b
2023/12/15 14:50:14  info unpack layer: sha256:e88f0d2e3e7849afafd4041fd86764a5f53b536aadb461dd3205b95459580653
2023/12/15 14:50:23  info unpack layer: sha256:5421ffbaa795a5726aa4cbbc087320c7a6588bee26a8ca9e1f1636244827f1e8
2023/12/15 14:50:23  info unpack layer: sha256:5de5bc716a2dd322f2e23e25d234d14676e7813dc4e82fc6620f6eb190818600
2023/12/15 14:50:26  info unpack layer: sha256:5c454f3b0b73f2cb9e33421f95afeb397b057b3eb4679bbf94a9968e75b7815b
2023/12/15 14:50:26  info unpack layer: sha256:6e258499a1cb942df31da05b81e1daaef80441d6d4956938b7cc5a16da2b7894
2023/12/15 14:50:26  info unpack layer: sha256:6e258499a1cb942df31da05b81e1daaef80441d6d4956938b7cc5a16da2b7894
2023/12/15 14:50:26  info unpack layer: sha256:835c15da49d683bb52e781fea7230c6d5e8fc9eb47cd8b3e9a96ead937b6fc7f
2023/12/15 14:50:26  info unpack layer: sha256:835c15da49d683bb52e781fea7230c6d5e8fc9eb47cd8b3e9a96ead937b6fc7f
2023/12/15 14:50:26  info unpack layer: sha256:8f3452a0019dc5658d852299ffa6b96f8403ad6d8c058a5e9add57bd004b5781
2023/12/15 14:50:26  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2023/12/15 14:50:26  info unpack layer: sha256:62e9366d14e7e5a7b04e68897fab7ff7f67694bd9fd358ac11c4fb50b91cf8d2
2023/12/15 14:50:28  info unpack layer: sha256:dd8f4fddbd3eee320c75b2d4606339aaed7ae3e0cf1f014a3957941a8a834afe
2023/12/15 14:50:28  info unpack layer: sha256:dd8f4fddbd3eee320c75b2d4606339aaed7ae3e0cf1f014a3957941a8a834afe
2023/12/15 14:50:28  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2023/12/15 14:50:28  info unpack layer: sha256:1d0817e0b5db957a9526eceb13733b65485ec36465621103e99ac2faa66cf6ed
INFO:    Creating SIF file...
TEST1
Illegal option --
Usage: /usr/bin/which [-a] args
/usr/local/current/singularity/4.0.1/bin:/home/bhuvaneshwark2/.local/bin:/home/bhuvaneshwark2/bin:/usr/local/slurm/bin:/usr/local/bin:/usr/X11R6/bin:/usr/local/jdk/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/mysql/bin

  /// MultiQC 🔍 | v1.15

|           multiqc | MultiQC Version v1.17 now available!
|           multiqc | Search path : /data/CCRCCDI/analysis/ccrtegs4/renee_test/output_test_sifcache
|         searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 883/883  
|             bbmap | Found 4 reports
|      fastq_screen | Found 16 reports
|            kraken | Found 4 reports
|            fastqc | Found 16 reports
|          cutadapt | Found 4 reports
|              star | Found 4 reports and 4 gene count files
|            preseq | Found 4 reports
|              rsem | Found 4 reports
|          qualimap | Found 4 BamQC reports
|             rseqc | Found 4 read_distribution reports
|             rseqc | Found 4 inner_distance reports
|             rseqc | Found 4 infer_experiment reports
|             rseqc | Found 4 tin reports
|            picard | Found 4 MarkDuplicates reports
|            picard | Found 4 RnaSeqMetrics reports
|          samtools | Found 4 flagstat reports
|           multiqc | Report      : Reports/multiqc_report.html
|           multiqc | Data        : Reports/multiqc_data
|           multiqc | MultiQC complete
==========
INFO:    Using cached SIF image
TEST2
/usr/bin/python
/usr/local/current/singularity/4.0.1/bin:/home/bhuvaneshwark2/.local/bin:/home/bhuvaneshwark2/bin:/usr/local/slurm/bin:/usr/local/bin:/usr/X11R6/bin:/usr/local/jdk/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/mysql/bin

  /// MultiQC 🔍 | v1.15

|           multiqc | MultiQC Version v1.17 now available!
|           multiqc | Search path : /data/CCRCCDI/analysis/ccrtegs4/renee_test/output_test_sifcache
|         searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 884/884  
|             bbmap | Found 4 reports
|      fastq_screen | Found 16 reports
|            kraken | Found 4 reports
|            fastqc | Found 16 reports
|          cutadapt | Found 4 reports
|              star | Found 4 reports and 4 gene count files
|            preseq | Found 4 reports
|              rsem | Found 4 reports
|          qualimap | Found 4 BamQC reports
|             rseqc | Found 4 read_distribution reports
|             rseqc | Found 4 inner_distance reports
|             rseqc | Found 4 infer_experiment reports
|             rseqc | Found 4 tin reports
|            picard | Found 4 MarkDuplicates reports
|            picard | Found 4 RnaSeqMetrics reports
|          samtools | Found 4 flagstat reports
|           multiqc | Report      : Reports/multiqc_report.html   (overwritten)
|           multiqc | Data        : Reports/multiqc_data   (overwritten)
|           multiqc | MultiQC complete
==========
kopardev commented 11 months ago

Try:

kelly-sovacool commented 11 months ago

will update on FRCE /mnt/projects/CCBR-Pipelines/pipelines/RENEE and see if we get the error

kelly-sovacool commented 10 months ago

Let's release v2.5.11 and ask Krithika to re-run (after #95)

kopardev commented 10 months ago

Seems like this is working now ... weird!!

image
kelly-sovacool commented 10 months ago

How bizarre! I'm glad it's working now but wish we knew why.

kopardev commented 10 months ago

@kelly-sovacool ... just double check if @Krithika-Bhuvan still has this error!!

slsevilla commented 10 months ago

I just completed a run on Biowulf with main branch and it worked for me. I think we should leave this open until we have Krithika test it

kelly-sovacool commented 10 months ago

Krithika is re-running it now

kelly-sovacool commented 10 months ago

Krithika's test succeeded!