search

Bishop-Laboratory / RLPipes

RLPipes: A standardized R-loop-mapping pipeline.
https://anaconda.org/bioconda/rlpipes
MIT License
3 stars 1 forks source link

md5sum mismatch when building rlseq conda environment, and after fixing build, subsequent errors in the rlseq environment #93

Open CicadaDennis opened 5 months ago

CicadaDennis commented 5 months ago

Hello, In working with a researcher at Indiana University, we found that the conda installed RLPipes had errors when building the rlseq conda environment with snakemake. (See example error below) I discovered two places where changing one value in the environment file lib/python3.9/site-packages/rlpipes/src/envs/rlseq.yaml would fix specific failures: changed

but eventually modified the file by removing the =xxxxx values after the version numbers for a large number of the items in the file, to see if it would build the environment without those parts of the file.

Example error:

CreateCondaEnvironmentException:
Could not create conda environment from /N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/site-packages/rlpipes/src/envs/rlseq.yaml:
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... 

done
ERROR conda.core.link:_execute(730): An error occurred while installing package 'bioconda::bioconductor-org.hs.eg.db-3.14.0-r41hdfd78af_0'.
Rolling back transaction: ...working... done
class: LinkError
message:
post-link script failed for package bioconda::bioconductor-org.hs.eg.db-3.14.0-r41hdfd78af_0
location of failed script: /N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/4dde69eb05a1c2ea0d2d2e8f5c598808/bin/.bioconductor-org.hs.eg.db-post-link.sh
==> script messages <==
<None>
==> script output <==
stdout: /N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/4dde69eb05a1c2ea0d2d2e8f5c598808/share/bioconductor-org.hs.eg.db-3.14.0-0/org.Hs.eg.db_3.14.0.tar.gz: FAILED
/N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/4dde69eb05a1c2ea0d2d2e8f5c598808/share/bioconductor-org.hs.eg.db-3.14.0-0/org.Hs.eg.db_3.14.0.tar.gz: FAILED
/N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/4dde69eb05a1c2ea0d2d2e8f5c598808/share/bioconductor-org.hs.eg.db-3.14.0-0/org.Hs.eg.db_3.14.0.tar.gz: FAILED
ERROR: post-link.sh was unable to download any of the following URLs with the md5sum ef7fc0096ec579f564a33f0f4869324a:
https://bioconductor.org/packages/3.14/data/annotation/src/contrib/org.Hs.eg.db_3.14.0.tar.gz
https://bioarchive.galaxyproject.org/org.Hs.eg.db_3.14.0.tar.gz
https://depot.galaxyproject.org/software/bioconductor-org.hs.eg.db/bioconductor-org.hs.eg.db_3.14.0_src_all.tar.gz

stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   413  100   413    0     0   4073      0 --:--:-- --:--:-- --:--:--  4089
md5sum: WARNING: 1 computed checksum did NOT match
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   153  100   153    0     0    966      0 --:--:-- --:--:-- --:--:--   968
md5sum: WARNING: 1 computed checksum did NOT match
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   153  100   153    0     0    641      0 --:--:-- --:--:-- --:--:--   642
md5sum: WARNING: 1 computed checksum did NOT match

return code: 1

kwargs:
{}

Traceback (most recent call last):
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/exceptions.py", line 1082, in __call__
    return func(*args, **kwargs)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda_env/cli/main.py", line 80, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda_env/cli/main_create.py", line 142, in execute
    result[installer_type] = installer.install(prefix, pkg_specs, args, env)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/mamba/mamba_env.py", line 130, in mamba_install
    conda_transaction.execute()
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/core/link.py", line 281, in execute
    self._execute(tuple(concat(interleave(itervalues(self.prefix_action_groups)))))
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/core/link.py", line 744, in _execute
    raise CondaMultiError(tuple(concatv(
conda.CondaMultiErrorclass: LinkError
message:
post-link script failed for package bioconda::bioconductor-org.hs.eg.db-3.14.0-r41hdfd78af_0
location of failed script: /N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/4dde69eb05a1c2ea0d2d2e8f5c598808/bin/.bioconductor-org.hs.eg.db-post-link.sh
==> script messages <==
<None>
==> script output <==
stdout: /N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/4dde69eb05a1c2ea0d2d2e8f5c598808/share/bioconductor-org.hs.eg.db-3.14.0-0/org.Hs.eg.db_3.14.0.tar.gz: FAILED
/N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/4dde69eb05a1c2ea0d2d2e8f5c598808/share/bioconductor-org.hs.eg.db-3.14.0-0/org.Hs.eg.db_3.14.0.tar.gz: FAILED
/N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/4dde69eb05a1c2ea0d2d2e8f5c598808/share/bioconductor-org.hs.eg.db-3.14.0-0/org.Hs.eg.db_3.14.0.tar.gz: FAILED
ERROR: post-link.sh was unable to download any of the following URLs with the md5sum ef7fc0096ec579f564a33f0f4869324a:
https://bioconductor.org/packages/3.14/data/annotation/src/contrib/org.Hs.eg.db_3.14.0.tar.gz
https://bioarchive.galaxyproject.org/org.Hs.eg.db_3.14.0.tar.gz
https://depot.galaxyproject.org/software/bioconductor-org.hs.eg.db/bioconductor-org.hs.eg.db_3.14.0_src_all.tar.gz

stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   413  100   413    0     0   4073      0 --:--:-- --:--:-- --:--:--  4089
md5sum: WARNING: 1 computed checksum did NOT match
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   153  100   153    0     0    966      0 --:--:-- --:--:-- --:--:--   968
md5sum: WARNING: 1 computed checksum did NOT match
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   153  100   153    0     0    641      0 --:--:-- --:--:-- --:--:--   642
md5sum: WARNING: 1 computed checksum did NOT match

return code: 1

kwargs:
{}

: <exception str() failed>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/condabin/mamba", line 11, in <module>
    sys.exit(main())
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/mamba/mamba.py", line 916, in main
    return mamba_env.main()
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/mamba/mamba_env.py", line 140, in main
    return conda_env_main()
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda_env/cli/main.py", line 91, in main
    return conda_exception_handler(do_call, args, parser)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/exceptions.py", line 1374, in conda_exception_handler
    return_value = exception_handler(func, *args, **kwargs)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/exceptions.py", line 1085, in __call__
    return self.handle_exception(exc_val, exc_tb)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/exceptions.py", line 1116, in handle_exception
    return self.handle_application_exception(exc_val, exc_tb)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/exceptions.py", line 1132, in handle_application_exception
    self._print_conda_exception(exc_val, exc_tb)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/exceptions.py", line 1136, in _print_conda_exception
    print_conda_exception(exc_val, exc_tb)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/exceptions.py", line 1059, in print_conda_exception
    stderrlog.error("\n%r\n", exc_val)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/logging/__init__.py", line 1475, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/logging/__init__.py", line 1598, in handle
    if (not self.disabled) and self.filter(record):
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/logging/__init__.py", line 806, in filter
    result = f.filter(record)
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/gateways/logging.py", line 61, in filter
    record.msg = record.msg % new_args
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/__init__.py", line 132, in __repr__
    errs.append(e.__repr__())
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/__init__.py", line 71, in __repr__
    return '%s: %s' % (self.__class__.__name__, text_type(self))
  File "/N/soft/rhel8/miniconda/python3.9.12/4.12.0/lib/python3.9/site-packages/conda/__init__.py", line 90, in __str__
    return text_type(self.message % self._kwargs)
ValueError: unsupported format character 'T' (0x54) at index 1210

  File "/N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 392, in create

However, after successfully building the conda environments, the run of RLPipes still had an error in the rlseq section of the workflow:

Error in rule rlseq:
    jobid: 17
    output: /N/scratch/scttest/Quartz/rlpipes_out/rlseq_report/JL01_S1_R1_001_hg38.html, /N/scratch/scttest/Quartz/rlpipes_out/rlseq_report/JL01_S1_R1_001_hg38.rda
    log: /N/scratch/scttest/Quartz/rlpipes_out/logs/rlseq_report/JL01_S1_R1_001_hg38__rlseq.log (check log file(s) for error message)
    conda-env: /N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/693ce31b9d1c10140f03171630c415e0

RuleException:
CalledProcessError in line 378 of /N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/site-packages/rlpipes/src/rlpipes.smk:
Command 'source /N/soft/rhel8/miniconda/python3.9.12/4.12.0/bin/activate '/N/scratch/scttest/Quartz/rlpipes_out/.snakemake/conda/693ce31b9d1c10140f03171630c415e0'; set -euo pipefail;  Rscript --vanilla /N/scratch/scttest/Quartz/rlpipes_out/.snakemake/scripts/tmpkhny0an_.RLSeq.R' returned non-zero exit status 1.
  File "/N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2349, in run_wrapper
  File "/N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/site-packages/rlpipes/src/rlpipes.smk", line 378, in __rule_rlseq
  File "/N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 569, in _callback
  File "/N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/concurrent/futures/thread.py", line 58, in run
  File "/N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 555, in cached_or_run
  File "/N/scratch/scttest/Quartz/rlpipes_cenv/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2381, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /N/scratch/scttest/Quartz/rlpipes_out/.snakemake/log/2024-03-15T124555.363780.snakemake.log

Sincerely, Cicada Dennis

CicadaDennis commented 5 months ago

Additional info on run error. The rlseq log JL01_S1_R1_001_hg38__rlseq.log contains:

$Coverage
[1] "/N/scratch/scttest/Quartz/rlpipes_out/coverage/JL01_S1_R1_001_hg38.bw"

$Peaks
[1] "/N/scratch/scttest/Quartz/rlpipes_out/peaks/JL01_S1_R1_001_hg38.broadPeak"

$genome
[1] "hg38"

Loading required package: BiocGenerics

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:stats’:

    IQR, mad, sd, var, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
    union, unique, unsplit, which.max, which.min

Loading required package: S4Vectors
Loading required package: stats4

Attaching package: ‘S4Vectors’

The following objects are masked from ‘package:base’:

    expand.grid, I, unname

Error in .order_seqlevels(chrom_sizes[, "chrom"]) : 
  !anyNA(m32) is not TRUE
Calls: <Anonymous> ... .get_chrom_info_for_registered_UCSC_genome -> GET_CHROM_SIZES -> .order_seqlevels -> stopifnot
Execution halted
millerh1 commented 3 months ago

Hi @CicadaDennis I have now updated the code to address this issue by removing the fine-grained package versions as you suggested. Please install from pypi or from GitHub for now (currently awaiting the official release of v0.9.4 on bioconda). Please try this out and let me know if you have any issues!