search

wtsi-team112 / ncov2019-artic-nf

A Nextflow pipeline for running the ARTIC network's fieldbioinformatics tools (https://github.com/artic-network/fieldbioinformatics), with a focus on ncov2019 - adapted to run at the Wellcome Sanger Institute
0 stars 4 forks source link

Feature/update bam to cram #34

Open sb43 opened 4 years ago

sb43 commented 4 years ago

Updated bamToCram command line and samtools version in conda,

sb43 commented 4 years ago

Hi @sb10 please check the error, it seems missing sif files, I have merged my master with upstream/master could that be an issue.

sb10 commented 4 years ago

Hi @sb10 please check the error, it seems missing sif files, I have merged my master with upstream/master could that be an issue.

I'm not sure if that's the issue. You can run what the github tests run locally to debug. It's .github/scripts/test_PR_against_release.sh that is failing, so just do what .github/workflows/pull_request.yml does before that test, then run it the same way.

sb43 commented 4 years ago

, Test continues even after failure in building containers. bash scripts/build_singularity_containers.sh

Possibly due to issue in conda 4.8.2

Please see issue reported here: https://github.com/conda/conda/issues/9681

>>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/conda_env/cli/main.py", line 80, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/opt/conda/lib/python3.7/site-packages/conda_env/cli/main_update.py", line 124, in execute
    result[installer_type] = installer.install(prefix, specs, args, env)
  File "/opt/conda/lib/python3.7/site-packages/conda_env/installers/conda.py", line 32, in install
    prune=getattr(args, 'prune', False), update_modifier=UpdateModifier.FREEZE_INSTALLED)
  File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 117, in solve_for_transaction
    should_retry_solve)
  File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 158, in solve_for_diff
    force_remove, should_retry_solve)
  File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 281, in solve_final_state
    ssc = self._run_sat(ssc)
  File "/opt/conda/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
    return f(*args, **kwds)
  File "/opt/conda/lib/python3.7/site-packages/conda/core/solve.py", line 808, in _run_sat
    should_retry_solve=ssc.should_retry_solve
  File "/opt/conda/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
    return f(*args, **kwds)
  File "/opt/conda/lib/python3.7/site-packages/conda/resolve.py", line 1318, in solve
    self.find_conflicts(specs, specs_to_add, history_specs)
  File "/opt/conda/lib/python3.7/site-packages/conda/resolve.py", line 347, in find_conflicts
    bad_deps = self.build_conflict_map(specs, specs_to_add, history_specs)
  File "/opt/conda/lib/python3.7/site-packages/conda/resolve.py", line 507, in build_conflict_map
    root, search_node, dep_graph, num_occurances)
  File "/opt/conda/lib/python3.7/site-packages/conda/resolve.py", line 369, in breadth_first_search_for_dep_graph
    last_spec = MatchSpec.union((path[-1], target_paths[-1][-1]))[0]
  File "/opt/conda/lib/python3.7/site-packages/conda/models/match_spec.py", line 481, in union
    return cls.merge(match_specs, union=True)
  File "/opt/conda/lib/python3.7/site-packages/conda/models/match_spec.py", line 475, in merge
    reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
  File "/opt/conda/lib/python3.7/site-packages/conda/models/match_spec.py", line 475, in <lambda>
    reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
  File "/opt/conda/lib/python3.7/site-packages/conda/models/match_spec.py", line 502, in _merge
    final = this_component.union(that_component)
  File "/opt/conda/lib/python3.7/site-packages/conda/models/match_spec.py", line 764, in union
    return '|'.join(options)
TypeError: sequence item 0: expected str instance, Channel found

$ /opt/conda/bin/conda-env update -f /extras.yml -n artic-ncov2019

environment variables: CIO_TEST= CONDA_AUTO_UPDATE_CONDA=false CONDA_ROOT=/opt/conda PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin REQUESTS_CA_BUNDLE= SSL_CERT_FILE=

 active environment : None
   user config file : /root/.condarc

populated config files : conda version : 4.8.2 conda-build version : not installed python version : 3.7.6.final.0 virtual packages : __glibc=2.28 base environment : /opt/conda (writable) channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /opt/conda/pkgs /root/.conda/pkgs envs directories : /opt/conda/envs /root/.conda/envs platform : linux-64 user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Linux/4.4.0-116-generic debian/10 glibc/2.28 UID:GID : 0:0 netrc file : None offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

ABORT: Aborting with RETVAL=255 Cleaning up... Built container /home/ubuntu/mymount2/heron/ncov2019-artic-nf/artic-ncov2019-illumina.sif Built container /home/ubuntu/mymount2/heron/ncov2019-artic-nf/artic-ncov2019-nanopore.sif

sb10 commented 4 years ago

Can you try with conda 4.8.3 and see if that fixes it?

sb43 commented 4 years ago

Can you try with conda 4.8.3 and see if that fixes it?

Doesn't look straightforward as Dockerfile ( environments/illumina/Dockerfile) uses prebuild FROM continuumio/miniconda3:latest which is 4.8.2

Also tried to upate conda after default installation, it doesn't seem to work: RUN /opt/conda/bin/conda update -n base -c defaults conda

sb43 commented 4 years ago

Can you try with conda 4.8.3 and see if that fixes it?

Doesn't look straightforward as Dockerfile ( environments/illumina/Dockerfile) uses prebuild FROM continuumio/miniconda3:latest which is 4.8.2

Also tried to upate conda after default installation, it doesn't seem to work: RUN /opt/conda/bin/conda update -n base -c defaults conda

@sb10 Shall I wait for upstream repo to fix the test for .sif generation ?

sb10 commented 4 years ago

Can you try with conda 4.8.3 and see if that fixes it?

Doesn't look straightforward as Dockerfile ( environments/illumina/Dockerfile) uses prebuild FROM continuumio/miniconda3:latest which is 4.8.2 Also tried to upate conda after default installation, it doesn't seem to work: RUN /opt/conda/bin/conda update -n base -c defaults conda

@sb10 Shall I wait for upstream repo to fix the test for .sif generation ?

By upstream you mean connor-lab/ncov2019-artic-nf? No, if the test needs improving, just make and commit changes to the test as part of this PR.

Sorry, I don't understand the technicalities around conda, so can't help. If you can't get it working, just drop the "index at the same time" feature and limit this PR to just converting all the bams.

kjsanger commented 4 years ago

Re. that Conda error, I was getting similar about 2 weeks ago, caused by upstream pinning Conda package versions in environments/extras.yml in such a way that the dependencies became unsolvable. Unfortunately Conda crashed on this in the same way as above so we couldn't see what the problem was.

Reverting Conda to the most recent 4.7.x release and running the build again may tell you what's wrong in the Conda environment. It worked in my case.

(My rule of thumb is to avoid running a version of Conda more recent than the most patched previous major release)

sb43 commented 4 years ago

Re. that Conda error, I was getting similar about 2 weeks ago, caused by upstream pinning Conda package versions in environments/extras.yml in such a way that the dependencies became unsolvable. Unfortunately Conda crashed on this in the same way as above so we couldn't see what the problem was.

Reverting Conda to the most recent 4.7.x release and running the build again may tell you what's wrong in the Conda environment. It worked in my case.

(My rule of thumb is to avoid running a version of Conda more recent than the most patched previous major release)

Thanks @kjsanger and @sb10, I will try to revert conda to 4.7.x and rerun the test

sb10 commented 4 years ago

For the conda issue, try rebasing with latest upstream, which is using miniconda v3 for the tests.

sb10 commented 4 years ago

Can you see if you can resolve the conflicts to allow a rebase of your changes?

sb10 commented 4 years ago

Or just cherry pick your changes in a new branch and do a PR direct to upstream.

sb43 commented 4 years ago

Or just cherry pick your changes in a new branch and do a PR direct to upstream.

I did already have PR with upstream master, other commits from upstream master were merged with my branch, these are already part of connonr-lab master branch.