Template merge - Githubissues

lekhakaranam commented 4 years ago

Many thanks to contributing to nf-core/kmermaid!

Please fill in the appropriate checklist below (delete whatever is not relevant). These are the most common things requested on pull requests (PRs).

PR checklist

[x] PR is to dev rather than master
[ ] This comment contains a description of changes (with reason)
[ ] If you've fixed a bug or added code that should be tested, add tests!
[ ] If necessary, also make a PR on the nf-core/kmermaid branch on the nf-core/test-datasets repo
[ ] Ensure the test suite passes (nextflow run . -profile test,docker).
[ ] Make sure your code lints (nf-core lint .).
[ ] Documentation in docs is updated
[ ] CHANGELOG.md is updated
[ ] README.md is updated

Learn more about contributing: https://github.com/nf-core/kmermaid/tree/master/.github/CONTRIBUTING.md

olgabot commented 4 years ago

@lekhakaranam I'm also getting docker build and other errors, seemingly because of sourmash/bam2fasta/pysam/samtools conflicts. @pranathivemuri can you offer any insight?

(base)
 Fri  7 Aug - 08:04  ~/code/nf-core/kmermaid--olgabot/lekha-template   template_merge ✔ 
  docker build -t nfcore/kmermaid:template .
Sending build context to Docker daemon  11.34MB
Step 1/11 : FROM nfcore/base:1.9
 ---> f1c63ec29383
Step 2/11 : LABEL authors="Olga Botvinnik"       description="Docker image containing all software requirements for the nf-core/kmermaid pipeline"
 ---> Using cache
 ---> 40bc4d13ad0d
Step 3/11 : COPY environment.yml /
 ---> 7e4421061611
Step 4/11 : RUN conda env create -f /environment.yml && conda clean -a
 ---> Running in 948333058ad7
Collecting package metadata (repodata.json): ...working... done

Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package libdeflate conflicts for:
bioconda::sourmash=3.2.2 -> bam2fasta[version='>=1.0.1'] -> pysam -> libdeflate[version='>=1.0,<1.1.0a0|>=1.3,<1.4.0a0|>=1.5,<1.6.0a0|>=1.6,<1.7.0a0']
pysam=0.15.4 -> libdeflate[version='>=1.5,<1.6.0a0']
bioconda::samtools=1.10 -> htslib[version='>=1.10,<1.11.0a0'] -> libdeflate[version='>=1.3,<1.4.0a0|>=1.6,<1.7.0a0']

The command '/bin/sh -c conda env create -f /environment.yml && conda clean -a' returned a non-zero code: 1

olgabot commented 4 years ago

Okay I changed sourmash to use sourmash-minimal from conda-forge instead since we're just using the basic features and isolated the problem to just pysam and samtools:

(base)
 ✘  Fri  7 Aug - 08:21  ~/code/nf-core/kmermaid--olgabot/lekha-template   template_merge ✔ 
  docker build -t nfcore/kmermaid:dev .
Sending build context to Docker daemon  11.36MB
Step 1/11 : FROM nfcore/base:1.9
 ---> f1c63ec29383
Step 2/11 : LABEL authors="Olga Botvinnik"       description="Docker image containing all software requirements for the nf-core/kmermaid pipeline"
 ---> Using cache
 ---> 40bc4d13ad0d
Step 3/11 : COPY environment.yml /
 ---> fce08e89220d
Step 4/11 : RUN conda env create -f /environment.yml && conda clean -a
 ---> Running in 3cbd60425d4f
Collecting package metadata (repodata.json): ...working... done

Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package libdeflate conflicts for:
pysam=0.15.4 -> libdeflate[version='>=1.5,<1.6.0a0']
bioconda::samtools=1.10 -> htslib[version='>=1.10,<1.11.0a0'] -> libdeflate[version='>=1.3,<1.4.0a0|>=1.6,<1.7.0a0']

olgabot commented 4 years ago

We're already at the latest version of samtools, bioconda::samtools=1.10 but right now I'm testing upgrading pysam=0.15.4 --> bioconda::pysam=0.16.0 and it seems to be at least building! We'll see if the tests pass :)

olgabot commented 4 years ago

We'll have to make a separate PR with just the environment/dockerfile changes I believe, so that this PR can be built with the new environment.

@lekhakaranam it looks like the tests from the old .travis.yml didn't get moved to the .github/workflows/ci.yml file for continuous integration. There's a lot of tests to migrate :) I started the file for you, but can you add the rest of the tests, except for the "checking for failure" tests? Anything that has if [ $? -eq 0 ] is a "checking for failure" test. For those, can you make a separate ci_check_failure.yml file that runs the pipeline, but with the "check for failure" aka non-exit code zero at the end?

Here's the code to add to the test:

; if [ $? -eq 0 ]; then exit 1 ; else echo "Correctly failed ${{ matrix.profile_flags }}" ; fi

olgabot commented 4 years ago

Argh still failing because of pysam/samtools issues (log below). @pranathivemuri do you mind bumping the pysam version for bam2fasta to 0.16.0 to avoid the conflict?

bam2fasta install log

``` Ran pip subprocess with arguments: ['/opt/conda/envs/nf-core-kmermaid-1.0.0dev/bin/python', '-m', 'pip', 'install', '-U', '-r', '/condaenv.49zicayg.requirements.txt'] Pip subprocess output: Collecting bam2fasta==1.0.6 Cloning https://github.com/czbiohub/bam2fasta.git (to revision master) to /tmp/pip-install-b_panww1/bam2fasta Collecting sencha==1.0.1 Cloning https://github.com/czbiohub/sencha.git (to revision master) to /tmp/pip-install-b_panww1/sencha Requirement already satisfied, skipping upgrade: screed>=0.9 in /opt/conda/envs/nf-core-kmermaid-1.0.0dev/lib/python3.7/site-packages (from bam2fasta==1.0.6->-r /condaenv.49zicayg.requirements.txt (line 1)) (1.0.4) Requirement already satisfied, skipping upgrade: pathos>=0.2.5 in /opt/conda/envs/nf-core-kmermaid-1.0.0dev/lib/python3.7/site-packages (from bam2fasta==1.0.6->-r /condaenv.49zicayg.requirements.txt (line 1)) (0.2.5) Collecting pysam==0.15.3 Downloading pysam-0.15.3.tar.gz (3.2 MB) ==> WARNING: A newer version of conda exists. <== current version: 4.7.12 latest version: 4.8.3 Please update conda by running $ conda update -n base -c defaults conda Pip subprocess error: Running command git clone -q https://github.com/czbiohub/bam2fasta.git /tmp/pip-install-b_panww1/bam2fasta WARNING: Requested bam2fasta==1.0.6 from git+https://github.com/czbiohub/bam2fasta.git@master#egg=bam2fasta==1.0.6 (from -r /condaenv.49zicayg.requirements.txt (line 1)), but installing version 1.0.6rc1 Running command git clone -q https://github.com/czbiohub/sencha.git /tmp/pip-install-b_panww1/sencha WARNING: Requested sencha==1.0.1 from git+https://github.com/czbiohub/sencha.git@master#egg=sencha==1.0.1 (from -r /condaenv.49zicayg.requirements.txt (line 2)), but installing version 1.0.1.dev37+g1c58ac4 ERROR: Command errored out with exit status 1: command: /opt/conda/envs/nf-core-kmermaid-1.0.0dev/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b_panww1/pysam/setup.py'"'"'; __file__='"'"'/tmp/pip-install-b_panww1/pysam/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-b_panww1/pysam/pip-egg-info cwd: /tmp/pip-install-b_panww1/pysam/ Complete output (32 lines): # pysam: no cython available - using pre-compiled C # pysam: htslib mode is shared # pysam: HTSLIB_CONFIGURE_OPTIONS=None # pysam: (sysconfig) CC=gcc -pthread -B /opt/conda/envs/nf-core-kmermaid-1.0.0dev/compiler_compat -Wl,--sysroot=/ # pysam: (sysconfig) CFLAGS=-Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes # pysam: (sysconfig) LDFLAGS=-L/opt/conda/envs/nf-core-kmermaid-1.0.0dev/lib -Wl,-rpath=/opt/conda/envs/nf-core-kmermaid-1.0.0dev/lib -Wl,--no-as-needed -Wl,--sysroot=/ checking for gcc... gcc -pthread -B /opt/conda/envs/nf-core-kmermaid-1.0.0dev/compiler_compat -Wl,--sysroot=/ checking whether the C compiler works... no configure: error: in `/tmp/pip-install-b_panww1/pysam/htslib': configure: error: C compiler cannot create executables See `config.log' for more details checking for gcc... gcc -pthread -B /opt/conda/envs/nf-core-kmermaid-1.0.0dev/compiler_compat -Wl,--sysroot=/ checking whether the C compiler works... no configure: error: in `/tmp/pip-install-b_panww1/pysam/htslib': configure: error: C compiler cannot create executables See `config.log' for more details # pysam: htslib configure options: None Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-b_panww1/pysam/setup.py", line 241, in htslib_make_options = run_make_print_config() File "/tmp/pip-install-b_panww1/pysam/setup.py", line 68, in run_make_print_config stdout = subprocess.check_output(["make", "-s", "print-config"]) File "/opt/conda/envs/nf-core-kmermaid-1.0.0dev/lib/python3.7/subprocess.py", line 395, in check_output **kwargs).stdout File "/opt/conda/envs/nf-core-kmermaid-1.0.0dev/lib/python3.7/subprocess.py", line 472, in run with Popen(*popenargs, **kwargs) as process: File "/opt/conda/envs/nf-core-kmermaid-1.0.0dev/lib/python3.7/subprocess.py", line 775, in __init__ restore_signals, start_new_session) File "/opt/conda/envs/nf-core-kmermaid-1.0.0dev/lib/python3.7/subprocess.py", line 1522, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'make': 'make' ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. CondaEnvException: Pip failed The command '/bin/sh -c conda env create -f /environment.yml && conda clean -a' returned a non-zero code: 1 ```

pranathivemuri commented 4 years ago

what are igenomes for?

pranathivemuri commented 4 years ago

@olgabot now the tests are actually failing, let me know if you would like to look at it together tomorrow - get_software_versions is failing but there is no exact way to find out which of the packages is missing, would like your input - I fixed the linting issues, missing schema.json, Dockerfile formatting

so after this PR - https://github.com/nf-core/kmermaid/pull/30 there was no automatic TEMPLATE branch setup as per the instructions here - https://nf-co.re/developers/sync#setting-up-a-pipeline-for-syncing-retrospectively which is forcing us to manually sync now. please correct me if I am wrong!

We have to definitely do it this time so we don't suffer through this again

olgabot commented 4 years ago

I think I see what's happening:

WARN: Access to undefined parameter `publish_dir_mode` -- Initialise it to a default value eg. `params.publish_dir_mode = some_value`
Error executing process > 'sourmash_compute_sketch_fastx_nucleotide (SRR4050380__molecule-dna__ksize-3__num_hashes-4__track_abundance-false)'

Caused by:
  No enum constant nextflow.processor.PublishDir.Mode.NULL

From: https://github.com/nf-core/kmermaid/pull/93/checks?check_run_id=1192829083

nf-core / kmermaid

Template merge #93

PR checklist