new wochenende env: python3.6 EOL Dec 2021

[colindaven]

upgrade check deps

[colindaven]

@poer-sophia Can you please make a new Wochenende minimal environment on a new branch.

• New minimap2 (see issues github)
• current python 3.9?
• test compatibility with haybaler, growth rate, plots, reporting etc.
• add bedtools to env if not already present (TODO), remove Toms bedtools binary 40mb file (done)

• If you can't complete it before your holiday don't worry, just put it in a branch and we can continue later.

What do you think ?

[poer-sophia]

Yeah I will see what I can do.

[poer-sophia]

what excatly is a minimal enviroment? What is the difference betwenn the normal and the minimal enviroment?

[colindaven]

The minimal includes the actual tools we need. Normal includes all packages, including the weird stuff. I think the minimal makes a lot more sense, but we haven't tested widely.

[poer-sophia]

Okay I will test and try to do this.

[poer-sophia]

I always though that only the packages and their depenencies that are imported at the beginning of python files are in the conda enviroment. But e.g. minimap2 never gets importet. So how does this work? What else than the imported packages need to be in the conda env?

And another thing that confuses me: in the config yaml there is a section for different toll and their paths. For a few tools there is really a path, e.g. path_afterqc: /mnt/ngsnfs/tools/afterQC/AfterQC-0.9.6/after.py but for other tool the path is just the name of the tool e.g. path_minimap2: minimap2. Why is that so?

[colindaven]

No, the conda env is a bunch of binary compiled files - tools (look in your miniconda installation in the bin directory). Can include perl, python etc too, but they are not python libraries.

Not sure I really get your question 1 though (maybe you just mean where are the python dependencies? These are sometimes in miniconda, but often installed locally by python -m pip)

? Here is minimap2 in the full conda env:

https://github.com/MHH-RCUG/Wochenende/blob/dev/env.wochenende.minimal.yml#L23

• minimap2=2.17

In config.yaml, you can either specify the full path to eg bwa say /usr/bin/bwa or just the binary itself, bwa. Then your system needs to be configured to find bwa, normally by using either a miniconda env or loaded globalenv or by having it in your $PATH variable (such as having installed it to a system directory like /usr/bin or /usr/local/bin etc...

[poer-sophia]

Trim galore scheint nicht mit python 3.9 kompatible zu sein...

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - trim-galore -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0|3.4.*']

Your python: python=3.9

[colindaven]

Ok, a py3.8 problem has been reported with a potential workaround: https://github.com/FelixKrueger/TrimGalore/issues/99

[colindaven]

Alternatively, it might by time to try mamba, it has a good reputation for resolving dependencies quicker than conda.

https://github.com/mamba-org/mamba

[poer-sophia]

Your first coment doesnt work... the only higher version that seems to work is 1.7. I will see if it works better with mamba

[poer-sophia]

Doesnt wotk either:

Pinned packages:

• python 3.9.5

Encountered problems while solving:

• nothing provides python 3.4* needed by cutadapt-1.10-py34_0

[colindaven]

You can see here https://bioconda.github.io/recipes/cutadapt/README.html?highlight=cutadapt#package-package%20'cutadapt'

that cutadapt needs py3.7 or below I believe. Certainly not 3.8 or 3.9, which is a real shame.

Make sure you set a modern version of cutadapt like 3.4-1

[poer-sophia]

Yes, should I then do the whole enviroment for pyhton 3.7?

[colindaven]

Sure, yep. Hope that it works better

[poer-sophia]

Seems to finally work. I just started testing to see if somethign is missing but until now I could install everything

[colindaven]

Nice. You can overwrite the wochenende_minimal.yml env specification, we all need to upgrade our conda envs before the end of the year anyway. Would be awesome to put this is a container but that is a bit tricky and time consuming, I started, Tobias tried a bit too, but not done yet.

[poer-sophia]

Everything seems to work with the minimal enviroment, I could push it to dev or a new branch i you like so you can test it too but I have a Problem with the test script @B1T0: If I do: bash run test_Wochenende_SLURM.sh

I get: ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...] pytest: error: unrecognized arguments: -n inifile: None rootdir: /mnt/ngsnfs/gen/rcug_lw/sophias_projekte/Wochenende/test

[colindaven]

Nice, please put in a branch "new_env" or similar

[poer-sophia]

New env is on branch minimal_env. The it is in the wochenende_minimal.yml but the name of the enviroment itself just wochenene.

[colindaven]

See follow up testing in https://github.com/MHH-RCUG/Wochenende/issues/149