search

hawaiidatascience / metaflowmics

C-MAIKI ITS and 16S pipelines
Apache License 2.0
3 stars 5 forks source link

Using conda for workflow dependencies #4

Closed razsultana closed 5 years ago

razsultana commented 5 years ago

Hi Cedric, I managed to get all the dependencies for the 16S pipeline resolved using the python-based conda package manager, instead of relying on the environment modules on the Manoa HPC plus the sources for all the R and python packages that you included in the repository. It's very simple to use and explained in the README_conda.md under the 16S pipeline. In my opinion, it's a more flexible and elegant way of resolving dependecies. For example, if you want to add another R package that has multiple dependencies - it's just a matter of adding one word to the build_conda_env script. Anyway, have a look and see if you want to include this in your workflow repository. Regards, Razvan

razsultana commented 5 years ago

BTW, something you might want to know about. The latest version of mothur available in the bioconda channel (1.42.0) is not working with the current version of the workflow. I spent a few days trying to fix it (it was failing at step 12-TaxaFilter), but then I gave up and downgraded to the previous available version (1.41.3) and it worked. Mothur has a bad habit of changing the file names that come out of certain functions and the filenames are kinda crazy anyway (I noticed you had a step that is simplifying them, but because of that, it made debugging the error even harder). This problem is solved now by requiring a specific version of mothur, but you might want to look into getting some unit testing in place (i.e. making sure each mothur function call outputs files as expected).

I actually ran into a similar problem when I was testing mothur and trying to follow the SOP for 454 sequencing, which was on their website - it didn't work with the mothur version available back then! I sent the message to the author, but got no reply. Anyway, it seems that there's a pattern and it most likely won't go away :) Thought you might want to know that.

Puumanamana commented 5 years ago

I actually spent hours trying to debug this version problem with mothur, I wish I saw your message earlier. I opened couple issues on their GitHub and they are currently working on it. In the meantime, my fix is just to use a different version of Mothur for some steps. But eventually, they will have a stable enough version I can use for good! It's not completely stable for now (because of Mothur), but I should be able to release it soon.

I also created a Dockerfile to make the pipeline more reproducible and easier to install locally.