Open npavlovikj opened 4 years ago
Hi Natasha,
Thanks for letting me know about this and for providing a detailed issue. Basically, these errors are of no concern. It just has to do with the fact that some of my coding, while it all works as far as I can tell, is a bit ugly. There are a range of ancillary files that could be generated by Cenote-Taker2, and sometimes they are not present at the end. That said, I've update the code to eliminate this message. Also, I've done a couple debugging updates since 3/26/2020, so please 'git pull' the repo.
Going through your log file, I did see that the IRF (inverted repeat finder) program isn't found.
/home/deogun/npavlovikj/.conda/envs/ct2/bin/cenote-taker2.0.1.sh: /home/deogun/npavlovikj/.conda/envs/ct2/bin/irf307.linux.exe: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
IRF was a bit trickier to include in the install as it's not available on Anaconda and is meant for 32 bit systems. A google search supports the idea that this type of error arises from 64 bit/32 bit incompatibility. You might try to resolve this, but this program is not strictly necessary for the rest of the code. You just won't know which contigs are flanked by inverted terminal repeats.
I hope this was at least a little bit helpful and please let me know if you run into any other issues.
Best,
Mike
Hi @mtisza1 ,
Thank you so much for your prompt reply and fixes, I highly appreciate it! I got the newest commit from the repo, and I can confirm that the messages do not appear anymore and the test job finished fine.
When we install tools on our HPC cluster, we try to package them in conda packages if that is feasible.
Therefore, I created conda package for cenote-taker2
and added it to our HCC conda channel, https://anaconda.org/hcc/cenote-taker2.
I followed the dependencies you have used in your conda environment and your code.
Everything with exception of phanotate
was available in bioconda
, and I added phanotate
in our channel.
You can see the full recipe here, https://github.com/unlhcc/hcc-conda-recipes/tree/master/recipes/cenote-taker2/2020.04.01.
It is always good to have the data and the script directories separate, so I use a variable CT2_DIR
to point to the data directory, and I patched the code respectively.
Briefly, if someone uses the conda package for cenote-taker2
, one needs to set the CT2_DIR
variable to point to the data directory, and follow the instructions in the post-link.sh
script to download the databases.
The download of the databases is not part of the conda recipe, and it is not common practice to do that mostly because of their size, so the download of the databases for cenote-taker2
is in the download-db.sh
script the user needs to run after the environment is created.
Regarding irf307.linux.exe
- you are right that it works only on 32-bit systems, and ours are 64-bit.
Unfortunately, looks like the provided binary is only for 32-bit systems.
In order to make IRF reproducible, I created a Docker image for it.
On our clusters we support Singularity, so replacing irf307.linux.exe $NONCIR 2 3 5 80 10 40 500000 10000 -d -h
with singularity exec docker://unlhcc/irf irf307.linux.exe $NONCIR 2 3 5 80 10 40 500000 10000 -d -h
works fine.
Using the mentioned conda package for cenote-taker2
with the Singularity image for IRF worked well with the test data on our clusters.
I am letting you know about this in case some of the users of cenote-taker2
notices the conda package.
I do not plan to maintain the recipe, unless you have updated the code and the user requests the changes.
However, you are more than welcome to modify and re-use the conda recipe if you want, or add your changes to our repo.
I am sorry for the long reply, and I hope what I wrote makes sense. If you have any questions, please let me know.
Thank you, Natasha
Hey Natasha,
This is really interesting! I'm a bit new to the whole packaging/making images thing, but I will try to keep this conda package updated if possible.
However, I tried to install your package conda install -c hcc cenote-taker2
and got an error:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
I noticed a few differences from my .yaml file including
yours:
tbl2asn-forever=25.7.1f bbmap=37.62 coreutils perl phanotate=2019.08.09 matplotlib-base=3.1.3 mine: tbl2asn=25.7 bbtools=37.62 cmake=3.14.0 matplotlib=3.1.3
Also I didn't see the channels that I listed in my .yaml in yours. I don't know if I'm doing something wrong, but I wanted to make you aware.
Let me know if you have ideas. Thanks!
Hi @mtisza1 ,
I am sorry, I forgot to mention that our HCC conda packages depend on packages from conda-forge
, bioconda
and defaults
as the main and well-maintained conda channels.
So, the conda command for installing should actually be:
conda create -n cenote-taker2 -c hcc -c conda-forge -c bioconda -c defaults cenote-taker2=2020.04.01
Also, I would suggest you to use conda create
instead of conda install
.
The first command will create a completely new environment with cenote-taker2
, while the latter one is installing packages in already existing environment which may cause incompatibility issues as you have observed.
When we create conda packages for our channel, we try to follow the infrastructure of bioconda
and install packages that only depend on the channels conda-forge
, bioconda
and defaults
.
I saw you used the AgBiome
channel.
You can definitely add packages from this channel - I didn't have the need to since all the dependencies are available in the main channels listed above.
Regarding the difference in the packages you listed:
As you can see, the idea with the dependencies in the recipe is to not use https://github.com/mtisza1/Cenote-Taker2/blob/master/install_scripts/cenote_install1.sh anymore. With conda recipes, it is preferable to have a conda package for each dependency. So, all the packages you are downloading and installing in the install script and now conda packages that can just be used.
I hope this makes sense.
Please try installing cenote-taker2
again, and let me know if you have any other questions or issues.
Thank you, Natasha
OK I've finally gotten around to trying this. It seems to work! I'll make a note on the repo about this option. I appreciate that you took the time to explain this to me, and I think this is a valuable addition.
Best regards,
Mike
P.S. I'm closing this issue.
Hi Natasha,
It's been a couple years and there have been many updates to Cenote-Taker 2
, now at v2.1.5
. Occasionally, I'm getting questions from people who are using the bioconda package you created, and it's difficult for me to give guidance about updates to them.
You are probably quite busy, but I was wondering if you'd be able to help me update the bioconda package. Overall, the install is much easier than it was 2 years ago.
Best,
Mike
Hi,
I am trying to install "cenote-taker2" on our HPC system for a local research group. I downloaded all databases and dependencies using the commit from 03/26/2020.
I did a test run you have provided in the Wiki page using the command:
Most of the programs finished ok I think, but I am getting the following error at the end:
These files indeed do not exist in the output directory, thus the message. I was wondering if this type of error message is familiar to you, and whether you have some suggestions on how to fix it. I am using the "testcontigs_DNA_ct2.fasta" file you have provided, and a dummy "template.sbt" file. Please find the complete log here, cenote-taker2.log
I am looking forward to hearing from you, and if you need any additional information, please let me know.
Thank you, Natasha