widdowquinn
commented
4 years ago There were a couple of separate problems involved in @J123T's issue:
- The documentation is not as clear as it could be, concerning the structure of this repository:
- The
masterbranch contains the working, stablefind_differential_primers.pycode that has been used in previous publications. The documentation/installation instructions for this are found at https://github.com/widdowquinn/find_differential_primers/blob/master/README.md - The branch that you land on when you come to this repository is the
diagnostic_primersbranch. This is the bleeding-edge live development ofpdp, which is the new version offind_differential_primers.pyand has a completely different CLI. Its documentation is at https://github.com/widdowquinn/find_differential_primers/blob/diagnostic_primers/README.md and is being activately developed at https://pdp.readthedocs.io/en/diagnostic_primers/index.html
This has caused confusion with other users, too: #48 and #49
I need to make the documentation clearer to reflect this.
-
A bulk change made to
biocondarecipes about a year ago silently broke thefind_differential_primerscondainstallation. I did not realise this until @J123T's issue. This issue prevented installation offind_differential_primersunder Python3 and, because the code is not Python2-compatible, it would not run under Python2. Details are in #51 -
The
Bio.alphabetclass was made obsolete inbiopython1.78. The version ofbiopythonwas not pinned in thebiocondarecipe, causing the observed error:
File "/home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/find_differential_primers.py-0.1.3.dev0-py3.7.egg/EGG-INFO/scripts/find_differential_primers.py", line 744, in check_ambiguity_codes
AttributeError: 'Seq' object has no attribute 'alphabet'- I have made a pull request to update the recipe, pinning
biopythonat a pre-1.78 version: https://github.com/bioconda/bioconda-recipes/pull/24756 - The workaround is to install a specific pre-1.78 version of
biopythonbefore runningfind_differential_primers.py, e.g.:
conda install -c bioconda biopython=1.77
J123T
nick-youngblut
Dear all,
A few days ago, I contacted Leighton via e-mail as I - completely new to the field of Ubuntu and command line usage - was not able to install Find differential Primers on Ubuntu 18.04.5 LTS. We decided to publish a summary of our conversation here, as it might help also others to get Find differential Primers running. Here is what we discussed:
First, I tried to install Find differential Primers following the instructions of the QuickStart Guide https://pdp.readthedocs.io/en/diagnostic_primers/quickstart.html#installation. However, both installation options (“pip install pdp” and “conda install -c bioconda pdp”) did not work. Leighton told me, that he released documentation early on ReadTheDocs and that the installation instructions for pip, conda etc. are not valid, yet. He said that “there are currently two active versions of pdp/find_differential_primers: v0.1.4 - the one used in previous publications (https://github.com/widdowquinn/find_differential_primers/tree/master), and the “new” version with a different command-line interface, which is the development version. The ReadTheDocs instructions apply only to the development version. This is not available on pip/conda because it is not quite ready yet.” Leighton told me that there are two different ways of installing Find differential Primers:
1) Installation via bioconda: this downloads and installs the complete package, plus the requirements necessary to run it, in your current conda environment. This is complete once you’ve executed
conda install -c bioconda find_differential_primers. The executable find_differential_primers.py should be available in your $PATH.2) Installation from source: this requires you to download the source code/repository from GitHub (git clone) then install (python setup.py install) and optionally install other dependencies if they were not already satisfied (note: this installs Python dependencies automatically, but the third-party dependencies such as Prodigal, EMBOSS etc are not installed). For installation from source, you can follow the instructions listed here: https://github.com/widdowquinn/find_differential_primers/blob/master/README.md
So I went on and tried to install Find differential Primers v0.1.4 via miniconda. Following the instruction of Leighton, I created a new conda environment, named “test1”, containing python2.7. I activated the environment and installed Find differential Primers. Here are the respective commands:
conda create --name test1 python=2.7 conda activate test1 conda install -c bioconda find_differential_primers
To test, if Find differential Primers was running, I executed the command “find_differential_primers.py -h” (the environment “test1” was still active!). Subsequently, options regarding the usage of Find differential Primers were listed, so installation was successful. Next, I downloaded three FASTA files from https://github.com/widdowquinn/find_differential_primers/tree/diagnostic_primers/tests/walkthrough/sequences and created a test.conf file with the following content:
Pba_SCRI1043 Pectobacterium,atrosepticum /home/treffonj/Find_differential_Primers/test/sequences/GCF_000011605.1.fasta - - - Pwa_CFBP_3304 Pectobacterium,wasabiae /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1.fasta - - - Pbe_NCPPB_2795 Pectobacterium,betavasculorum /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1.fasta - - -
In terminal I navigated to the directory containing the test.conf file and ran the command “find_differential_primers.py -i test.conf -v” (the environment “test1” was still active!). The output I got differed from the output shown on the web site https://github.com/widdowquinn/find_differential_primers/blob/master/README.md.
(test1) treffonj@ukmsrv284:~/Find_differential_Primers/test$ find_differential_primers.py -i test.conf -v INFO: # find_differential_primers.py logfile INFO: # Run: Thu Oct 8 12:04:18 2020 INFO: {'psizemin': 50, 'oligoosize': 20, 'verbose': True, 'mingc': 30, 'maxgc': 80, 'minsize': 18, 'maxtm': 60, 'mismatchpercent': 10, 'useblast': False, 'sge': False, 'blastdb': None, 'mintm': 58, 'logfile': None, 'cleanonly': False, 'osize': 20, 'oligominsize': 13, 'maxsize': 22, 'ogcpercent': 55, 'filename': 'test.conf', 'oligootm': 69, 'log_dir': None, 'oligoogcpercent': 55, 'noclassify': False, 'noprimer3': False, 'noprodigal': False, 'blast_exe': 'blastn', 'psizemax': 150, 'otm': 59, 'oligomintm': 68, 'oligomingc': 30, 'oligomaxtm': 70, 'prodigal_exe': 'prodigal', 'oligomaxpolyx': 3, 'single_product': None, 'oligomaxgc': 80, 'numreturn': 20, 'outdir': 'differential_primer_results', 'maxpolyx': 3, 'cpus': 8, 'noprimersearch': False, 'hybridprobe': False, 'oligomaxsize': 30, 'filtergc3prime': False, 'nocds': False, 'debug': False, 'clean': False, 'keep_logs': False, 'eprimer3_exe': 'eprimer3', 'psizeopt': 100} INFO: [] INFO: Creating list of genomes from config file test.conf ... INFO: ... created GenomeData object for Pba_SCRI1043 ... INFO: GenomeData object: Pba_SCRI1043 Families: ['Pectobacterium', 'atrosepticum'] Sequence file: /home/treffonj/Find_differential_Primers/test/sequences/GCF_000011605.1.fasta Feature file: None Primers file: None PrimerSearch file: None Primers: 0
INFO: ... created GenomeData object for Pwa_CFBP_3304 ... INFO: GenomeData object: Pwa_CFBP_3304 Families: ['Pectobacterium', 'wasabiae'] Sequence file: /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1.fasta Feature file: None Primers file: None PrimerSearch file: None Primers: 0
INFO: ... created GenomeData object for Pbe_NCPPB_2795 ... INFO: GenomeData object: Pbe_NCPPB_2795 Families: ['Pectobacterium', 'betavasculorum'] Sequence file: /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1.fasta Feature file: None Primers file: None PrimerSearch file: None Primers: 0
INFO: ... created 3 GenomeData objects (0.000s) INFO: Checking for multiple sequences ... INFO: ... /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1.fasta describes multiple sequences ... INFO: Concatenating sequences from /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1.fasta ... INFO: ... wrote concatenated data to /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1_concatenated.fas (0.075s) INFO: ... clearing feature and primer file locations ... INFO: ... /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1.fasta describes multiple sequences ... INFO: Concatenating sequences from /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1.fasta ... INFO: ... wrote concatenated data to /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1_concatenated.fas (0.069s) INFO: ... clearing feature and primer file locations ... INFO: ... checked 3 GenomeData objects (0.293s) INFO: Checking for ambiguity symbols... INFO: Checking /home/treffonj/Find_differential_Primers/test/sequences/GCF_000011605.1.fasta... INFO: Checking /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1_concatenated.fas... WARNING: Found 1 ambiguity codes in /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1_concatenated.fas (replacing) INFO: Writing new sequences to /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1_concatenated_noamb.fas INFO: ... clearing feature and primer file locations ... INFO: Checking /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1_concatenated.fas... INFO: ... checked 3 objects (0.256s) INFO: loading sequence from /home/treffonj/Find_differential_Primers/test/sequences/GCF_000011605.1.fasta INFO: loading sequence from /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1_concatenated_noamb.fas INFO: loading sequence from /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1_concatenated.fas INFO: Python executable location: /home/treffonj/miniconda2/envs/test1/bin/python Traceback (most recent call last): File "/home/treffonj/miniconda2/envs/test1/bin/find_differential_primers.py", line 2020, in
embossversion = str(embossversion, "utf-8")
TypeError: str() takes at most 1 argument (2 given)
Leighton told me the following: “bioconda appear to have accidentally broken the installation. I wrote find_differential_primers.py to work with Python3; I never meant it to work with Python2, but bioconda have - without asking - modified my installation definition to make it only work with Python2 (which is why I had to ask you to make the test1 environment be Python2.7; otherwise it wouldn’t have installed at all). Unfortunately, you’ve hit on one of the many differences between Python2 and Python3 that mean find_differentlal_primers.py doesn’t run at all under Python2”
Creating a conda environment running with Python3 might circumvent this problem. To do this, he told me to run the following commands:
conda create --name fdp python=3.7 conda activate fdp git clone git@github.com:widdowquinn/find_differential_primers.git cd find_differential_primers git checkout master conda install --file requirements-thirdparty-primer3_114.txt python setup.py install
To install the more recent version (pdp), this should work:
conda create --name pdp python=3.7 conda activate pdp git clone git@github.com:widdowquinn/find_differential_primers.git cd find_differential_primers/ conda install --file requirements.txt conda install --file requirements-thirdparty-primer3_237.txt python setup.py install
Caution: If you do not have a SSH key, the command “git clone git@github.com:widdowquinn/find_differential_primers.git” will fail! You must set up this key following the instructions here: https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh. If you don’t want to set up this key, you can simply use the command “git clone https://github.com/widdowquinn/find_differential_primers.git”instead.
I chose the first instruction and created the “fdp” environment, activated it, cloned the repository with the command “git clone https://github.com/widdowquinn/find_differential_primers.git”, entered the directory find_differential_primers and listed the content of this directory:
(fdp) treffonj@ukmsrv284:~/find_differential_primers$ ls bin pdp.py requirements.txt CITATION README.md setup.py diagnostic_primers requirements-dev.txt tests docs requirements-pip.txt walkthrough.sh git_hooks requirements-thirdparty-primer3_114.txt LICENSE requirements-thirdparty-primer3_237.txt
After executed “git checkout master”, the content of the directory looked like this:
(fdp) treffonj@ukmsrv284:~/find_differential_primers$ git checkout master Branch 'master' set up to track remote branch 'master' from 'origin'. Switched to a new branch 'master'
(fdp) treffonj@ukmsrv284:~/find_differential_primers$ ls CITATION LICENSE README.md requirements-pip.txt requirements.txt setup.py utils find_differential_primers pylints requirements-dev.txt requirements-thirdparty.txt sample tests
This command deleted the files “requirements-thirdparty-primer3_114.txt” and “requirements-thirdparty-primer3_237.txt” and, instead, created the file “requirements-thirdparty.txt”. So, instead of using the command “conda install --file requirements-thirdparty-primer3_114.txt” suggested by Leighton, I used the command “conda install --file requirements-thirdparty.txt”, which worked well. Subsequently, I ran “python setup.py install”.
Again, to test if Find differential Primers was running, I executed the command “find_differential_primers.py -h” and got options regarding the usage of Find differential Primers. Everything was fine. Then, as before, in terminal I went to the directory containing the test.conf file and ran the command “find_differential_primers.py -i test.conf –v”, which resulted in the following output (which is still different from the output shown on the web site https://github.com/widdowquinn/find_differential_primers/blob/master/README.md)
(fdp) treffonj@ukmsrv284:~/Find_differential_Primers/test$ find_differential_primers.py -i test.conf -v INFO: # find_differential_primers.py logfile INFO: # Run: Fri Oct 9 08:29:27 2020 INFO: {'filename': 'test.conf', 'outdir': 'differential_primer_results', 'numreturn': 20, 'hybridprobe': False, 'filtergc3prime': False, 'single_product': None, 'prodigal_exe': 'prodigal', 'eprimer3_exe': 'eprimer3', 'blast_exe': 'blastn', 'blastdb': None, 'useblast': False, 'nocds': False, 'noprodigal': False, 'noprimer3': False, 'noprimersearch': False, 'noclassify': False, 'osize': 20, 'minsize': 18, 'maxsize': 22, 'otm': 59, 'mintm': 58, 'maxtm': 60, 'ogcpercent': 55, 'mingc': 30, 'maxgc': 80, 'psizeopt': 100, 'psizemin': 50, 'psizemax': 150, 'maxpolyx': 3, 'mismatchpercent': 10, 'oligoosize': 20, 'oligominsize': 13, 'oligomaxsize': 30, 'oligootm': 69, 'oligomintm': 68, 'oligomaxtm': 70, 'oligoogcpercent': 55, 'oligomingc': 30, 'oligomaxgc': 80, 'oligomaxpolyx': 3, 'cpus': 8, 'sge': False, 'clean': False, 'cleanonly': False, 'logfile': None, 'verbose': True, 'debug': False, 'keep_logs': False, 'log_dir': None} INFO: [] INFO: Creating list of genomes from config file test.conf ... /home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/find_differential_primers.py-0.1.3.dev0-py3.7.egg/EGG-INFO/scripts/find_differential_primers.py:686: DeprecationWarning: 'U' mode is deprecated for line in [l.strip() for l in open(filename, 'rU') INFO: ... created GenomeData object for Pba_SCRI1043 ... INFO: GenomeData object: Pba_SCRI1043 Families: ['Pectobacterium', 'atrosepticum'] Sequence file: /home/treffonj/Find_differential_Primers/test/sequences/GCF_000011605.1.fasta Feature file: None Primers file: None PrimerSearch file: None Primers: 0
INFO: ... created GenomeData object for Pwa_CFBP_3304 ... INFO: GenomeData object: Pwa_CFBP_3304 Families: ['Pectobacterium', 'wasabiae'] Sequence file: /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1.fasta Feature file: None Primers file: None PrimerSearch file: None Primers: 0
INFO: ... created GenomeData object for Pbe_NCPPB_2795 ... INFO: GenomeData object: Pbe_NCPPB_2795 Families: ['Pectobacterium', 'betavasculorum'] Sequence file: /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1.fasta Feature file: None Primers file: None PrimerSearch file: None Primers: 0
INFO: ... created 3 GenomeData objects (0.000s) INFO: Checking for multiple sequences ... /home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/find_differential_primers.py-0.1.3.dev0-py3.7.egg/EGG-INFO/scripts/find_differential_primers.py:714: DeprecationWarning: 'U' mode is deprecated seqdata = [s for s in SeqIO.parse(open(gd_obj.seqfilename, 'rU'), INFO: ... /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1.fasta describes multiple sequences ... INFO: Concatenating sequences from /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1.fasta ... /home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/find_differential_primers.py-0.1.3.dev0-py3.7.egg/EGG-INFO/scripts/find_differential_primers.py:451: DeprecationWarning: 'U' mode is deprecated 'rU'), INFO: ... wrote concatenated data to /home/treffonj/Find_differential_Primers/test/sequences/GCF_000291725.1_concatenated.fas (0.057s) INFO: ... clearing feature and primer file locations ... INFO: ... /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1.fasta describes multiple sequences ... INFO: Concatenating sequences from /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1.fasta ... INFO: ... wrote concatenated data to /home/treffonj/Find_differential_Primers/test/sequences/GCF_000749845.1_concatenated.fas (0.054s) INFO: ... clearing feature and primer file locations ... INFO: ... checked 3 GenomeData objects (0.230s) INFO: Checking for ambiguity symbols... INFO: Checking /home/treffonj/Find_differential_Primers/test/sequences/GCF_000011605.1.fasta... /home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/find_differential_primers.py-0.1.3.dev0-py3.7.egg/EGG-INFO/scripts/find_differential_primers.py:741: DeprecationWarning: 'U' mode is deprecated seqlist = list(SeqIO.parse(open(gd_obj.seqfilename, 'rU'), 'fasta')) Traceback (most recent call last): File "/home/treffonj/miniconda2/envs/fdp/bin/find_differential_primers.py", line 4, in
import('pkg_resources').run_script('find-differential-primers.py==0.1.3.dev0', 'find_differential_primers.py')
File "/home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/pkg_resources/init.py", line 650, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/pkg_resources/init.py", line 1453, in run_script
exec(script_code, namespace, namespace)
File "/home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/find_differential_primers.py-0.1.3.dev0-py3.7.egg/EGG-INFO/scripts/find_differential_primers.py", line 1663, in
File "/home/treffonj/miniconda2/envs/fdp/lib/python3.7/site-packages/find_differential_primers.py-0.1.3.dev0-py3.7.egg/EGG-INFO/scripts/find_differential_primers.py", line 744, in check_ambiguity_codes
AttributeError: 'Seq' object has no attribute 'alphabet'
This is where I got stuck now. Leighton, do you have any idea how to go on?
@ Leighton: If there is anything wrong in my description, please do not hesitate to correct it