help with configuration file

[proukakis]

Hello, I wonder if you could help me as I have limited computing skills. I presume the config file needs the various things specified in a tab-delimited text files and the primer parameters are optional, based on primer3. Is this right? if you can provide an example of a config file it would be very helpful. thanks christos c.proukakis@ucl.ac.uk

[proukakis]

I think the config file is OK now, but this happens when I run it. any help will be very welcome

primer3 was installed via brew

thanks christos

primerDesign christos$ python primerdesign.py /usr/local/Cellar/primer3/2.3.7/bin/PrimerDesign.cfg /Users/christos/Dropbox/research/NGS/MPS_C29CER/SV_primers1.txt 2017-04-03 18:37:57.159896 Traceback (most recent call last): File "primerdesign.py", line 477, in main() File "primerdesign.py", line 29, in main file.write(config.get("General", "check_first_chr")) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 618, in get raise NoOptionError(option, section) ConfigParser.NoOptionError: No option 'check_first_chr' in section: 'General'

[zichner]

I will look into this issue. Can you please attach your config as well as the SV file.

[proukakis]

many thanks, the config as created for me by a bioinformatician (will copy her in- she had got the code working a couple of years ago on Linux), and the SV file is attached. I should add that I am working on a mac (OS EL Capitan), not sure if this could be the problem.

PrimerDesign.cfg.zip

SV_primers1.txt

[proukakis]

making some progress... we have a newt config and simplified variant list xls, which seem to work, but now I get this

python primerdesign.py /usr/local/Cellar/primer3/2.3.7/bin/PrimerDesign.cfg /Users/christos/Dropbox/research/NGS/MPS_C29CER/SVprimers1.xls 2017-04-04 14:22:17.843446 Number of variants left: 1 Traceback (most recent call last): File "primerdesign.py", line 477, in main() File "primerdesign.py", line 38, in main primerPairs = determine_primer_pairs(varListTmp, primerTargetSize, primerNum, config) File "primerdesign.py", line 132, in determine_primer_pairs primerTargetSeqFwd, primerTargetSeqRev = get_primer_target_sequence(*variant, primerTargetSize=primerTargetSize, primerOffset=minPrimerOffset, samtools=config.get("Programs","samtools"), genomeFile=config.get("General","reference_genome")) File "primerdesign.py", line 416, in get_primer_target_sequence targetSeq1 = get_DNA_sequence(svStartChr, targetSeq1Start, targetSeq1End, samtools, genomeFile).upper() File "primerdesign.py", line 399, in get_DNA_sequence p1 = subprocess.Popen([samtools, "faidx", genomeFile, chrRegionString], stdout=subprocess.PIPE) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 8] Exec format error

SVprimers1.zip PrimerDesign.cfg.zip

[proukakis]

hi again...

after some jiggling, still havent cracked it

python primerdesign.py /usr/local/Cellar/primer3/2.3.7/bin/PrimerDesign.cfg /Users/christos/Desktop/SV_primers1.csv /Users/christos/Desktop/output.txt 2017-04-04 17:24:56.191786 Traceback (most recent call last): File "primerdesign.py", line 477, in main() File "primerdesign.py", line 32, in main varList = read_variant_list_from_file(sys.argv[2]) File "primerdesign.py", line 70, in read_variant_list_from_file svStartChr, svStartPos, svEndChr, svEndPos, svType, svComment = row ValueError: need more than 1 value to unpack

I read somewhere that this "value" needs the user name, so I did

python primerdesign.py christos /usr/local/Cellar/primer3/2.3.7/bin/PrimerDesign.cfg /Users/christos/Desktop/SV_primers1.csv /Users/christos/Desktop/output.txt 2017-04-04 17:25:48.909324 Traceback (most recent call last): File "primerdesign.py", line 477, in main() File "primerdesign.py", line 24, in main maxSeqDist = config.getint("PrimerDesign", "max_dist_for_sequencing") File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 359, in getint return self._get(section, int, option) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 356, in _get return conv(self.get(section, option)) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'PrimerDesign'

latest .csv test file for variants attached (config as before) SV_primers1.csv.zip

[zichner]

Hi,

thanks for the information and the files. It seems that in your SV file the third column (chromosome of second break point) is empty. However, it should either be a chromosome identifier or . Furthermore, it might be that the file has to be tabulator separated rather than comma separated.

Here is an example file: exampleVar.txt

Can you please try to call the primerDesign tool with that variants file: python primerdesign.py /usr/local/Cellar/primer3/2.3.7/bin/PrimerDesign.cfg exampleVar.txt /Users/christos/Desktop/output.txt

(you don't need to specify your user name anywhere)

[proukakis]

many thanks, had started with .txt files but thought that they were causing trouble

but maybe it just doesn't work on mac...

python primerdesign.py /usr/local/Cellar/primer3/2.3.7/bin/PrimerDesign.cfg /Users/christos/Desktop/exampleVar.txt /Users/christos/Desktop/output.txt 2017-04-05 11:55:50.910575 Number of variants left: 16 Traceback (most recent call last): File "primerdesign.py", line 477, in main() File "primerdesign.py", line 38, in main primerPairs = determine_primer_pairs(varListTmp, primerTargetSize, primerNum, config) File "primerdesign.py", line 132, in determine_primer_pairs primerTargetSeqFwd, primerTargetSeqRev = get_primer_target_sequence(*variant, primerTargetSize=primerTargetSize, primerOffset=minPrimerOffset, samtools=config.get("Programs","samtools"), genomeFile=config.get("General","reference_genome")) File "primerdesign.py", line 416, in get_primer_target_sequence targetSeq1 = get_DNA_sequence(svStartChr, targetSeq1Start, targetSeq1End, samtools, genomeFile).upper() File "primerdesign.py", line 399, in get_DNA_sequence p1 = subprocess.Popen([samtools, "faidx", genomeFile, chrRegionString], stdout=subprocess.PIPE) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 8] Exec format error

can you think of anything else to try? Is there any chance I might ask you, as a massive favour, to run a one-off design of ~10 sets if I send you a .txt file later? (I would be more than happy to acknowledge this if we ever publish something).

Don't worry if you cant help, not a problem, I'll try the old-fashioned way.

best christos

[zichner]

There seems to be a problem executing samtools. However, I am not sure whether this is a problem with Mac OSX or with samtools.

Can you please try to execute /Volumes/SD/bwa0712/samtools to see if this is working.

[proukakis]

hi, so i hope I understand this - looks like you are right

by executing as you said - nothing happens

Christoss-MacBook-Pro:~ christos$ /Volumes/SD/bwa0712/samtools Christoss-MacBook-Pro:~ christos$

but is can be invoked simply from anywhere like this

Christoss-MacBook-Pro:~ christos$ samtools

Program: samtools (Tools for alignments in the SAM format) Version: 1.3 (using htslib 1.3)

Usage: samtools [options]

Commands: -- Indexing dict create a sequence dictionary file faidx index/extract FASTA index index alignment

-- Editing calmd recalculate MD/NM tags and '=' bases fixmate fix mate information reheader replace BAM header rmdup remove PCR duplicates targetcut cut fosmid regions (for fosmid pool only) addreplacerg adds or replaces RG tags

-- File operations collate shuffle and group alignments by name cat concatenate BAMs merge merge sorted alignments mpileup multi-way pileup sort sort alignment file split splits a file by read group quickcheck quickly check if SAM/BAM/CRAM file appears intact fastq converts a BAM to a FASTQ fasta converts a BAM to a FASTA

-- Statistics bedcov read depth per BED region depth compute the depth flagstat simple stats idxstats BAM index stats phase phase heterozygotes stats generate stats (former bamcheck)

-- Viewing flags explain BAM flags tview text alignment viewer view SAM<->BAM<->CRAM conversion depad convert padded BAM to unpadded BAM

[zichner]

Ok. In this case, modify your config file accordingly (replace /Volumes/SD/bwa0712/samtools by samtools) and re-run the pipeline.

[proukakis]

of course, makes total sense, will get the file sorted and try again, many thanks again

[proukakis]

hi, we still can't get it- when I change the config as you say, it seems to start, but then stops

we tried running with sudo, and also sudo chmod +x primerdesign.py

now python primerdesign.py /usr/local/Cellar/primer3/2.3.7/bin/PrimerDesign.cfg /Users/christos/Desktop/exampleVar.txt 2017-04-06 18:42:21.621108 Number of variants left: 16 Unable to open file /usr/local/Cellar/primer3/2.3.7/bin/stack.ds 2017-04-06 18:42:22.073035 Primers to test: 0 Traceback (most recent call last): File "primerdesign.py", line 477, in main() File "primerdesign.py", line 38, in main primerPairs = determine_primer_pairs(varListTmp, primerTargetSize, primerNum, config) File "primerdesign.py", line 136, in determine_primer_pairs uniqPrimers = determine_unique_primers(set(primerSeqList), config.get("Programs", "blast"), config.get("General", "reference_genome"), config.getint("PrimerDesign","min_mismatches"), config.getint("PrimerDesign","min_mismatches_close3p"), config.getint("PrimerDesign","min_dist3p"), config.getint("PrimerDesign","max_mismatches_forbidden"), config.get("Programs","num_cpus"), config.get("PrimerDesign","word_size")) File "primerdesign.py", line 386, in determine_unique_primers blastOutput = run_blast(potUniquePrimers, eValue, blast, genomeFile, numThreads, wordSize, seqidListFile) File "primerdesign.py", line 344, in run_blast p1 = subprocess.Popen([blast, "-task", "blastn", "-db", genomeFile, "-seqidlist", seqidListFile, "-evalue", str(eValue), "-num_threads", numThreads, "-outfmt", "6 std gaps nident", "-dust", "no", "-gapopen", "4", "-gapextend", "2", "-penalty", "-2", "-reward", "2", "-word_size", wordSize, "-max_target_seqs", "2"], stdin=subprocess.PIPE, stdout=subprocess.PIPE) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 13] Permission denied

is this the problem? Unable to open file /usr/local/Cellar/primer3/2.3.7/bin/stack.ds i have no such file in my primer3 (installed via homebrew)

[proukakis]

so, I solved one problem

python primerdesign.py /Users/christos/Desktop/PrimerDesign.cfg /Users/christos/Desktop/exampleVar.txt 2017-04-07 20:46:17.433498 Number of variants left: 16 Unable to open file /Volumes/SD/primer3-2.3.7/src/primer3_configstack.ds 2017-04-07 20:46:17.846387 Primers to test: 0 Traceback (most recent call last): File "primerdesign.py", line 477, in main() File "primerdesign.py", line 38, in main primerPairs = determine_primer_pairs(varListTmp, primerTargetSize, primerNum, config) File "primerdesign.py", line 136, in determine_primer_pairs uniqPrimers = determine_unique_primers(set(primerSeqList), config.get("Programs", "blast"), config.get("General", "reference_genome"), config.getint("PrimerDesign","min_mismatches"), config.getint("PrimerDesign","min_mismatches_close3p"), config.getint("PrimerDesign","min_dist3p"), config.getint("PrimerDesign","max_mismatches_forbidden"), config.get("Programs","num_cpus"), config.get("PrimerDesign","word_size")) File "primerdesign.py", line 386, in determine_unique_primers blastOutput = run_blast(potUniquePrimers, eValue, blast, genomeFile, numThreads, wordSize, seqidListFile) File "primerdesign.py", line 344, in run_blast p1 = subprocess.Popen([blast, "-task", "blastn", "-db", genomeFile, "-seqidlist", seqidListFile, "-evalue", str(eValue), "-num_threads", numThreads, "-outfmt", "6 std gaps nident", "-dust", "no", "-gapopen", "4", "-gapextend", "2", "-penalty", "-2", "-reward", "2", "-word_size", wordSize, "-max_target_seqs", "2"], stdin=subprocess.PIPE, stdout=subprocess.PIPE) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 13] Permission denied

**I wondered if the path / name primer3_configstack.ds was wrong, so I copied the stack.ds from the primer_config directory into the parent src, and renamed it primer3_configstack.ds

this removed the error, but it came up with another file in the same directory, so i moved / renamed that - several files. Might have been simpler to edit the code, but no idea how!**

So now it progresses more, finds primers, but cant go all the way

python primerdesign.py /Users/christos/Desktop/PrimerDesign.cfg /Users/christos/Desktop/exampleVar.txt 2017-04-07 21:28:49.931580 Number of variants left: 16 2017-04-07 21:28:54.249571 Primers to test: 213 Traceback (most recent call last): File "primerdesign.py", line 477, in main() File "primerdesign.py", line 38, in main primerPairs = determine_primer_pairs(varListTmp, primerTargetSize, primerNum, config) File "primerdesign.py", line 136, in determine_primer_pairs uniqPrimers = determine_unique_primers(set(primerSeqList), config.get("Programs", "blast"), config.get("General", "reference_genome"), config.getint("PrimerDesign","min_mismatches"), config.getint("PrimerDesign","min_mismatches_close3p"), config.getint("PrimerDesign","min_dist3p"), config.getint("PrimerDesign","max_mismatches_forbidden"), config.get("Programs","num_cpus"), config.get("PrimerDesign","word_size")) File "primerdesign.py", line 386, in determine_unique_primers blastOutput = run_blast(potUniquePrimers, eValue, blast, genomeFile, numThreads, wordSize, seqidListFile) File "primerdesign.py", line 344, in run_blast p1 = subprocess.Popen([blast, "-task", "blastn", "-db", genomeFile, "-seqidlist", seqidListFile, "-evalue", str(eValue), "-num_threads", numThreads, "-outfmt", "6 std gaps nident", "-dust", "no", "-gapopen", "4", "-gapextend", "2", "-penalty", "-2", "-reward", "2", "-word_size", wordSize, "-max_target_seqs", "2"], stdin=subprocess.PIPE, stdout=subprocess.PIPE) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init errread, errwrite) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 13] Permission denied

SO i dont know why it cant execute these processes- might it be a solvable permissions issue, or a more tricky problem related to the python on my mac, at which point i think i have to give up

[proukakis]

I think i fixed it! It was failing the blast subprocess, so I added shell=True to the script

if seqidListFile == "": p1 = subprocess.Popen([blast, "-task", "blastn", "-db", genomeFile, "-evalue", str(eValue), "-num_threads", numThreads, "-outfmt", "6 std gaps nident", "-dust", "no", "-gapopen", "4", "-gapextend", "2", "-penalty", "-2", "-reward", "2", "-word_size", wordSize, "-max_target_seqs", "500"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True) else: p1 = subprocess.Popen([blast, "-task", "blastn", "-db", genomeFile, "-seqidlist", seqidListFile, "-evalue", str(eValue), "-num_threads", numThreads, "-outfmt", "6 std gaps nident", "-dust", "no", "-gapopen", "4", "-gapextend", "2", "-penalty", "-2", "-reward", "2", "-word_size", wordSize, "-max_target_seqs", "2"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)

[zichner]

Is there still any help needed or can I close the issue?

[proukakis]

Yes please close it thanks

The code was very useful

Sent from my iPhone

On 27 Apr 2017, at 4:57 pm, zichner notifications@github.com<mailto:notifications@github.com> wrote:

Is there still any help needed or can I close the issue?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/zichner/primerDesign/issues/1#issuecomment-297837189, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ARocSCgPFV_1Foqj2DqGtRMbcqpkPmd_ks5r0QEngaJpZM4Mql-0.