search

jim-bo / silp2

Scaffolding using Integer Linear Programming
MIT License
2 stars 1 forks source link

AttributeError: 'ParameterGroup' object has no attribute 'polishing' #1

Open TakashiKoyama opened 8 years ago

TakashiKoyama commented 8 years ago

Hi,

First of all, I appreciate you to distribute such a nice tool. I have recently started scaffolding my de novo assembled genome with BAC-end sequences by Sanger using silp2. In the orient step, I have been stuck. Could you tell me how to resolve this issue? Below is my code.

echo '# start silp align'
silp.py align -w scf -a aln -p 35 -c assembly.fasta -q1 BES_trimmed_PE1.fastq -q2 BES_trimmed_PE2.fastq
echo '# start sipl pair'
silp.py pair -w scf -a aln -c assembly.fasta -s1 aln/tmp1.sam -s2 aln/tmp2.sam -k 2 -l assembly.length
echo '# start silp nodes'
silp.py nodes -w scf -c assembly.fasta
echo '# start silp edges'
silp.py edges -w scf -i 120000 -s 20000 -fr -s1 aln/read1.sam -s2 aln/read2.sam
echo '# start silp bundles'
silp.py bundles -w scf -b 1 -i 120000 -s 20000 -r aln/ant -p 90 -bup 1
echo '# start silp decompose'
silp.py decompose -w scf -m 2500
echo '# start silp orient'
silp.py orient -w scf -z 0

And stdout I got.

# start silp align
Settings:
  Output files: "/home/takashi/Documents/BESscaffolding/SILP2/aln/index/index.*.bt2"
  Line rate: 6 (line is 64 bytes)
  Lines per side: 1 (side is 64 bytes)
  Offset rate: 4 (one in 16)
  FTable chars: 10
  Strings: unpacked
  Max bucket size: default
  Max bucket size, sqrt multiplier: default
  Max bucket size, len divisor: 4
  Difference-cover sample period: 1024
  Endianness: little
  Actual local endianness: little
  Sanity checking: disabled
  Assertions: disabled
  Random seed: 0
  Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
  /home/takashi/Documents/BESscaffolding/SILP2/assembly.fasta
Building a SMALL index
Reading reference sizes
  Time reading reference sizes: 00:00:05
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time to join reference sequences: 00:00:03
bmax according to bmaxDivN setting: 155187256
Using parameters --bmax 116390442 --dcv 1024
  Doing ahead-of-time memory usage test
  Passed!  Constructing with these parameters: --bmax 116390442 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
  Building sPrime
  Building sPrimeOrder
  V-Sorting samples
  V-Sorting samples time: 00:00:08
  Allocating rank array
  Ranking v-sort output
  Ranking v-sort output time: 00:00:03
  Invoking Larsson-Sadakane on ranks
  Invoking Larsson-Sadakane on ranks time: 00:00:04
  Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
  (Using difference cover)
  Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
  Binary sorting into buckets
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Binary sorting into buckets time: 00:00:17
Splitting and merging
  Splitting and merging time: 00:00:00
Avg bucket size: 7.75936e+07 (target: 116390441)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 8
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:06
  Sorting block of length 101199570
  (Using difference cover)
  Sorting block time: 00:00:28
Returning block of 101199571
Getting block 2 of 8
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:07
  Sorting block of length 90421546
  (Using difference cover)
  Sorting block time: 00:00:24
Returning block of 90421547
Getting block 3 of 8
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:06
  Sorting block of length 45538686
  (Using difference cover)
  Sorting block time: 00:00:10
Returning block of 45538687
Getting block 4 of 8
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:08
  Sorting block of length 113149112
  (Using difference cover)
  Sorting block time: 00:00:32
Returning block of 113149113
Getting block 5 of 8
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:07
  Sorting block of length 80332818
  (Using difference cover)
  Sorting block time: 00:00:22
Returning block of 80332819
Getting block 6 of 8
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:07
  Sorting block of length 63168716
  (Using difference cover)
  Sorting block time: 00:00:17
Returning block of 63168717
Getting block 7 of 8
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:07
  Sorting block of length 105811513
  (Using difference cover)
  Sorting block time: 00:00:30
Returning block of 105811514
Getting block 8 of 8
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:05
  Sorting block of length 21127059
  (Using difference cover)
  Sorting block time: 00:00:04
Returning block of 21127060
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 183457979
fchr[G]: 310217601
fchr[T]: 437014066
fchr[$]: 620749027
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 226621869 bytes to primary EBWT file: /home/takashi/Documents/BESscaffolding/SILP2/aln/index/index.1.bt2
Wrote 155187264 bytes to secondary EBWT file: /home/takashi/Documents/BESscaffolding/SILP2/aln/index/index.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
    len: 620749027
    bwtLen: 620749028
    sz: 155187257
    bwtSz: 155187257
    lineRate: 6
    offRate: 4
    offMask: 0xfffffff0
    ftabChars: 10
    eftabLen: 20
    eftabSz: 80
    ftabLen: 1048577
    ftabSz: 4194308
    offsLen: 38796815
    offsSz: 155187260
    lineSz: 64
    sideSz: 64
    sideBwtSz: 48
    sideBwtLen: 192
    numSides: 3233068
    numLines: 3233068
    ebwtTotLen: 206916352
    ebwtTotSz: 206916352
    color: 0
    reverse: 0
Total time for call to driver() for forward index: 00:05:25
Reading reference sizes
  Time reading reference sizes: 00:00:04
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time to join reference sequences: 00:00:03
  Time to reverse reference sequence: 00:00:00
bmax according to bmaxDivN setting: 155187256
Using parameters --bmax 116390442 --dcv 1024
  Doing ahead-of-time memory usage test
  Passed!  Constructing with these parameters: --bmax 116390442 --dcv 1024
Constructing suffix-array element generator
Building DifferenceCoverSample
  Building sPrime
  Building sPrimeOrder
  V-Sorting samples
  V-Sorting samples time: 00:00:08
  Allocating rank array
  Ranking v-sort output
  Ranking v-sort output time: 00:00:03
  Invoking Larsson-Sadakane on ranks
  Invoking Larsson-Sadakane on ranks time: 00:00:05
  Sanity-checking and returning
Building samples
Reserving space for 12 sample suffixes
Generating random suffixes
QSorting 12 sample offsets, eliminating duplicates
QSorting sample offsets, eliminating duplicates time: 00:00:00
Multikey QSorting 12 samples
  (Using difference cover)
  Multikey QSorting samples time: 00:00:00
Calculating bucket sizes
  Binary sorting into buckets
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Binary sorting into buckets time: 00:00:17
Splitting and merging
  Splitting and merging time: 00:00:00
Avg bucket size: 8.86784e+07 (target: 116390441)
Converting suffix-array elements to index image
Allocating ftab, absorbFtab
Entering Ebwt loop
Getting block 1 of 7
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:05
  Sorting block of length 58576397
  (Using difference cover)
  Sorting block time: 00:00:15
Returning block of 58576398
Getting block 2 of 7
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:06
  Sorting block of length 63720664
  (Using difference cover)
  Sorting block time: 00:00:16
Returning block of 63720665
Getting block 3 of 7
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:06
  Sorting block of length 109097354
  (Using difference cover)
  Sorting block time: 00:00:31
Returning block of 109097355
Getting block 4 of 7
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:07
  Sorting block of length 88154108
  (Using difference cover)
  Sorting block time: 00:00:25
Returning block of 88154109
Getting block 5 of 7
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:08
  Sorting block of length 92535300
  (Using difference cover)
  Sorting block time: 00:00:25
Returning block of 92535301
Getting block 6 of 7
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:07
  Sorting block of length 102556162
  (Using difference cover)
  Sorting block time: 00:00:30
Returning block of 102556163
Getting block 7 of 7
  Reserving size (116390442) for bucket
  Calculating Z arrays
  Calculating Z arrays time: 00:00:00
  Entering block accumulator loop:
  10%
  20%
  30%
  40%
  50%
  60%
  70%
  80%
  90%
  100%
  Block accumulator loop time: 00:00:06
  Sorting block of length 106109036
  (Using difference cover)
  Sorting block time: 00:00:30
Returning block of 106109037
Exited Ebwt loop
fchr[A]: 0
fchr[C]: 183457979
fchr[G]: 310217601
fchr[T]: 437014066
fchr[$]: 620749027
Exiting Ebwt::buildToDisk()
Returning from initFromVector
Wrote 226621869 bytes to primary EBWT file: /home/takashi/Documents/BESscaffolding/SILP2/aln/index/index.rev.1.bt2
Wrote 155187264 bytes to secondary EBWT file: /home/takashi/Documents/BESscaffolding/SILP2/aln/index/index.rev.2.bt2
Re-opening _in1 and _in2 as input streams
Returning from Ebwt constructor
Headers:
    len: 620749027
    bwtLen: 620749028
    sz: 155187257
    bwtSz: 155187257
    lineRate: 6
    offRate: 4
    offMask: 0xfffffff0
    ftabChars: 10
    eftabLen: 20
    eftabSz: 80
    ftabLen: 1048577
    ftabSz: 4194308
    offsLen: 38796815
    offsSz: 155187260
    lineSz: 64
    sideSz: 64
    sideBwtSz: 48
    sideBwtLen: 192
    numSides: 3233068
    numLines: 3233068
    ebwtTotLen: 206916352
    ebwtTotSz: 206916352
    color: 0
    reverse: 1
Total time for backward call to driver() for mirror index: 00:05:24
2756 reads; of these:
  2756 (100.00%) were unpaired; of these:
    579 (21.01%) aligned 0 times
    2177 (78.99%) aligned exactly 1 time
    0 (0.00%) aligned >1 times
78.99% overall alignment rate
2756 reads; of these:
  2756 (100.00%) were unpaired; of these:
    533 (19.34%) aligned 0 times
    2222 (80.62%) aligned exactly 1 time
    1 (0.04%) aligned >1 times
80.66% overall alignment rate
# start sipl pair
[INFO] creating name array 1
[INFO] reading lines
[INFO] copying data
/home/samba/public/software/silp2/creation/align.py:235: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  elif file_name != None and data == None:
/home/samba/public/software/silp2/creation/align.py:237: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  elif file_name != None and data != None:
[INFO] sorting name array 1
[INFO] creating name array 2
[INFO] reading lines
[INFO] copying data
[INFO] sorting name array 2
[INFO] unique: buffering: 0 2177
[INFO] unique: buffering: 0 2224
# start silp nodes
# start silp edges
# start silp bundles
/usr/local/pyenv/versions/2.7.8/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/usr/local/pyenv/versions/2.7.8/lib/python2.7/site-packages/numpy/core/_methods.py:70: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
[INFO] starting repeat based slimming
/home/samba/public/software/silp2/creation/bundles.py:210: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if name not in repcnts or repcnts[name] == None:
[INFO] contigs with repeat regions in 90.00 threshold: 3043 of 358189
[INFO] bundles effected by repeats: 67 of 120
[INFO] bundles removed by repeats: 61 of 120
[INFO] bundles removed by neg dist: 6
[INFO] total bundles: 59
# start silp decompose
[INFO] loading info
# start silp orient
[DEBUG] solving: con_239537: 499: None
Traceback (most recent call last):
  File "/usr/local/genome/bin/silp.py", line 264, in <module>
    args.func(WorkPaths(args), args)
  File "/home/samba/public/software/silp2/scaffold/part1.py", line 486, in run_orientation
    sol = _decomp_solve(BG, subcc, ILP)
  File "/home/samba/public/software/silp2/scaffold/part1.py", line 231, in _decomp_solve
    ILP.load(subb)
  File "/home/samba/public/software/silp2/optimize/orient.py", line 112, in load
    self._cpx.parameters.mip.polishing.time.set(1800)
AttributeError: 'ParameterGroup' object has no attribute 'polishing'

I really appreciate all of your helps.

Thanks, Takashi

jim-bo commented 8 years ago

Hi @TakashiKoyama thanks for using SILP.

I'm guessing that error is occuring because of change to CPLEX. The polishing parameter is not necessary and you can probably just comment out that line of code.

self._cpx.parameters.mip.polishing.time.set(1800) in the file /home/samba/public/software/silp2/optimize/orient.py at line 112.

If that works I can patch the code in github.

THanks for the issue report, James

TakashiKoyama commented 8 years ago

Hi James,

Thank you for the reply. I could execute silp.py orientwithout error but got another error in silp.py orderas below.

$ silp.py order -w scf
[INFO] solving ILP
[INFO] solving order: 12
[INFO] solving order: 5
[INFO] solving order: 3
[INFO] solving shorties: 358169
[ERROR] exception raised during solve: CPLEX Error  1016: Promotional version. Problem size limits exceeded.

Could you tell me a solution?

Takashi

jim-bo commented 8 years ago

@TakashiKoyama it looks like your version of CPLEX has some limits on problem size. If you are part of an academic institution IBM provides the full version of CPLEX for free:

https://www.ibm.com/developerworks/community/blogs/jfp/entry/cplex_studio_in_ibm_academic_initiative?lang=en