search

AmpliconSuite / AmpliconSuite-pipeline

A quickstart tool for AmpliconArchitect. Performs all preliminary steps (alignment, CNV calling, seed interval detection) required prior to running AmpliconArchitect. Previously called PrepareAA.
Other
48 stars 25 forks source link

An error with MOSEK call in AmpliconArchitect.py #59

Open carrie1s opened 1 month ago

carrie1s commented 1 month ago

version: ampliconsuite 1.3.2 mosek 10.1.31

command: AmpliconSuite-pipeline.py -s XXX -t 12 --bam ../bam/XX.bam --run_AA --run_AC --ref hg38

Report error when running the step: AmpliconArchitect.py --ref GRCh38 --downsample 10 --bed XX_AA_CNV_SEEDS.bed --bam ../bam/XX.bam --runmode FULL --extendmode EXPLORE --out XX

ERROR LOG: INFO:MOSEK:Beginning MOSEK call DEBUG:MOSEK:Problem DEBUG:MOSEK: Name : DEBUG:MOSEK: Objective sense : minimize DEBUG:MOSEK: Type : CONIC (conic optimization problem) DEBUG:MOSEK: Constraints : 674 DEBUG:MOSEK: Affine conic cons. : 858 (2574 rows) DEBUG:MOSEK: Disjunctive cons. : 0 DEBUG:MOSEK: Cones : 0 DEBUG:MOSEK: Scalar variables : 1716 DEBUG:MOSEK: Matrix variables : 0 DEBUG:MOSEK: Integer variables : 0 DEBUG:MOSEK: DEBUG:MOSEK:Optimizer started. DEBUG:MOSEK:Presolve started. DEBUG:MOSEK:Linear dependency checker started. DEBUG:MOSEK:Linear dependency checker terminated. DEBUG:MOSEK:Eliminator started. DEBUG:MOSEK:Freed constraints in eliminator : 2 DEBUG:MOSEK:Eliminator terminated. DEBUG:MOSEK:Eliminator - tries : 1 time : 0.00 DEBUG:MOSEK:Lin. dep. - tries : 1 time : 0.00 DEBUG:MOSEK:Lin. dep. - primal attempts : 1 successes : 1 DEBUG:MOSEK:Lin. dep. - dual attempts : 0 successes : 0 DEBUG:MOSEK:Lin. dep. - primal deps. : 0 dual deps. : 0 DEBUG:MOSEK:Presolve terminated. Time: 0.01 DEBUG:MOSEK:Optimizer - threads : 24 DEBUG:MOSEK:Optimizer - solved problem : the primal DEBUG:MOSEK:Optimizer - Constraints : 672 DEBUG:MOSEK:Optimizer - Cones : 856 DEBUG:MOSEK:Optimizer - Scalar variables : 2568 conic : 2568 DEBUG:MOSEK:Optimizer - Semi-definite variables: 0 scalarized : 0 DEBUG:MOSEK:Factor - setup time : 0.00 DEBUG:MOSEK:Factor - dense det. time : 0.00 GP order time : 0.00 DEBUG:MOSEK:Factor - nonzeros before factor : 1769 after factor : 2188 DEBUG:MOSEK:Factor - dense dim. : 0 flops : 1.91e+04 DEBUG:MOSEK:ITE PFEAS DFEAS GFEAS PRSTATUS POBJ DOBJ MU TIME DEBUG:MOSEK:0 2.6e+00 9.2e+05 1.5e+07 0.00e+00 1.481523133e+07 -6.891673134e+02 1.0e+00 0.01 DEBUG:MOSEK:1 8.6e-01 3.1e+05 8.6e+06 -1.00e+00 1.481247189e+07 -3.374965357e+03 3.3e-01 0.02 DEBUG:MOSEK:2 3.7e-01 1.3e+05 5.6e+06 -1.00e+00 1.480721678e+07 -8.420865348e+03 1.4e-01 0.02 DEBUG:MOSEK:3 1.0e-01 3.6e+04 2.9e+06 -1.00e+00 1.478185341e+07 -3.265364634e+04 3.9e-02 0.03 DEBUG:MOSEK:4 3.0e-02 1.1e+04 1.6e+06 -1.00e+00 1.469091742e+07 -1.189922701e+05 1.2e-02 0.03 DEBUG:MOSEK:5 1.2e-02 4.2e+03 1.0e+06 -9.99e-01 1.445276664e+07 -3.425507493e+05 4.6e-03 0.03 DEBUG:MOSEK:6 5.2e-03 1.8e+03 6.6e+05 -9.96e-01 1.378685075e+07 -9.575357576e+05 2.0e-03 0.04 DEBUG:MOSEK:7 1.6e-03 5.7e+02 3.6e+05 -9.87e-01 1.072088581e+07 -3.763634589e+06 6.2e-04 0.04 DEBUG:MOSEK:8 2.1e-04 7.5e+01 1.2e+05 -9.45e-01 -1.509111365e+07 -2.719934969e+07 8.2e-05 0.04 DEBUG:MOSEK:9 6.3e-05 2.3e+01 4.2e+04 -5.57e-01 -4.934651656e+07 -5.694897247e+07 2.5e-05 0.04 DEBUG:MOSEK:10 4.2e-05 1.5e+01 2.4e+04 2.48e-01 -5.583431514e+07 -6.141871470e+07 1.6e-05 0.05 DEBUG:MOSEK:11 1.6e-05 5.7e+00 6.0e+03 4.71e-01 -6.713890867e+07 -6.960237043e+07 6.2e-06 0.05 DEBUG:MOSEK:12 6.1e-06 2.2e+00 1.4e+03 7.90e-01 -7.234123926e+07 -7.333062025e+07 2.4e-06 0.05 DEBUG:MOSEK:13 1.4e-06 4.9e-01 1.6e+02 9.22e-01 -7.491807945e+07 -7.514736454e+07 5.3e-07 0.06 DEBUG:MOSEK:14 5.2e-07 1.8e-01 3.7e+01 9.84e-01 -7.536359378e+07 -7.545053101e+07 2.0e-07 0.06 DEBUG:MOSEK:15 1.5e-07 5.3e-02 5.8e+00 9.94e-01 -7.555050335e+07 -7.557572224e+07 5.8e-08 0.06 DEBUG:MOSEK:16 2.0e-08 7.1e-03 2.9e-01 9.98e-01 -7.561503075e+07 -7.561838064e+07 7.7e-09 0.07 DEBUG:MOSEK:17 2.6e-09 9.3e-04 1.4e-02 1.00e+00 -7.562296054e+07 -7.562339930e+07 1.0e-09 0.07 DEBUG:MOSEK:18 1.9e-10 6.9e-05 2.8e-04 1.00e+00 -7.562406459e+07 -7.562409717e+07 7.5e-11 0.07 DEBUG:MOSEK:19 1.6e-10 5.3e-05 1.9e-04 9.98e-01 -7.562408461e+07 -7.562410986e+07 5.8e-11 0.08 DEBUG:MOSEK:20 1.6e-10 5.3e-05 1.9e-04 9.98e-01 -7.562408461e+07 -7.562410986e+07 5.8e-11 0.09 DEBUG:MOSEK:21 1.6e-10 5.3e-05 1.9e-04 9.98e-01 -7.562408461e+07 -7.562410986e+07 5.8e-11 0.10 DEBUG:MOSEK:Optimizer terminated. Time: 0.12 DEBUG:MOSEK: DEBUG:MOSEK: DEBUG:MOSEK:Interior-point solution summary DEBUG:MOSEK: Problem status : UNKNOWN DEBUG:MOSEK: Solution status : UNKNOWN DEBUG:MOSEK: Primal. obj: -7.5624084608e+07 nrm: 1e+03 Viol. con: 5e-06 var: 0e+00 acc: 0e+00 DEBUG:MOSEK: Dual. obj: -7.5624109859e+07 nrm: 4e+06 Viol. con: 0e+00 var: 2e+00 acc: 0e+00 ERROR:MOSEK:Error when using MOSEK: Failed to solve to optimality. Solution status solsta.unknown INFO:MOSEK:Saved MOSEK inputs to mosekinput-1.json. Submit that file to support to reproduce the issue.

I have run 6 samples, only this sample report the error. The other 5 samples did not show any issues. The amplicon1 information generated well but amplicon2 encountered this issue. I cannot solve this problem. Is my mosek version wrong or any other specific settings? Please help me.

jluebeck commented 1 month ago

Thank you, would you kindly be able to provide the mosekinput-1.json file? You can email it to jluebeck [at] ucsd.edu. I can work with the MOSEK team to understand the issue.

Thank you, Jens

carrie1s commented 1 month ago

Thank you, would you kindly be able to provide the mosekinput-1.json file? You can email it to jluebeck [at] ucsd.edu. I can work with the MOSEK team to understand the issue.

Thank you, Jens

OK. I uploaded both the mosekinput-1.json, log file and summary file here. And I have also emailed to jluebeck [at] ucsd.edu. 320dm_shTOP2B_summary.txt 320dm_shTOP2B.log mosekinput-1.json

Thank you very much, Carrie

jluebeck commented 1 month ago

Thank you, Carrie. My last question is if you can share the [sample]_amplicon2_graph.txt file with me as well?

Thank you! Jens

jluebeck commented 1 month ago

Hi Carrie,

I have identified the source of the issue (it is a rare convergence issue with Mosek's optimization) and we can provide a fix for this in the next update to AA. Please give us a few weeks to provide a new release with some other features coming down the pipe.

Thanks, Jens

carrie1s commented 1 month ago

Thank you, Carrie. My last question is if you can share the [sample]_amplicon2_graph.txt file with me as well?

Thank you! Jens

Sure, Jens. I apologize for not checking my emails in a timely manner. Here are the files generated in output directory "XX_AA_results" . I have packed them up. 320dm_shTOP2B_AA_results.zip

It's great that you found the problem. I hope to see the updated version soon. Many thanks to the maintenance team for your efforts.

Thanks, Carrie

carrie1s commented 1 month ago

Also the "XX_cnvkit_output" Here: 320dm_shTOP2B_cnvkit_output.zip

jluebeck commented 1 month ago

Thank you - this is very helpful. In the meantime before the fix, I recommend re-running AmpliconSuite-pipeline.py with the following parameter: --AA_insert_sdevs 9

Thanks, Jens

jluebeck commented 1 week ago

A bypass for the Mosek issue is included in AA v 1.4.r2 (AmpliconSuite v1.3.3), and an error will be logged but it will not crash. However when this occurs it is best for the user to re-run with --AA_insert_sdevs 9.