search

saezlab / Covid19

We use our tools to analysis Covid19 RNAseq datasets
4 stars 3 forks source link

cbc times out before completing. #1

Open djinnome opened 4 years ago

djinnome commented 4 years ago

Hi folks,

Thanks for providing this example of how to apply COSMOS to viral pathogenesis data. I am trying to reproduce your pipeline, and I got as far as runCarnival, but the problem now is that I don't have access to CPLEX, so I installed lpSolve and CBC, but neither solver completes. This is the error I get with CBC:

CarnivalResults_NHBEvsCOV2 <-runCARNIVAL(
   #solverPath="/opt/ibm/ILOG/CPLEX_Studio129/cplex/bin/x86-64_linux/cplex",
    solverPath = "/usr/local/bin/cbc",
    netObj=NetworkCarnival_df,
    measObj=as.data.frame(tf_activities_NHBEvsCOV2_stat_top50),
    # inputObj = inputObj,
    # DOTfig=TRUE, 
    dir_name="ResultsCARNIVAL",
    weightObj=t(pathways_NHBEvsCOV2_zscore_inputCarnival),
    # nodeID = 'gene',
    timelimit = 1200,
  solver = 'cbc')
saveRDS(CarnivalResults_NHBEvsCOV2, file = "ResultsCARNIVAL/NHBE_noinput.rds")
OutputCyto(CarnivalResults_NHBEvsCOV2, 
  outputFile="ResultsCARNIVAL/NHBE_noinput")
Your network contains identifiers with '/' symbol and they will
            be replaced with '_'Welcome to the CBC MILP Solver 
Version: 2.10.3 
Build Date: Sep 16 2020 

command line - /usr/local/bin/cbc testFile_1_1.lp -seconds 1200 -ratio 0.0001 solve printi csv solu results_cbc_1_1.txt (default strategy 1)
### CoinLpIO::is_invalid_name(): Name B_DDX19B_GLE1_NDC1_NUP107_NUP133_NUP155_NUP160_NUP188_NUP205_NUP210_NUP214_NUP35_NUP37_NUP43_NUP54_NUP58_NUP62_NUP85_NUP88_NUP93_POM121_RANBP2_SEH1L_1 is too long
### CoinLpIO::are_invalid_names(): Invalid name: vnames[131255]: B_DDX19B_GLE1_NDC1_NUP107_NUP133_NUP155_NUP160_NUP188_NUP205_NUP210_NUP214_NUP35_NUP37_NUP43_NUP54_NUP58_NUP62_NUP85_NUP88_NUP93_POM121_RANBP2_SEH1L_1
### CoinLpIO::is_invalid_name(): Name B_MT_ND1_MT_ND2_MT_ND3_MT_ND4_MT_ND5_MT_ND6_NDUFA1_NDUFA10_NDUFA11_NDUFA12_NDUFA13_NDUFA2_NDUFA3_NDUFA4_NDUFA5_NDUFA6_NDUFA7_NDUFA8_NDUFA9_NDUFAB1_NDUFB1_NDUFB10_NDUFB11_NDUFB2_NDUFB3_NDUFB4_NDUFB5_NDUFB6_NDUFB7_NDUFB8_NDUFB9_NDUFC1_NDUFC2_NDUFS1_NDUFS2_NDUFS3_NDUFS4_NDUFS5_NDUFS6_NDUFS7_NDUFS8_NDUFV1_NDUFV2_NDUFV3_1 is too long
### CoinLpIO::are_invalid_names(): Invalid name: vnames[135814]: B_MT_ND1_MT_ND2_MT_ND3_MT_ND4_MT_ND5_MT_ND6_NDUFA1_NDUFA10_NDUFA11_NDUFA12_NDUFA13_NDUFA2_NDUFA3_NDUFA4_NDUFA5_NDUFA6_NDUFA7_NDUFA8_NDUFA9_NDUFAB1_NDUFB1_NDUFB10_NDUFB11_NDUFB2_NDUFB3_NDUFB4_NDUFB5_NDUFB6_NDUFB7_NDUFB8_NDUFB9_NDUFC1_NDUFC2_NDUFS1_NDUFS2_NDUFS3_NDUFS4_NDUFS5_NDUFS6_NDUFS7_NDUFS8_NDUFV1_NDUFV2_NDUFV3_1
### CoinLpIO::is_invalid_name(): Name dist_DDX19B_GLE1_NDC1_NUP107_NUP133_NUP155_NUP160_NUP188_NUP205_NUP210_NUP214_NUP35_NUP37_NUP43_NUP54_NUP58_NUP62_NUP85_NUP88_NUP93_POM121_RANBP2_SEH1L is too long
### CoinLpIO::are_invalid_names(): Invalid name: vnames[142299]: dist_DDX19B_GLE1_NDC1_NUP107_NUP133_NUP155_NUP160_NUP188_NUP205_NUP210_NUP214_NUP35_NUP37_NUP43_NUP54_NUP58_NUP62_NUP85_NUP88_NUP93_POM121_RANBP2_SEH1L
### CoinLpIO::is_invalid_name(): Name dist_MT_ND1_MT_ND2_MT_ND3_MT_ND4_MT_ND5_MT_ND6_NDUFA1_NDUFA10_NDUFA11_NDUFA12_NDUFA13_NDUFA2_NDUFA3_NDUFA4_NDUFA5_NDUFA6_NDUFA7_NDUFA8_NDUFA9_NDUFAB1_NDUFB1_NDUFB10_NDUFB11_NDUFB2_NDUFB3_NDUFB4_NDUFB5_NDUFB6_NDUFB7_NDUFB8_NDUFB9_NDUFC1_NDUFC2_NDUFS1_NDUFS2_NDUFS3_NDUFS4_NDUFS5_NDUFS6_NDUFS7_NDUFS8_NDUFV1_NDUFV2_NDUFV3 is too long
### CoinLpIO::are_invalid_names(): Invalid name: vnames[142743]: dist_MT_ND1_MT_ND2_MT_ND3_MT_ND4_MT_ND5_MT_ND6_NDUFA1_NDUFA10_NDUFA11_NDUFA12_NDUFA13_NDUFA2_NDUFA3_NDUFA4_NDUFA5_NDUFA6_NDUFA7_NDUFA8_NDUFA9_NDUFAB1_NDUFB1_NDUFB10_NDUFB11_NDUFB2_NDUFB3_NDUFB4_NDUFB5_NDUFB6_NDUFB7_NDUFB8_NDUFB9_NDUFC1_NDUFC2_NDUFS1_NDUFS2_NDUFS3_NDUFS4_NDUFS5_NDUFS6_NDUFS7_NDUFS8_NDUFV1_NDUFV2_NDUFV3
### CoinLpIO::readLp(): Invalid column names
Now using default column names.
seconds was changed from 1e+100 to 1200
ratioGap was changed from 0 to 0.0001
Continuous objective value is 11.1338 - 1.82 seconds
Cgl0002I 2 variables fixed
Cgl0003I 0 fixed, 0 tightened bounds, 37600 strengthened rows, 766652 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 29283 strengthened rows, 688616 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 69052 strengthened rows, 726778 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 32934 strengthened rows, 560880 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 3617 strengthened rows, 33174 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 13309 strengthened rows, 34602 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 21226 strengthened rows, 33718 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 8344 strengthened rows, 33312 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 443 strengthened rows, 32058 substitutions
Cgl0004I processed model has 212427 rows, 46996 columns (42027 integer (38504 of which binary)) and 784253 elements
Cbc0038I Initial state - 339 integers unsatisfied sum - 151.252
Cbc0038I Pass   1: (149.10 seconds) suminf.   38.49505 (99) obj. 30.7945 iterations 18427
Cbc0038I Pass   2: (150.48 seconds) suminf.   33.47525 (96) obj. 37.1854 iterations 363
Cbc0038I Pass   3: (150.72 seconds) suminf.   31.47525 (92) obj. 39.1436 iterations 70
Cbc0038I Pass   4: (156.23 seconds) suminf.   21.16984 (105) obj. 38.4532 iterations 1826
Cbc0038I Pass   5: (234.06 seconds) suminf.   21.05068 (380) obj. 39.9821 iterations 22707
Cbc0038I Pass   6: (291.79 seconds) suminf.   21.04961 (122) obj. 39.9821 iterations 13444
Cbc0038I Pass   7: (305.00 seconds) suminf.  127.56625 (367) obj. 51.3513 iterations 4783
Cbc0038I Pass   8: (340.89 seconds) suminf.   31.62756 (768) obj. 49.5476 iterations 9881
Cbc0038I Pass   9: (360.56 seconds) suminf.   31.78319 (145) obj. 49.7815 iterations 5368
Cbc0038I Pass  10: (363.90 seconds) suminf.   28.61054 (167) obj. 50.0033 iterations 881
Cbc0038I Pass  11: (364.25 seconds) suminf.   28.61054 (167) obj. 50.0033 iterations 107
Cbc0038I Pass  12: (364.67 seconds) suminf.   28.61054 (167) obj. 50.0033 iterations 112
Cbc0038I Pass  13: (365.07 seconds) suminf.   28.61054 (167) obj. 50.0033 iterations 123
Cbc0038I Pass  14: (377.34 seconds) suminf.  121.45324 (359) obj. 61.0389 iterations 4175
Cbc0038I Pass  15: (394.11 seconds) suminf.   30.51679 (507) obj. 59.6484 iterations 5018
Cbc0038I Pass  16: (422.36 seconds) suminf.   30.51517 (208) obj. 59.6483 iterations 7082
Cbc0038I Pass  17: (422.86 seconds) suminf.   29.51517 (206) obj. 59.6483 iterations 117
Cbc0038I Pass  18: (423.08 seconds) suminf.   29.51517 (206) obj. 59.6483 iterations 41
Cbc0038I Pass  19: (436.35 seconds) suminf.  125.91640 (406) obj. 71.439 iterations 4529
Cbc0038I Pass  20: (450.05 seconds) suminf.   38.81379 (215) obj. 70.2227 iterations 4264
Cbc0038I Pass  21: (452.10 seconds) suminf.   35.36534 (211) obj. 69.8621 iterations 470
Cbc0038I Pass  22: (452.64 seconds) suminf.   35.36534 (211) obj. 69.8621 iterations 135
Cbc0038I Pass  23: (462.05 seconds) suminf.  122.33543 (397) obj. 82.8115 iterations 2972
Cbc0038I Pass  24: (478.78 seconds) suminf.   43.97370 (246) obj. 81.7172 iterations 5027
Cbc0038I Pass  25: (478.99 seconds) suminf.   41.47370 (241) obj. 81.6172 iterations 33
Cbc0038I Pass  26: (480.20 seconds) suminf.   40.00380 (239) obj. 81.7692 iterations 415
Cbc0038I Pass  27: (480.49 seconds) suminf.   40.00380 (239) obj. 81.7692 iterations 133
Cbc0038I Pass  28: (480.88 seconds) suminf.   40.00380 (239) obj. 81.7692 iterations 104
Cbc0038I Pass  29: (481.41 seconds) suminf.   40.00380 (239) obj. 81.7692 iterations 157
Cbc0038I Pass  30: (490.42 seconds) suminf.  139.14359 (449) obj. 93.3807 iterations 3592
Cbc0038I No solution found this major pass
Cbc0038I Before mini branch and bound, 36279 integers at bound fixed and 2875 continuous of which 2574 were internal integer and 0 internal continuous
Cbc0038I Mini branch and bound did not improve solution (490.70 seconds)
Cbc0038I After 490.70 seconds - Feasibility pump exiting - took 394.85 seconds
Cbc0031I 920 added rows had average density of 2.6380435
Cbc0013I At root node, 920 cuts changed objective from 11.133753 to 20.354545 in 10 passes
Cbc0014I Cut generator 0 (Probing) - 5486 row cuts average 2.1 elements, 0 column cuts (0 active)  in 17.073 seconds - new frequency is 1
Cbc0014I Cut generator 1 (Gomory) - 429 row cuts average 14.7 elements, 0 column cuts (0 active)  in 2.701 seconds - new frequency is -100
Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 2.203 seconds - new frequency is -100
Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 0.015 seconds - new frequency is -100
Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 6840 row cuts average 2.7 elements, 0 column cuts (0 active)  in 0.812 seconds - new frequency is 1
Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 1.720 seconds - new frequency is -100
Cbc0014I Cut generator 6 (TwoMirCuts) - 6292 row cuts average 3.1 elements, 0 column cuts (0 active)  in 5.642 seconds - new frequency is 1
Cbc0014I Cut generator 7 (ZeroHalf) - 64 row cuts average 2.0 elements, 0 column cuts (0 active)  in 93.457 seconds - new frequency is -100
Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 20.354545 (970.65 seconds)
Cbc0020I Exiting on maximum time
Cbc0005I Partial search - best objective 1e+50 (best possible 20.354545), took 90729 iterations and 1 nodes (1109.72 seconds)
Cbc0032I Strong branching done 22 times (2101 iterations), fathomed 0 nodes and fixed 0 variables
Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost
Cuts at root node changed objective from 11.1338 to 20.3545
Probing was tried 11 times and created 5843 cuts of which 0 were active after adding rounds of cuts (17.139 seconds)
Gomory was tried 10 times and created 429 cuts of which 0 were active after adding rounds of cuts (2.701 seconds)
Knapsack was tried 10 times and created 0 cuts of which 0 were active after adding rounds of cuts (2.203 seconds)
Clique was tried 10 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.015 seconds)
MixedIntegerRounding2 was tried 11 times and created 9156 cuts of which 0 were active after adding rounds of cuts (0.897 seconds)
FlowCover was tried 10 times and created 0 cuts of which 0 were active after adding rounds of cuts (1.720 seconds)
TwoMirCuts was tried 11 times and created 6436 cuts of which 0 were active after adding rounds of cuts (6.338 seconds)
ZeroHalf was tried 10 times and created 64 cuts of which 0 were active after adding rounds of cuts (93.457 seconds)
ImplicationCuts was tried 1 times and created 426 cuts of which 0 were active after adding rounds of cuts (0.005 seconds)

Result - Stopped on time limit

No feasible solution found
Lower bound:                    20.355
Enumerated nodes:               1
Total iterations:               90729
Time (CPU seconds):             1109.73
Time (Wallclock seconds):       1126.38

Option for printingOptions changed from normal to csv
Total time (CPU seconds):       1110.94   (Wallclock seconds):       1127.83
alberto-valdeolivas commented 4 years ago

Hi Jeremie,

Thanks for your interest in our tools and repository. Just to clarify a bit, this is CARNIVAL. COSMOS is the Causal integration of multi-omics data with prior knowledge. In this repo, we are only using transcriptomics data and a protein-protein interaction network to run CARNIVAL. In the COSMOS publication, transcriptomics, metabolomics and phosphoproteomics are integrated in a more complex causal network also containing, among others, metabolic reactions. Then CARNIVAL is applied on these more complex networks in COSMOS.

I strongly recommend you to use cplex to run CARNIVAL. We included the possibility of using the cbc and lpSolve solvers because they are totally free. However, we show that the performance of cplex, mainly in large networks, is significantly better. Cplex has a free version for academia. In case you want to keep working with cbc, you must increase the timelimit parameter. The error you are getting means that cbc was not able to find a "feasible" solution in the specified timelimit.

Best regards, Alberto.

djinnome commented 3 years ago

Hi @alberto-valdeolivas Thanks for the clarification! I would love to use cplex, but IBM does not consider a national lab to be an academic institution. I then tried CBC with longer time limits, but to no avail.

Fortunately, the folks at Gurobi granted me a license as long as I use it for academic work. I have successfully run gurobi on the LP file that CARNIVAL generates (although the 255 character limit in gurobi required me to rename a couple of the variables), and then I went ahead and extended CARNIVAL with gurobi functionality. Now if solver="gurobi" then it will create a gurobi Command file with the appropriate gurobi parameters, and it will generate a result file similar to CBC output that is parsed by exportResultGurobi.R. I went ahead and created a pull request, but before you accept, I wanted to discuss the possibility of a general fix for the 255 character limit on the variables that gurobi chokes on.

alberto-valdeolivas commented 3 years ago

Hi again Jeremy,

Many thanks! I will discuss with the person in charge of the CARNIVAL package maintenance in order to add this new solver.

Best, Alberto.