search

Electrostatics / apbs

Software for biomolecular electrostatics and solvation calculations
http://www.poissonboltzmann.org/
Other
90 stars 24 forks source link

Very slow boundary element execution #66

Open sobolevnrm opened 4 years ago

sobolevnrm commented 4 years ago

Hi @lwwilson1 --

Could you please help @intendo and me with a performance issue in BEM? The details follow.

From the output below you can see that the bem-pKa test (15) that was switched from MSMS to NanoShaper skin mode is 75 minutes:

Test project /Users/d3c572/git/apbs/build
      Start  1: born_test
1/19 Test  #1: born_test ........................   Passed   12.83 sec
      Start  2: actin-dimer-auto_test
2/19 Test  #2: actin-dimer-auto_test ............   Passed   91.52 sec
      Start  3: actin-dimer-parallel_test
3/19 Test  #3: actin-dimer-parallel_test ........   Passed  290.72 sec
      Start  4: alkanes_test
4/19 Test  #4: alkanes_test .....................   Passed    7.79 sec
      Start  5: FKBP_test
5/19 Test  #5: FKBP_test ........................   Passed   24.73 sec
      Start  6: hca-bind_test
6/19 Test  #6: hca-bind_test ....................   Passed   11.11 sec
      Start  7: ionize_test
7/19 Test  #7: ionize_test ......................   Passed    4.32 sec
      Start  8: ion-pmf_test
8/19 Test  #8: ion-pmf_test .....................   Passed    4.73 sec
      Start  9: ion-protein_test
9/19 Test  #9: ion-protein_test .................   Passed   30.82 sec
      Start 10: pka-lig_test
10/19 Test #10: pka-lig_test .....................   Passed   30.68 sec
      Start 11: point-pmf_test
11/19 Test #11: point-pmf_test ...................   Passed    2.00 sec
      Start 12: solv_test
12/19 Test #12: solv_test ........................   Passed    1.49 sec
      Start 13: geoflow_test
13/19 Test #13: geoflow_test .....................   Passed    4.58 sec
      Start 14: bem_test
14/19 Test #14: bem_test .........................   Passed   29.56 sec
      Start 15: bem-pKa_test
15/19 Test #15: bem-pKa_test .....................   Passed  4472.47 sec
      Start 16: bem-binding-energy_test
16/19 Test #16: bem-binding-energy_test ..........   Passed  519.04 sec
      Start 17: pbam_test
17/19 Test #17: pbam_test ........................   Passed    0.12 sec
      Start 18: pbsam_test
18/19 Test #18: pbsam_test .......................   Passed    5.59 sec
      Start 19: protein-rna_test
19/19 Test #19: protein-rna_test .................   Passed  170.10 sec

100% tests passed, 0 tests failed out of 19

Total Test time (real) = 5714.23 sec

There seems like there are some large calculations in one of the bem-pKa test, 2LZT-ASH66.in. Is this due to the large number of atoms (1961), vertices (931532), or triangles (1862940)? It really slows down when it reaches: Creating tree for 1862940 particles with max 500 per node

(apbs) (base) WE36241:bem-pKa d3c572$ ../../build/bin/apbs 2LZT-ASH66.in

[snip header]

This executable compiled on Aug 28 2020 at 16:52:59

Parsing input file 2LZT-ASH66.in...
rank 0 size 1...
Parsed input file.
Got paths for 1 molecules
Reading PQR-format atom data from test_proteins/2LZT-ASH66.pqr.
  1961 atoms
  Centered at (1.044e+00, 1.408e+01, 2.125e+01)
  Net charge 9.00e+00 e
Preparing to run 1 PBE calculations.
----------------------------------------
CALCULATION #1 (comp_solv): BOUNDARY ELEMENT
  Setting up problem...
  Molecule ID: 1
  Linearized traditional PBE
  Zero boundary conditions
  2 ion species (0.000 M ionic strength):
    0.950 A-radius, 1.000 e-charge, 0.000 M concentration
    1.810 A-radius, -1.000 e-charge, 0.000 M concentration
  Solute dielectric: 20.000
  Solvent dielectric: 78.540
  Using "molecular" surface definition; no smoothing
  Solvent probe radius: 1.400 A
  Temperature:  298.000 K

Treecode order: 3
Max # of nodes: 500
    MAC (theta): 0.800000

      Mesh flag: 2

Setting up the TABI input...
Running NanoShaper...

<<INFO>> Starting NanoShaper 0.7.5
<<INFO>> Loading atoms....
<<INFO>> Read 1961 atoms
<<INFO>> Geometric baricenter ->  0.9775 14.163 21.2475
<<INFO>> Grid is 537
<<INFO>> MAX 27.7775 40.963 48.0475
<<INFO>> MIN -25.8225 -12.637 -5.5525
<<INFO>> Perfil 90 %
<<INFO>> Rmaxdim 48.335
<<INFO>> Allocating memory...ok!
<<INFO>> Initialization completed
<<INFO>> Building skin surface..
<<INFO>> Regular triangulation....ok!
<<INFO>> Computing 3-Delaunay patches...ok!
<<INFO>> Computing 0-Delaunay patches...ok!
<<INFO>> Computing 1-Delaunay patches...ok!
<<INFO>> Computing 2-Delaunay patches...ok!
<<INFO>> Surface build-up time.. 0 [s]
<<INFO>> Shrinking value 0.45
<<INFO>> Number of mixed cells -> 32660
<<INFO>> Number of del_point/vor_cell -> 1345
<<INFO>> Number of del_edge/vor_facet -> 9759
<<INFO>> Number of del_facet/vor_edge -> 17200
<<INFO>> Number of del_cell/vor_point -> 8601
<<INFO>> Use load balancing: 1
<<INFO>> Detected 12 logical cores
<<INFO>> Setting 12 threads
<<INFO>> Inside id value is 5
<<INFO>> Ray-tracing panel 0...ok!
<<INFO>> Ray-tracing panel 1...ok!
<<INFO>> Ray-tracing panel 2...ok!
<<INFO>> Ray-tracing computation time.. 1 [s]
<<INFO>> Approximated 0 rays (0.00000 %)
<<INFO>> Cavity detection time is 0 [s]
<<INFO>> Assembling octrees..ok!
<<INFO>> Surface computation time.. 2 [s]
<<INFO>> Estimated volume 16143.8 [A^3]
<<INFO>> Triangulating Surface...
<<INFO>> Setting 12 threads
<<INFO>> Generating MC vertices...ok!
<<INFO>> MC added 0 non analytical vertices
<<INFO>> Triangles done
<<INFO>> MC time is 1 [s]
<<INFO>> Total, grid conformant, surface area is 6209.03 [A^2]
<<INFO>> Number of vertices 931532 number of triangles 1862940
<<INFO>> Saving in MSMS format, no patch info, no nearest atom..
<<INFO>> Smoothing surface...
<<INFO>> Saving in MSMS format, no patch info, no nearest atom..ok!
<<INFO>> Total Triangulation time 5 [s]
<<INFO>> Cleaning memory...ok!

<<INFO>> If you use NanoShaper please cite these works:
<<CITATION>>   S. Decherchi, W. Rocchia, "A general and Robust Ray-Casting-Based Algorithm for Triangulating Surfaces at the Nanoscale"; PlosOne
<<CITATION>>   link: http://www.plosone.org/article/metrics/info%3Adoi%2F10.1371%2Fjournal.pone.0059744

Finished reading .vert file...
Finished reading .face file...

0 faces have been deleted...
Total suface area = 6198.26093813872466853

Initializing treecode...
Setting up arrays for Taylor expansion...
x-limits of box: -15.472667, 17.427667
y-limits of box: -6.159000, 34.483667
z-limits of box: -2.914333, 45.409667
Creating tree for 1862940 particles with max 500 per node
iteration no.=1, error=7.924293e-02
iteration no.=2, error=1.900933e-02
iteration no.=3, error=5.577112e-03
iteration no.=4, error=1.981469e-03
iteration no.=5, error=6.735350e-04
iteration no.=6, error=2.060250e-04
iteration no.=7, error=6.499776e-05
Cleaning up the tree structure...
Memory has been cleaned for TABIPB!

Solvation energy = -363.465631 kJ/mol
Free energy = -6849.581187 kJ/mol

The max and min potential and normal derivatives on elements area:
potential 28.210307 -9.041446
norm derv 32.618396 -43.390486

The max and min potential and normal derivatives on vertices area:
potential 28.019487 -8.977629
norm derv 32.276886 -42.726383

Returning to APBS caller...

Solvation energy and Coulombic energy in kJ/mol...

  Global net ELEC energy = -3.634656314849E+02
  Global net COULOMBIC energy = -6.486115555583E+03

  Calculating forces...
----------------------------------------
CLEANING UP AND SHUTTING DOWN...
Destroying force arrays.
No energy arrays to destroy.
Destroying multigrid structures.
Destroying finite element structures.
Destroying 1 molecules
Final memory usage:  0.001 MB total, 1.170 MB high water

Thanks for using APBS!

Here is the list of output files from the tests:

(apbs) (base) WE36241:bem-pKa d3c572$ ls -alFst
total 229896
    24 -rw-r--r--   1 d3c572  admin       8501 Aug 28 18:16 2LZT-noASH66.out
   264 -rw-r--r--   1 d3c572  admin      73078 Aug 28 18:16 io.mc
229504 -rw-r--r--   1 d3c572  admin  116793839 Aug 28 18:16 surface_potential.dat
     0 drwxr-xr-x  15 d3c572  admin        480 Aug 28 17:40 ./
    24 -rw-r--r--   1 d3c572  admin       8497 Aug 28 17:40 2LZT-ASH66.out
    24 -rw-r--r--   1 d3c572  admin       8372 Aug 28 17:02 ASH66.out
     8 -rw-r--r--   1 d3c572  admin        863 Aug 25 20:22 ASP66.in
     8 -rw-r--r--   1 d3c572  admin        857 Aug 25 20:22 ASH66.in
     8 -rw-r--r--   1 d3c572  admin        889 Aug 25 20:22 2LZT-noASP66.in
     8 -rw-r--r--   1 d3c572  admin        889 Aug 25 20:22 2LZT-noASH66.in
     8 -rw-r--r--   1 d3c572  admin        887 Aug 25 20:22 2LZT-ASP66.in
     8 -rw-r--r--   1 d3c572  admin        887 Aug 25 20:22 2LZT-ASH66.in
     0 drwxr-xr-x  27 d3c572  admin        864 Aug 21 09:39 ../
     0 drwxr-xr-x   8 d3c572  admin        256 Aug 21 09:39 test_proteins/
     8 -rw-r--r--   1 d3c572  admin       1085 Aug 21 09:39 README.md
lwwilson1 commented 4 years ago

My bad on this--It had slipped my mind the value of the sdens parameter should be updated on the examples. The density parameter in msms has a slightly different meaning than the one in NanoShaper (in msms, number of surface elements scales roughly linearly with sdens; in Nanoshaper, it scales quadratically). I'm fairly certain sdens 2.47 instead of sdens 10 will result in the tests taking about the same time they used to, but I'll test to make sure before I change anything and update the tests.

sobolevnrm commented 4 years ago

Thank you!

lwwilson1 commented 4 years ago

See #68