beagle-dev / beagle-lib

general purpose library for evaluating the likelihood of sequence evolution on trees
MIT License
124 stars 57 forks source link

Error when using GPU acceleration #222

Closed mlzhangli closed 2 days ago

mlzhangli commented 4 days ago

This is my GPU information

$ nvidia-smi 
Tue Jun 25 15:36:28 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:04:00.0 Off |                  N/A |
| 36%   36C    P0    59W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 42%   41C    P0    61W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:08:00.0 Off |                  N/A |
| 45%   42C    P0    62W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:09:00.0 Off |                  N/A |
| 49%   42C    P0    56W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

And beagle_info in beast

$ ./beast -beagle_info

        BEAST v1.10.4 Prerelease #bc6cbd9, 2002-2018
       Bayesian Evolutionary Analysis Sampling Trees
                 Designed and developed by
   Alexei J. Drummond, Andrew Rambaut and Marc A. Suchard

               Department of Computer Science
                   University of Auckland
                  alexei@cs.auckland.ac.nz

             Institute of Evolutionary Biology
                  University of Edinburgh
                     a.rambaut@ed.ac.uk

              David Geffen School of Medicine
           University of California, Los Angeles
                     msuchard@ucla.edu

                Downloads, Help & Resources:
                    http://beast.community

Source code distributed under the GNU Lesser General Public License:
            http://github.com/beast-dev/beast-mcmc

                     BEAST developers:
    Alex Alekseyenko, Guy Baele, Trevor Bedford, Filip Bielejec, Erik Bloomquist, Matthew Hall,
    Joseph Heled, Sebastian Hoehna, Denise Kuehnert, Philippe Lemey, Wai Lok Sibon Li,
    Gerton Lunter, Sidney Markowitz, Vladimir Minin, Michael Defoin Platel,
            Oliver Pybus, Chieh-Hsi Wu, Walter Xie

                         Thanks to:
        Roald Forsberg, Beth Shapiro and Korbinian Strimmer

Using BEAGLE library v4.0.0 (PRE-RELEASE) for accelerated, parallel likelihood evaluation
2009-, BEAGLE Working Group - https://beagle-dev.github.io/
Citation: Ayres et al (2019) Systematic Biology 68: 1052-1061 | doi:10.1093/sysbio/syz020

BEAGLE resources available:
0 : CPU (x86_64)
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_SSE VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_CPU FRAMEWORK_CPU

1 : NVIDIA GeForce GTX 1080 Ti (OpenCL 3.0 CUDA)
    Global memory (MB): 11178
    Clock speed (Ghz): 1.58
    Number of compute units: 28
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH COMPUTATION_ASYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_OPENCL PARALLELOPS_STREAMS PARALLELOPS_GRID

2 : NVIDIA GeForce GTX 1080 Ti (OpenCL 3.0 CUDA)
    Global memory (MB): 11178
    Clock speed (Ghz): 1.58
    Number of compute units: 28
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH COMPUTATION_ASYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_OPENCL PARALLELOPS_STREAMS PARALLELOPS_GRID

3 : NVIDIA GeForce GTX 1080 Ti (OpenCL 3.0 CUDA)
    Global memory (MB): 11178
    Clock speed (Ghz): 1.58
    Number of compute units: 28
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH COMPUTATION_ASYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_OPENCL PARALLELOPS_STREAMS PARALLELOPS_GRID

4 : NVIDIA GeForce GTX 1080 Ti (OpenCL 3.0 CUDA)
    Global memory (MB): 11178
    Clock speed (Ghz): 1.58
    Number of compute units: 28
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH COMPUTATION_ASYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_OPENCL PARALLELOPS_STREAMS PARALLELOPS_GRID

when I run beast it says

$ ~/zl/BEASTv1.10.4/bin/beast -beagle_gpu -beagle_order 4 -overwrite FA1090_gubbins.filtered_polymorphic_sites_3.xml
...
Read alignment: alignment
  Sequences = 1075
      Sites = 30408
   Datatype = nucleotide
Using BEAGLE TreeLikelihood
  Branch rate model used: discretizedBranchRates
  Using BEAGLE resource 4: NVIDIA GeForce GTX 1080 Ti (OpenCL 3.0 CUDA)
    Global memory (MB): 11178
    Clock speed (Ghz): 1.58
    Number of compute units: 28
    with instance flags:  PRECISION_DOUBLE COMPUTATION_SYNCH EIGEN_REAL SCALING_MANUAL SCALERS_RAW VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_OPENCL
  Ignoring ambiguities in tree likelihood.
  With 1 unique site patterns.
  Using rescaling scheme : delayed (delay rescaling until first overflow)
Creating swap operator for parameter default.branchRates.categories (weight=10.0)
Creating swap operator for parameter Continent.branchRates.categories (weight=10.0)
Optimization Schedule: log
Creating CTMC Scale Reference Prior model.
Acting on subtree of size 1075
.....
# BEAST v1.10.4 Prerelease #bc6cbd9
# Generated Tue Jun 25 15:28:20 CST 2024 [seed=1719300475878]
# -beagle_gpu -beagle_order 4 -overwrite FA1090_gubbins.filtered_polymorphic_sites_3.xml
# keywords: discretized_branch_rates
state   Joint           Prior           Likelihood      age(root)       default.uced.mean   Continent.uced.mean Continent.nonZeroRates
0   -7519863.1833   -208979.2897    -7310883.8935   1959.76         1.00000         1.00000         36      -

OpenCL error: CL_MEM_OBJECT_ALLOCATION_FAILURE from file <zl/beagle-lib/libhmsbeagle/GPU/GPUInterfaceOpenCL.cpp>, line 682.

However, the seq length in FA1090_gubbins.filtered_polymorphic_sites_3.xml is 30408, I use a different xml whose seq length is 2232367 and Sequences is 603, and this command works.

xji3 commented 3 days ago

@mlzhangli We recommend using the cuda build of BEAGLE on NVIDIA cards. You are currently using the openCL build and it seems that you didn't build the CUDA version of beagle. Please follow the instructions to build beagle with cmake and let us know (https://github.com/beagle-dev/beagle-lib/wiki/LinuxInstallInstructions).

mlzhangli commented 2 days ago

Oh I forgot to mention the cuda part, it didn't work at first so I changed to openCL, here is the info about it

$ ~/zl/BEASTv1.10.4/bin/beast -beagle_info

        BEAST v1.10.4 Prerelease #bc6cbd9, 2002-2018
       Bayesian Evolutionary Analysis Sampling Trees
                 Designed and developed by
   Alexei J. Drummond, Andrew Rambaut and Marc A. Suchard

               Department of Computer Science
                   University of Auckland
                  alexei@cs.auckland.ac.nz

             Institute of Evolutionary Biology
                  University of Edinburgh
                     a.rambaut@ed.ac.uk

              David Geffen School of Medicine
           University of California, Los Angeles
                     msuchard@ucla.edu

                Downloads, Help & Resources:
                    http://beast.community

Source code distributed under the GNU Lesser General Public License:
            http://github.com/beast-dev/beast-mcmc

                     BEAST developers:
    Alex Alekseyenko, Guy Baele, Trevor Bedford, Filip Bielejec, Erik Bloomquist, Matthew Hall,
    Joseph Heled, Sebastian Hoehna, Denise Kuehnert, Philippe Lemey, Wai Lok Sibon Li,
    Gerton Lunter, Sidney Markowitz, Vladimir Minin, Michael Defoin Platel,
            Oliver Pybus, Chieh-Hsi Wu, Walter Xie

                         Thanks to:
        Roald Forsberg, Beth Shapiro and Korbinian Strimmer

Using BEAGLE library v4.0.0 (PRE-RELEASE) for accelerated, parallel likelihood evaluation
2009-, BEAGLE Working Group - https://beagle-dev.github.io/
Citation: Ayres et al (2019) Systematic Biology 68: 1052-1061 | doi:10.1093/sysbio/syz020

BEAGLE resources available:
0 : CPU (x86_64)
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_SSE VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_CPU FRAMEWORK_CPU

1 : NVIDIA GeForce GTX 1080 Ti
    Global memory (MB): 11179
    Clock speed (Ghz): 1.58
    Number of cores: 3584
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH COMPUTATION_ASYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_CUDA PARALLELOPS_STREAMS PARALLELOPS_GRID

2 : NVIDIA GeForce GTX 1080 Ti
    Global memory (MB): 11179
    Clock speed (Ghz): 1.58
    Number of cores: 3584
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH COMPUTATION_ASYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_CUDA PARALLELOPS_STREAMS PARALLELOPS_GRID

3 : NVIDIA GeForce GTX 1080 Ti
    Global memory (MB): 11179
    Clock speed (Ghz): 1.58
    Number of cores: 3584
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH COMPUTATION_ASYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_CUDA PARALLELOPS_STREAMS PARALLELOPS_GRID

4 : NVIDIA GeForce GTX 1080 Ti
    Global memory (MB): 11179
    Clock speed (Ghz): 1.58
    Number of cores: 3584
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH COMPUTATION_ASYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_GPU FRAMEWORK_CUDA PARALLELOPS_STREAMS PARALLELOPS_GRID

$ ~/zl/BEASTv1.10.4/bin/beast -beagle_gpu -beagle_order 4 -overwrite FA1090_gubbins.filtered_polymorphic_sites_3.xml

Creating the tree model, 'treeModel'
  taxon count = 1075
  tree height = 60.29346452278587

Using discretized relaxed clock model.
  over sampling = 1
  parametric model = exponentialDistributionModel
   rate categories = 1

Using discretized relaxed clock model.
  over sampling = 1
  parametric model = exponentialDistributionModel
   rate categories = 1

Creating state frequencies model 'frequencies': Initial frequencies = {0.25, 0.25, 0.25, 0.25}

Creating site rate model: 
  4 category discrete gamma with initial shape = 0.5
  initial proportion of invariant sites = 0.5

Using BEAGLE DataLikelihood Delegate
CUDA error: "Out of memory" (2) from file <zl/beagle-lib/libhmsbeagle/GPU/GPUInterfaceCUDA.cpp>, line 596.
xji3 commented 2 days ago

@mlzhangli From the cuda error, it seems that your card does not have enough memory to run your analysis.