SBEL GPU DEM-Engine

A dual-GPU DEM solver with complex grain geometry support

Quick links

Description

DEM-Engine, nicknamed DEME, does Discrete Element Method simulations:

• Using up to two GPUs at the same time (works great on consumer and data center GPUs).
• With the particles having complex shapes represented by clumped spheres.
• With support for customizable contact force models (want to add a non-standard cohesive force, or an electrostatic repulsive force? You got this).
• With an emphasis on computational efficiency. As a rule of thumb, using 3-sphere clump elements, simulating 1 million elements for 1 million time steps takes around 1 hour on two RTX 3080s.
• Supporting a wide range of problems with flexible API designs. Deformable meshes and grain breakage can be simulated by leveraging the explicit controls given to the user.
• With support for co-simulation with other C/C++ packages, such as Chrono.

Currently DEME is a C++ package with an API design similar to Chrono's, and should be easy to learn for existing Chrono users.

You can find the movies of some of DEME's demos here.

You are welcome to discuss DEME on Project Chrono's forum.

PyDEME

DEME is now available as a Python package, pyDEME. It is quick to install and pick up its usage by trying this Python version. If you want to maximize the performance and use the cutting-edge features, you can instead install the C++ version of DEME from source.

To install pyDEME, use a Linux machine, install CUDA if you do not already have it. Useful installation instructions may be found here.

Some additional troubleshooting tips for getting CUDA ready:

• I recommend getting the newest CUDA. But note that the recent releases CUDA 12.1, 12.2 and 12.3 appear to cause troubles with jitify and you should not use them with DEME.
• On WSL this code may be buildable (and this is the guide for installing CUDA on WSL), but may not run. This is due to the many limitations on unified memory and pinned memory support on WSL. We are looking into it and for now, a native Linux machine or cluster is recommended.

Once CUDA is ready, you can pip install pyDEME. In your conda environement, do

conda create -n pyDEME python=3.11
conda activate pyDEME
conda install cmake
pip3 install DEME

You can also install pyDEME via conda install: (Please don't use conda install for now, it is not yet behaving correctly)

conda create -n pyDEME python=3.11

conda activate pyDEME

conda install -c projectchrono pydeme

pyDEME can be replaced with an environement name of your choice. Other Python versions other than 3.11 should work as well.

Then Python scripts can be executed in this environment. To understand the content of each Python demo, refer to the explanations of the C++ demos with the same names in Examples section.

Compilation

You can also build C++ DEME from source. It allows for potentially more performance and more tailoring.

On a Linux machine, install CUDA. The newest release is recommended.

Once CUDA is ready, clone this project and then:

git submodule init
git submodule update

This will pull the submodule NVIDIA/jitify so that we can do runtime compilation.

Then, one typical choice is to make a build directory in it. Then in the build directory, use cmake to configure the compilation. An example:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..

You may want to use this information if you need to update cmake to the newest.

We suggest that you install a cmake GUI such as ccmake, and ninja_build generator, to better help you configure the project. In this case, the example above can be done like this alternatively:

mkdir build
cd build
ccmake -G Ninja ..

You generally do not have to change the build options in the GUI, but preferably you can change CMAKE_BUILD_TYPE to Release, and if you need to install this package as a library you can specify a CMAKE_INSTALL_PREFIX.

Some additional troubleshooting tips for generating the project:

• If some dependencies such as CUB are not found, then you probably need to manually set $PATH and $LD_LIBRARY_PATH. An example is given below for a specific version of CUDA, note it may be different on your machine or cluster. You should also inspect if nvidia-smi and nvcc --version give correct returns.

  export CPATH=/usr/local/cuda-12.5/targets/x86_64-linux/include${CPATH:+:${CPATH}}
  export PATH=/usr/local/cuda-12.5/bin${PATH:+:${PATH}}
  export PATH=/usr/local/cuda-12.5/lib64/cmake${PATH:+:${PATH}}
  export LD_LIBRARY_PATH=/usr/local/cuda-12.5/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
  export CUDA_HOME=/usr/local/cuda-12.5

Finally, build the project.

ninja

Some additional troubleshooting tips for building the project:

• If you see some grammatical errors during compilation, such as filesystem not being a member of std or arguments not expanded with ..., then manually setting the flag TargetCXXStandard to STD_CXX17 might help.
• If CUB is not found, then you may manually set it in the ccmake GUI as /usr/local/cuda/lib64/cmake/cub. It may be a slightly different path on your machine or cluster.
• If libcudacxx is not found, then you may manually set it in the ccmake GUI as /usr/local/cuda-12.0/targets/x86_64-linux/lib/cmake/libcudacxx. Depending on your CUDA version it may be a slightly different path on your machine or cluster. You may also try to find these packages using find.

Examples

After the build process is done, you can start trying out the demos.

• ./bin/DEMdemo_SingleSphereCollide can be used to test a correct installation. If it runs outputting a lot of texts (those are debug messages; the user do not have to worry about the content) and stops without an error in the end, the installation is probably good.
• An all-rounder beginner example featuring a bladed mixer interacting with complex shaped particles: ./bin/DEMdemo_Mixer.
• A place to learn how prescribed motions work in this package, using either analytical boundaries or particle-represented boundaries: ./bin/DEMdemo_Centrifuge and ./bin/DEMdemo_Sieve.
• A few representative engineering experiments reproduced in DEM simulations, which potentially serve as starting points for your own DEM scripts: /bin/DEMdemo_BallDrop, ./bin/DEMdemo_ConePenetration, /bin/DEMdemo_RotatingDrum, ./bin/DEMdemo_Repose, ./bin/DEMdemo_Plow.
• ./bin/DEMdemo_WheelDP shows how to load a checkpointed configuration file to instantly generate a settled granular terrain, then run a drawbar-pull test on it. This demo therefore requires you to first finish the two GRCPrep demos to obtain the terrain checkpoint file. The granular terrain in these demos features DEM particles with a variety of sizes and shapes.
• ./bin/DEMdemo_WheelDPSimplified is a simplified version of the previous drawbar-pull test which has no prerequisite. The terrain is simpilified to be made of only one type of irregular-shaped particles. It serves as a quick starting point for people who want to create similar experiments.
• ./bin/DEMdemo_Indentation is a more advanced examples showing the usage of the custom additional properties (called wildcards) that you can associate with the simulation entities, and use them in the force model and/or change them in simulation then deposit them into the output files. Wildcards have more use cases especially if coupled together with a custom force model, as shown in some of the follwing demos.
• ./bin/DEMdemo_Electrostatic simulates a pile of complex-shaped and charged granular particles interacting with a mesh that is also charged. Its purpose is to show how to define a non-local force (electrostatic force) which takes effect even when the bodies are not in contact, using a custom force model file. This idea can be extended to modeling a custom cohesion force etc.
• ./bin/DEMdemo_FlexibleMesh simulates a deforming mesh interacting with DEM particles. The intention is to show that the user can extract the force pairs acting on a mesh, then update the mesh with deformation information. DEME does not care how this deformation is calculated. Presumably the user can feed the forces to their own solid mechanics solver to get the deformation. DEME does not come with a built-in linear solver so for simplicity, in this demo the mesh deformation is instead prescribed.
• ./bin/DEMdemo_GameOfLife is a fun game-of-life simulator built with the package, showing the flexibility in terms of how you can use this tool.
• ./bin/DEMdemo_Fracture_Box simulates a concrete bar breaking using a custom force model that creates inter-particle bonds and lets them break under certain conditions. This is a showcase for advanced usage of custom models that involves per-contact wildcard variables.
• It is a good idea to read the comment lines at the top of the demo files to understand what they each does.

The documentations for DEME are hosted on Chrono website (work in progress).

Some additional troubleshooting tips for running the demos:

• If errors similar to CUDA_ERROR_UNSUPPORTED_PTX_VERSION are encountered while you run the demos, or (rarely) the simulations proceed without detecting any contacts, then please make sure the CUDA installation is the same version as when the code is compiled.
• Another cause for the simulations proceeding without detecting any contacts, could be the force kernel silently failed. This could also lead to a too-many-geometries-in-bin crash. The cause is usually that the force kernel was launched with too many threads per block, therefore not enough registers can be leveraged. This can be avoided by calling SetForceCalcThreadsPerBlock prior to the start of simulation with the argument being 256 or even smaller choices like 128.
• Used your own force model but got runtime compilation error like expression must have pointer-to-object type but it has type "float", or unknown variable "delta_time"? Check out what we did in demo DEMdemo_Electrostatic. You may need to manually specify what material properties are pairwise and what contact wildcards you have using SetMustPairwiseMatProp and SetPerContactWildcards.
• Just running provided demos or a script that used to work, but the jitification of the force model failed or the simulation fails at the first kernel call (probably in DEMCubContactDetection.cu)? Then did you pull a new version and just re-built in-place? A new update may modify the force model, and the force model in DEME are given as text files so might not be automatically copied over when the project is re-built. I am sorry for the trouble it might cause, but you can do a clean re-build from an empty directory and it should fix the problem. Do not forget to first commit your own branches' changes and relocate the data you generated in the build directory. Another solution is to copy everything in src/DEM to the DEM directory in the build directory, then everything in src/kernel to the kernel directory in the build directory, then try again.

Limitations

DEME is designed to simulate the interaction among clump-represented particles, the interaction between particles and mesh-represented bodies, as well as the interaction between particles and analytical boundaries. DEME does not resolve mesh–mesh or mesh–analytical contacts.

• It is able to handle mesh-represented bodies with relatively simple physics, for example a meshed plow moving through granular materials with a prescribed velocity, or several meshed projectiles flying and hitting the granular ground.
• However, if the bodies' physics are complex multibody problems, say it is a vehicle that has joint-connected parts and a motor with certain driving policies, or the meshed bodies have collisions among themselves that needs to be simulated, then DEME alone does not have the infrastructure to handle them. But you can install DEME as a library and do coupled simulations with other tools such as Chrono, where DEME is exclusively tasked with handling the granular materials and the influence they exert on the outside world (with high efficiency, of course). See the following section.

Install as library

Set the CMAKE_INSTALL_PREFIX flag in cmake GUI to your desired installation path and then

ninja install

We provide examples of linking against both Chrono and DEME for co-simulations in chrono-projects. You need to checkout the feature/DEME branch after cloning the code.

You need to build chrono-projects linking against a Chrono installation (Chrono installation guide is here; note you have to make install to install Chrono, not just build it), then link against DEME. The steps for building chrono-projects:

• Start by linking against Chrono. Set Chrono_DIR. It should be in <your_Chrono_install_dir>/lib/cmake. Then configure the project;
• Make sure ENABLE_PROJECTS to ON and configure the project;
• Linkage against Chrono is done, now move on to link against DEME. Set ENABLE_DEME_TESTS to ON. Then configure the project;
• Set DEME_DIR when prompted. It should be in <your_DEME_install_dir>/lib64/cmake/DEME. Then configure the project.
• You may see the ChPF_DIR option being prompted. If you did not build DEME with USE_CHPF being ON, you can ignore this option. Otherwise CMake will refuse to generate, and you need to set this option to be <your_DEME_install_dir>/lib64/cmake/ChPF.

Then build the project and you should be able to run the demo scripts that demonstrate the co-simulation between DEME and Chrono.

More documentations on using this package for co-simulations are being added.

Licensing

This project should be treated as the collective intellectual property of the Author(s) and the University of Wisconsin - Madison. The following copyright statement should be included in any new or modified source files

Copyright (c) 2021, Simulation-Based Engineering Laboratory
Copyright (c) 2021, University of Wisconsin - Madison

SPDX-License-Identifier: BSD-3-Clause

New authors should add their name to the file CONTRIBUTORS.md rather than to individual copyright headers.

Notes on code included from Project Chrono

This project exists independently of Chrono so developers should be sure to include the appropriate BSD license header on any code which is sourced from Chrono::GPU(DEM) or other parts of Chrono.

SAMPLE header for files sourced from Chrono

Copyright (c) 2021, SBEL GPU Development Team
Copyright (c) 2021, University of Wisconsin - Madison

SPDX-License-Identifier: BSD-3-Clause

This file contains modifications of the code authored by the Project Chrono 
Development Team. The original license can be found below:

Copyright (c) 2016, Project Chrono Development Team
All rights reserved.

Use of this source code is governed by a BSD-style license that can be found
in the LICENSE file at the top level of the distribution and at
http://projectchrono.org/license-chrono.txt. A copy of the license is below.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

 - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 
 - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 
 - Neither the name of the nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Citation

See the paper that explains the design and usage of DEME and cite

@article{zhang_2024_deme,
title = {Chrono {DEM-Engine}: A Discrete Element Method dual-{GPU} simulator with customizable contact forces and element shape},
journal = {Computer Physics Communications},
volume = {300},
pages = {109196},
year = {2024},
issn = {0010-4655},
doi = {https://doi.org/10.1016/j.cpc.2024.109196},
author = {Ruochun Zhang and Bonaventura Tagliafierro and Colin {Vanden Heuvel} and Shlok Sabarwal and Luning Bakke and Yulong Yue and Xin Wei and Radu Serban and Dan Negruţ},
keywords = {Discrete Element Method, GPU computing, Physics-based simulation, Scientific package, BSD3 open-source},
}

See the paper on using DEME for simulating rover dynamics and cite

@article{ruochunGRC-DEM2023,
      title={A {GPU}-accelerated simulator for the {DEM} analysis of granular systems composed of clump-shaped elements}, 
      author={Ruochun Zhang and Colin {Vanden Heuvel} and Alexander Schepelmann and Arno Rogg and Dimitrios Apostolopoulos and Samuel Chandler and Radu Serban and Dan Negrut},
      year={2024},
      journal={Engineering with Computers},
      doi={https://doi.org/10.1007/s00366-023-01921-9}
}