search

google-deepmind / mujoco

Multi-Joint dynamics with Contact. A general purpose physics simulator.
https://mujoco.org
Apache License 2.0
7.76k stars 770 forks source link

Asking for advice on improving the simulation performance #986

Closed breakds closed 1 year ago

breakds commented 1 year ago

Hi,

Greetings! We have recently been utilizing MuJoCo for our project involving the simulation of a Unitree Go1 quadrupedal robot navigating across varied terrains. However, we're facing some performance bottlenecks during our training cycles. Consequently, we're seeking your insights and advice on potential ways to enhance the simulation speed.

Below, I've provided some context and specifics from our experiment conducted on an AMD Ryzen 9 5950x CPU, which has 16 cores and 32 threads.

Question 1: Discrepancies in Simulation Speed Factors

The MuJoCo MPC paper indicates that a single CPU thread can simulate a 27-DoF humanoid system 4000 times faster than real-time.

for example, the humanoid (a 27-DoF system) can be simulated 4000 times faster than realtime on a single CPU thread.

To replicate this performance, we ran testspeed on the humanoid model and achieved a real:sim factor of roughly 150 times for a single thread. Although this surpasses the speed reported in post #897, it falls far short of the 4000x speed cited in the paper.

Running 10000 steps per thread at dt = 0.005 ...

Summary for all 16 threads

 Total simulation time  : 0.44 s
 Total steps per second : 366686
 Total realtime factor  : 1833.43 x
 Total time per step    : 2.7 µs

Details for thread 0

 Simulation time      : 0.34 s
 Steps per second     : 29722
 Realtime factor      : 148.61 x
 Time per step        : 33.6 µs

 Broadphase accuracy  : 44.85%
 Midphase accuracy    : 40.36%
 Contacts per step    : 12.56
 Constraints per step : 43.56
 Degrees of freedom   : 27

 Internal profiler for thread 0 (µs per step)
             step :   33.5  (100.00 %)
          forward :   31.8  ( 95.04 %)
         position :   17.1  ( 51.03 %)
         velocity :    1.7  (  5.08 %)
        actuation :    0.3  (  0.92 %)
     acceleration :    0.6  (  1.72 %)
       constraint :   12.0  ( 35.76 %)
   pos_kinematics :    2.4  (  7.07 %)
      pos_inertia :    2.0  (  5.84 %)
    pos_collision :    8.3  ( 24.73 %)
         pos_make :    4.3  ( 12.80 %)
      pos_project :    0.0  (  0.08 %)
            other :    1.8  (  5.49 %)

Can anyone guide us on how we can approach this 4000x magnitude? Are there any specific configurations, compilation flags, or model modifications that we need to consider? We are building MuJoCo from the source and any advice regarding this matter would be highly appreciated!

Below is the build log when I build MuJoCo from source. AVX should be enabled by default.

Build Log for MuJoCo ``` mujoco> unpacking sources mujoco> unpacking source archive /nix/store/wxivcwsqz6zylr31hnfx1cckbwl8wr3i-source mujoco> source root is source mujoco> patching sources mujoco> applying patch /nix/store/fpka17kjcczv7zc1cmq4j7p7dpf49pcp-dependencies.patch mujoco> patching file cmake/MujocoDependencies.cmake mujoco> patching file simulate/cmake/SimulateDependencies.cmake mujoco> configuring mujoco> fixing cmake files... mujoco> cmake flags: -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/doc/mujoco -DCMAKE_INSTALL_INFODIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/info -DCMAKE_INSTALL_MANDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/man -DCMAKE_INSTALL_OLDINCLUDEDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include -DCMAKE_INSTALL_SBINDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/sbin -DCMAKE_INSTALL_BINDIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_OSX_SYSROOT= -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/d9fndiing52fkalp5knfalrvlb3isi6w-gcc-wrapper-12.2.0/bin/strip -DCMAKE_RANLIB=/nix/store/d9fndiing52fkalp5knfalrvlb3isi6w-gcc-wrapper-12.2.0/bin/ranlib -DCMAKE_AR=/nix/store/d9fndiing52fkalp5knfalrvlb3isi6w-gcc-wrapper-12.2.0/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=/nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6 mujoco> -- The C compiler identification is GNU 12.2.0 mujoco> -- The CXX compiler identification is GNU 12.2.0 mujoco> -- Detecting C compiler ABI info mujoco> -- Detecting C compiler ABI info - done mujoco> -- Check for working C compiler: /nix/store/d9fndiing52fkalp5knfalrvlb3isi6w-gcc-wrapper-12.2.0/bin/gcc - skipped mujoco> -- Detecting C compile features mujoco> -- Detecting C compile features - done mujoco> -- Detecting CXX compiler ABI info mujoco> -- Detecting CXX compiler ABI info - done mujoco> -- Check for working CXX compiler: /nix/store/d9fndiing52fkalp5knfalrvlb3isi6w-gcc-wrapper-12.2.0/bin/g++ - skipped mujoco> -- Detecting CXX compile features mujoco> -- Detecting CXX compile features - done mujoco> -- Performing Test CAN_BUILD_AVX mujoco> -- Performing Test CAN_BUILD_AVX - Success mujoco> -- Performing Test SUPPORTS_LLD mujoco> -- Performing Test SUPPORTS_LLD - Failed mujoco> -- Performing Test SUPPORTS_GOLD mujoco> -- Performing Test SUPPORTS_GOLD - Success mujoco> -- Performing Test SUPPORTS_GC_SECTIONS mujoco> -- Performing Test SUPPORTS_GC_SECTIONS - Success mujoco> -- mujoco::FindOrFetch: checking for targets in package `qhull` mujoco> -- mujoco::FindOrFetch: checking for targets in package `qhull` - target `qhull` not defined. mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `qhull` mujoco> -- Looking for sys/types.h mujoco> -- Looking for sys/types.h - found mujoco> -- Looking for inttypes.h mujoco> -- Looking for inttypes.h - found mujoco> -- Looking for stddef.h mujoco> -- Looking for stddef.h - found mujoco> -- Looking for stdint.h mujoco> -- Looking for stdint.h - found mujoco> -- Check size of off_t mujoco> -- Check size of off_t - done mujoco> -- Looking for fseeko mujoco> -- Looking for fseeko - found mujoco> -- Looking for ftello mujoco> -- Looking for ftello - found mujoco> -- Looking for PRIdMAX mujoco> -- Looking for PRIdMAX - found mujoco> -- mujoco> -- ========== qhull Build Information ========== mujoco> -- Build Version: 8.1-alpha1 mujoco> -- Install Prefix (CMAKE_INSTALL_PREFIX): /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6 mujoco> -- Binary Directory (BIN_INSTALL_DIR): bin mujoco> -- Library Directory (LIB_INSTALL_DIR): lib mujoco> -- Include Directory (INCLUDE_INSTALL_DIR): include mujoco> -- Documentation Directory (DOC_INSTALL_DIR): share/doc/qhull mujoco> -- Man Pages Directory (MAN_INSTALL_DIR): share/man/man1 mujoco> -- CMake Directory (CMAKE_INSTALL_DIR): lib/cmake/QHull mujoco> -- PkgConfig Directory (PKGCONFIG_INSTALL_DIR):lib/pkgconfig mujoco> -- Build Type (CMAKE_BUILD_TYPE): Release mujoco> -- Build static libraries: ON mujoco> -- Build shared library: OFF mujoco> -- Use shared library for linking apps: OFF mujoco> -- Build tests: OFF mujoco> -- To override these options, add -D{OPTION_NAME}=... to the cmake command mujoco> -- Build the debug targets -DCMAKE_BUILD_TYPE=Debug mujoco> -- mujoco> -- To build and install qhull, enter "make" and "make install" mujoco> -- To smoketest qhull, enter "ctest" mujoco> -- mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `qhull` - Done mujoco> -- mujoco::FindOrFetch: checking for targets in package `tinyxml2` mujoco> -- mujoco::FindOrFetch: checking for targets in package `tinyxml2` - target `tinyxml2` not defined. mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `tinyxml2` mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `tinyxml2` - Done mujoco> -- mujoco::FindOrFetch: checking for targets in package `tinyobjloader` mujoco> -- mujoco::FindOrFetch: checking for targets in package `tinyobjloader` - target `tinyobjloader` not defined. mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `tinyobjloader` mujoco> CMake Deprecation Warning at build/_deps/tinyobjloader-src/CMakeLists.txt:5 (cmake_minimum_required): mujoco> Compatibility with CMake < 2.8.12 will be removed from a future version of mujoco> CMake. mujoco> Update the VERSION argument value or use a ... suffix to tell mujoco> CMake that the project does not need compatibility with older versions. mujoco> mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `tinyobjloader` - Done mujoco> -- mujoco::FindOrFetch: checking for targets in package `ccd` mujoco> -- mujoco::FindOrFetch: checking for targets in package `ccd` - target `ccd` not defined. mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `ccd` mujoco> CMake Deprecation Warning at build/_deps/ccd-src/CMakeLists.txt:1 (cmake_minimum_required): mujoco> Compatibility with CMake < 2.8.12 will be removed from a future version of mujoco> CMake. mujoco> Update the VERSION argument value or use a ... suffix to tell mujoco> CMake that the project does not need compatibility with older versions. mujoco> mujoco> CMake Warning (dev) at build/_deps/ccd-src/CMakeLists.txt:9 (project): mujoco> Policy CMP0048 is not set: project() command manages VERSION variables. mujoco> Run "cmake --help-policy CMP0048" for policy details. Use the cmake_policy mujoco> command to set the policy and suppress this warning. mujoco> The following variable(s) would be set to empty: mujoco> PROJECT_VERSION mujoco> PROJECT_VERSION_MAJOR mujoco> PROJECT_VERSION_MINOR mujoco> PROJECT_VERSION_PATCH mujoco> This warning is for project developers. Use -Wno-dev to suppress it. mujoco> mujoco> -- Performing Test COMPILER_SUPPORTS_VISIBILITY mujoco> -- Performing Test COMPILER_SUPPORTS_VISIBILITY - Success mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `ccd` - Done mujoco> -- mujoco::FindOrFetch: checking for targets in package `absl` mujoco> -- mujoco::FindOrFetch: checking for targets in package `absl` - target `absl::core_headers` not defined. mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `abseil-cpp` mujoco> -- Performing Test ABSL_INTERNAL_AT_LEAST_CXX17 mujoco> -- Performing Test ABSL_INTERNAL_AT_LEAST_CXX17 - Success mujoco> -- Performing Test CMAKE_HAVE_LIBC_PTHREAD mujoco> -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success mujoco> -- Found Threads: TRUE mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `abseil-cpp` - Done mujoco> -- mujoco::FindOrFetch: checking for targets in package `GTest` mujoco> -- mujoco::FindOrFetch: checking for targets in package `GTest` - target `gtest` not defined. mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `googletest` mujoco> -- Could NOT find Python (missing: Python_EXECUTABLE Interpreter) mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `googletest` - Done mujoco> -- mujoco::FindOrFetch: checking for targets in package `benchmark` mujoco> -- mujoco::FindOrFetch: checking for targets in package `benchmark` - target `benchmark::benchmark` not defined. mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `benchmark` mujoco> -- Failed to find LLVM FileCheck mujoco> -- Could NOT find Git (missing: GIT_EXECUTABLE) mujoco> -- Version: 1.7.1 mujoco> -- Looking for shm_open in rt mujoco> -- Looking for shm_open in rt - found mujoco> -- Performing Test HAVE_CXX_FLAG_WALL mujoco> -- Performing Test HAVE_CXX_FLAG_WALL - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WEXTRA mujoco> -- Performing Test HAVE_CXX_FLAG_WEXTRA - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WSHADOW mujoco> -- Performing Test HAVE_CXX_FLAG_WSHADOW - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL mujoco> -- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WERROR mujoco> -- Performing Test HAVE_CXX_FLAG_WERROR - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE mujoco> -- Performing Test HAVE_CXX_FLAG_WSUGGEST_OVERRIDE - Success mujoco> -- Performing Test HAVE_CXX_FLAG_PEDANTIC mujoco> -- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success mujoco> -- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS mujoco> -- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 mujoco> -- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed mujoco> -- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING mujoco> -- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS mujoco> -- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED mujoco> -- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING mujoco> -- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success mujoco> -- Performing Test HAVE_CXX_FLAG_WD654 mujoco> -- Performing Test HAVE_CXX_FLAG_WD654 - Failed mujoco> -- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY mujoco> -- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed mujoco> -- Performing Test HAVE_CXX_FLAG_COVERAGE mujoco> -- Performing Test HAVE_CXX_FLAG_COVERAGE - Success mujoco> -- Compiling and running to test HAVE_STD_REGEX mujoco> -- Performing Test HAVE_STD_REGEX -- success mujoco> -- Compiling and running to test HAVE_GNU_POSIX_REGEX mujoco> -- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile mujoco> -- Compiling and running to test HAVE_POSIX_REGEX mujoco> -- Performing Test HAVE_POSIX_REGEX -- success mujoco> -- Compiling and running to test HAVE_STEADY_CLOCK mujoco> -- Performing Test HAVE_STEADY_CLOCK -- success mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `benchmark` - Done mujoco> -- mujoco::FindOrFetch: checking for targets in package `mujoco` mujoco> -- mujoco::FindOrFetch: checking for targets in package `mujoco` - found mujoco> -- mujoco::FindOrFetch: checking for targets in package `glfw3` mujoco> -- mujoco::FindOrFetch: checking for targets in package `glfw3` - target `glfw` not defined. mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `glfw3` mujoco> -- Using X11 for window creation mujoco> -- Found X11: /nix/store/gl91xz6rfkdkkqwpxmmizjmciw6sd3jg-xorgproto-2021.5/include mujoco> -- Looking for XOpenDisplay in /nix/store/gzs13l92w3308ichywy2013bw2n6l413-libX11-1.8.4/lib/libX11.so;/nix/store/pfn3d53934dswdsf269kmi2acbcjkq2p-libXext-1.3.4/lib/libXext.so mujoco> -- Looking for XOpenDisplay in /nix/store/gzs13l92w3308ichywy2013bw2n6l413-libX11-1.8.4/lib/libX11.so;/nix/store/pfn3d53934dswdsf269kmi2acbcjkq2p-libXext-1.3.4/lib/libXext.so - found mujoco> -- Looking for gethostbyname mujoco> -- Looking for gethostbyname - found mujoco> -- Looking for connect mujoco> -- Looking for connect - found mujoco> -- Looking for remove mujoco> -- Looking for remove - found mujoco> -- Looking for shmat mujoco> -- Looking for shmat - found mujoco> -- mujoco::FindOrFetch: Using FetchContent to retrieve `glfw3` - Done mujoco> -- mujoco::FindOrFetch: checking for targets in package `mujoco` mujoco> -- mujoco::FindOrFetch: checking for targets in package `mujoco` - found mujoco> -- mujoco::FindOrFetch: checking for targets in package `glfw3` mujoco> -- mujoco::FindOrFetch: checking for targets in package `glfw3` - found mujoco> -- Configuring done mujoco> -- Generating done mujoco> CMake Warning: mujoco> Manually-specified variables were not used by the project: mujoco> CMAKE_EXPORT_NO_PACKAGE_REGISTRY mujoco> mujoco> -- Build files have been written to: /build/source/build mujoco> cmake: enabled parallel building mujoco> cmake: enabled parallel installing mujoco> cmake version 3.25.3 mujoco> CMake suite maintained and supported by Kitware (kitware.com/cmake). mujoco> CMakeCache.txt bin lib nix src mujoco> CMakeFiles ccd-targets.cmake model plugin mujoco> Makefile cmake_install.cmake mujocoConfig.cmake sample mujoco> _deps compile_commands.json mujocoConfigVersion.cmake simulate mujoco> ------------------------------------------------------------ mujoco> building mujoco> build flags: -j64 SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash mujoco> [ 0%] Building C object _deps/ccd-build/src/CMakeFiles/ccd.dir/ccd.c.o mujoco> [ 0%] Building C object _deps/ccd-build/src/CMakeFiles/ccd.dir/mpr.c.o mujoco> [ 0%] Building CXX object _deps/tinyobjloader-build/CMakeFiles/tinyobjloader.dir/tiny_obj_loader.cc.o mujoco> [ 4%] Building C object _deps/ccd-build/src/CMakeFiles/ccd.dir/polytope.c.o mujoco> [ 4%] Building C object _deps/ccd-build/src/CMakeFiles/ccd.dir/vec3.c.o mujoco> [ 4%] Building CXX object _deps/tinyxml2-build/CMakeFiles/tinyxml2.dir/tinyxml2.cpp.o mujoco> [ 4%] Building C object _deps/ccd-build/src/CMakeFiles/ccd.dir/support.c.o mujoco> [ 4%] Building CXX object CMakeFiles/lodepng.dir/_deps/lodepng-src/lodepng.cpp.o mujoco> [ 8%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/init.c.o mujoco> [ 8%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/input.c.o mujoco> [ 8%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/context.c.o mujoco> [ 8%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/monitor.c.o mujoco> [ 8%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/global_r.c.o mujoco> [ 8%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/window.c.o mujoco> [ 8%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/x11_monitor.c.o mujoco> [ 12%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/stat_r.c.o mujoco> [ 16%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/x11_init.c.o mujoco> [ 16%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/poly2_r.c.o mujoco> [ 16%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/x11_window.c.o mujoco> [ 16%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/vulkan.c.o mujoco> [ 16%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/merge_r.c.o mujoco> [ 16%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/libqhull_r.c.o mujoco> [ 20%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/geom_r.c.o mujoco> [ 20%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/posix_time.c.o mujoco> [ 20%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/geom2_r.c.o mujoco> [ 20%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/egl_context.c.o mujoco> [ 20%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/xkb_unicode.c.o mujoco> [ 20%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/osmesa_context.c.o mujoco> [ 20%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/qset_r.c.o mujoco> [ 25%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/posix_thread.c.o mujoco> [ 25%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/glx_context.c.o mujoco> [ 25%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/poly_r.c.o mujoco> [ 25%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/userprintf_rbox_r.c.o mujoco> [ 25%] Building C object _deps/glfw3-build/src/CMakeFiles/glfw.dir/linux_joystick.c.o mujoco> [ 25%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/userprintf_r.c.o mujoco> [ 25%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/io_r.c.o mujoco> [ 25%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/mem_r.c.o mujoco> [ 33%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/rboxlib_r.c.o mujoco> [ 33%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/usermem_r.c.o mujoco> [ 33%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/random_r.c.o mujoco> [ 33%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/user_r.c.o mujoco> [ 33%] Building C object _deps/qhull-build/CMakeFiles/qhullstatic_r.dir/src/libqhull_r/accessors_r.c.o mujoco> [ 33%] Linking C static library ../../../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libccd.a mujoco> [ 33%] Built target ccd mujoco> [ 37%] Linking C static library ../../../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libglfw3.a mujoco> [ 37%] Built target glfw mujoco> [ 41%] Linking CXX static library ../../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libtinyxml2.a mujoco> [ 41%] Built target tinyxml2 mujoco> [ 41%] Linking C static library ../../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libqhullstatic_r.a mujoco> [ 41%] Built target qhullstatic_r mujoco> [ 41%] Linking CXX static library ../../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libtinyobjloader.a mujoco> [ 41%] Built target tinyobjloader mujoco> [ 41%] Linking CXX static library nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/liblodepng.a mujoco> [ 41%] Built target lodepng mujoco> [ 45%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_callback.c.o mujoco> [ 45%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_collision_box.c.o mujoco> [ 45%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_collision_driver.c.o mujoco> [ 45%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_collision_convex.c.o mujoco> [ 45%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_collision_primitive.c.o mujoco> [ 50%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_core_constraint.c.o mujoco> [ 50%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_core_smooth.c.o mujoco> [ 50%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_crossplatform.c.o mujoco> [ 50%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_derivative.c.o mujoco> [ 50%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_derivative_fd.c.o mujoco> [ 54%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_forward.c.o mujoco> [ 54%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_inverse.c.o mujoco> [ 54%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_io.c.o mujoco> [ 54%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_passive.c.o mujoco> [ 58%] Building CXX object CMakeFiles/mujoco.dir/src/engine/engine_plugin.cc.o mujoco> [ 58%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_print.c.o mujoco> [ 58%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_ray.c.o mujoco> [ 58%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_resource.c.o mujoco> [ 58%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_sensor.c.o mujoco> [ 58%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_setconst.c.o mujoco> [ 58%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_solver.c.o mujoco> [ 58%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_util_blas.c.o mujoco> [ 62%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_support.c.o mujoco> [ 62%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_util_errmem.c.o mujoco> [ 62%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_util_misc.c.o mujoco> [ 62%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_util_solve.c.o mujoco> [ 62%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_util_spatial.c.o mujoco> [ 62%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_vfs.c.o mujoco> [ 66%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_util_sparse.c.o mujoco> [ 66%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_vis_init.c.o mujoco> [ 66%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_vis_interact.c.o mujoco> [ 70%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_vis_state.c.o mujoco> [ 70%] Building C object CMakeFiles/mujoco.dir/src/engine/engine_vis_visualize.c.o mujoco> [ 70%] Building CXX object CMakeFiles/mujoco.dir/src/user/user_composite.cc.o mujoco> [ 70%] Building CXX object CMakeFiles/mujoco.dir/src/user/user_mesh.cc.o mujoco> [ 70%] Building CXX object CMakeFiles/mujoco.dir/src/user/user_model.cc.o mujoco> [ 70%] Building CXX object CMakeFiles/mujoco.dir/src/xml/xml_api.cc.o mujoco> [ 70%] Building CXX object CMakeFiles/mujoco.dir/src/user/user_util.cc.o mujoco> [ 75%] Building CXX object CMakeFiles/mujoco.dir/src/user/user_objects.cc.o mujoco> [ 79%] Building CXX object CMakeFiles/mujoco.dir/src/xml/xml_native_reader.cc.o mujoco> [ 79%] Building CXX object CMakeFiles/mujoco.dir/src/xml/xml_native_writer.cc.o mujoco> [ 79%] Building CXX object CMakeFiles/mujoco.dir/src/xml/xml_base.cc.o mujoco> [ 79%] Building CXX object CMakeFiles/mujoco.dir/src/xml/xml_util.cc.o mujoco> [ 79%] Building CXX object CMakeFiles/mujoco.dir/src/xml/xml_urdf.cc.o mujoco> [ 79%] Building CXX object CMakeFiles/mujoco.dir/src/xml/xml_numeric_format.cc.o mujoco> [ 79%] Building CXX object CMakeFiles/mujoco.dir/src/xml/xml.cc.o mujoco> [ 79%] Building C object CMakeFiles/mujoco.dir/src/render/render_context.c.o mujoco> [ 83%] Building CXX object CMakeFiles/mujoco.dir/src/render/glad/loader.cc.o mujoco> [ 83%] Building C object CMakeFiles/mujoco.dir/src/render/glad/glad.c.o mujoco> [ 83%] Building C object CMakeFiles/mujoco.dir/src/render/render_gl2.c.o mujoco> [ 83%] Building C object CMakeFiles/mujoco.dir/src/render/render_gl3.c.o mujoco> [ 87%] Building C object CMakeFiles/mujoco.dir/src/render/render_util.c.o mujoco> [ 87%] Building C object CMakeFiles/mujoco.dir/src/ui/ui_main.c.o mujoco> [ 87%] Linking CXX shared library nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libmujoco.so mujoco> [ 87%] Built target mujoco mujoco> [ 87%] Building CXX object plugin/sensor/CMakeFiles/sensor.dir/sensor.cc.o mujoco> [ 87%] Building CXX object plugin/sensor/CMakeFiles/sensor.dir/touch_grid.cc.o mujoco> [ 87%] Building CXX object plugin/elasticity/CMakeFiles/elasticity.dir/elasticity.cc.o mujoco> [ 87%] Building CXX object sample/CMakeFiles/compile.dir/compile.cc.o mujoco> [ 91%] Building CXX object sample/CMakeFiles/testspeed.dir/testspeed.cc.o mujoco> [ 91%] Building CXX object plugin/elasticity/CMakeFiles/elasticity.dir/cable.cc.o mujoco> [ 91%] Building CXX object plugin/elasticity/CMakeFiles/elasticity.dir/solid.cc.o mujoco> [ 91%] Building CXX object sample/CMakeFiles/derivative.dir/derivative.cc.o mujoco> [ 91%] Building CXX object simulate/CMakeFiles/platform_ui_adapter.dir/glfw_dispatch.cc.o mujoco> [ 91%] Building CXX object simulate/CMakeFiles/platform_ui_adapter.dir/glfw_adapter.cc.o mujoco> [ 91%] Building CXX object simulate/CMakeFiles/platform_ui_adapter.dir/platform_ui_adapter.cc.o mujoco> [ 95%] Building CXX object sample/CMakeFiles/record.dir/record.cc.o mujoco> [ 95%] Building CXX object sample/CMakeFiles/testxml.dir/testxml.cc.o mujoco> [ 95%] Building CXX object sample/CMakeFiles/basic.dir/basic.cc.o mujoco> [ 95%] Linking CXX executable ../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/basic mujoco> [ 95%] Linking CXX executable ../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/record mujoco> [ 95%] Built target basic mujoco> [ 95%] Built target platform_ui_adapter mujoco> [ 95%] Built target record mujoco> [ 95%] Building CXX object simulate/CMakeFiles/libsimulate.dir/simulate.cc.o mujoco> [ 95%] Linking CXX executable ../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/derivative mujoco> [ 95%] Linking CXX executable ../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/compile mujoco> [ 95%] Built target derivative mujoco> [ 95%] Built target compile mujoco> [ 95%] Linking CXX executable ../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/testspeed mujoco> [ 95%] Built target testspeed mujoco> [100%] Linking CXX shared library ../../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libsensor.so mujoco> [100%] Built target sensor mujoco> [100%] Linking CXX shared library ../../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libelasticity.so mujoco> [100%] Built target elasticity mujoco> [100%] Linking CXX static library ../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libsimulate.a mujoco> [100%] Built target libsimulate mujoco> [100%] Building CXX object simulate/CMakeFiles/simulate.dir/main.cc.o mujoco> [100%] Linking CXX executable ../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/testxml mujoco> [100%] Built target testxml mujoco> [100%] Linking CXX executable ../nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/simulate mujoco> [100%] Built target simulate mujoco> installing mujoco> install flags: -j64 SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash install mujoco> [ 4%] Built target lodepng mujoco> [ 4%] Built target ccd mujoco> [ 8%] Built target tinyxml2 mujoco> [ 8%] Built target tinyobjloader mujoco> [ 25%] Built target qhullstatic_r mujoco> [ 41%] Built target glfw mujoco> [ 87%] Built target mujoco mujoco> [ 91%] Built target derivative mujoco> [ 91%] Built target sensor mujoco> [ 91%] Built target compile mujoco> [ 91%] Built target basic mujoco> [ 95%] Built target testspeed mujoco> [ 95%] Built target testxml mujoco> [ 95%] Built target platform_ui_adapter mujoco> [ 95%] Built target record mujoco> [100%] Built target elasticity mujoco> [100%] Built target libsimulate mujoco> [100%] Built target simulate mujoco> Install the project... mujoco> -- Install configuration: "Release" mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/simulate mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/basic mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/compile mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/derivative mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/record mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/testspeed mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/bin/testxml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libmujoco.so.2.3.6 mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/libmujoco.so mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjdata.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjexport.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjmacro.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjmodel.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjplugin.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjrender.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjtnum.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjui.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjvisualize.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mjxmacro.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/include/mujoco/mujoco.h mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/cmake/mujoco/mujocoTargets.cmake mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/cmake/mujoco/mujocoTargets-release.cmake mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/cmake/mujoco/mujocoConfig.cmake mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/lib/cmake/mujoco/mujocoConfigVersion.cmake mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/flag mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/flag/flag.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/slider_crank mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/slider_crank/slider_crank.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/balloons mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/balloons/balloons.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/jelly.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/cable.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/press.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/scene.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/belt.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/a.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/coil.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/floppy.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/plugin/touch_grid.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/humanoid mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/humanoid/22_humanoids.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/humanoid/humanoid.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/humanoid/humanoid.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/humanoid/README.md mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/tendon_arm mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/tendon_arm/arm26.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/hammock mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/hammock/hammock.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/hammock/humanoid_body.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/adhesion mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/adhesion/active_adhesion.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/adhesion/README.md mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/softbox.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/grid2pin.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/particle_free.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/particle.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/scene.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/particle_free2d.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/asset mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/asset/carpet.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/asset/marble.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/asset/sponge.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/cloth.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/composite/loop.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/humanoid100 mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/humanoid100/humanoid100.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/humanoid100/humanoid_body.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green_white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue_orange_yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green_orange.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/orange_yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/red_white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/orange_white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/red_yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue_red.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green_red_white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue_red_yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/red.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green_orange_white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue_orange.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue_yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue_orange_white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green_red_yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue_white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green_orange_yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue_red_white.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/blue.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green_red.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/green_yellow.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/orange_red.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/assets/orange.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/README.md mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/cube/cube_3x3x3.xml mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/mug mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/mug/mug.obj mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/mug/mug.png mujoco> -- Installing: /nix/store/hj2d8zyd3yjb5n6s5m1m1k22i5rbniwh-mujoco-2.3.6/share/mujoco/model/mug/mug.xml ```

Question 2:

For our specific Go1 robot and terrain map setting (see the model here), the simulation speed dramatically decreases compared to the humanoid model.

Running 10000 steps per thread at dt = 0.005 ...

Summary for all 16 threads

 Total simulation time  : 10.41 s
 Total steps per second : 15377
 Total realtime factor  : 76.88 x
 Total time per step    : 65.0 µs

Details for thread 0

 Simulation time      : 8.27 s
 Steps per second     : 1209
 Realtime factor      : 6.05 x
 Time per step        : 826.9 µs

 Broadphase accuracy  : 24.62%
 Midphase accuracy    : 12.47%
 Contacts per step    : 40.70
 Constraints per step : 254.77
 Degrees of freedom   : 18

 Internal profiler for thread 0 (µs per step)
             step :  826.7  (100.00 %)
          forward :  825.7  ( 99.88 %)
         position :  178.0  ( 21.53 %)
         velocity :    2.2  (  0.26 %)
        actuation :    0.2  (  0.03 %)
     acceleration :    0.3  (  0.04 %)
       constraint :  642.7  ( 77.74 %)
   pos_kinematics :    2.2  (  0.26 %)
      pos_inertia :    1.0  (  0.12 %)
    pos_collision :  160.1  ( 19.37 %)
         pos_make :   14.5  (  1.75 %)
      pos_project :    0.0  (  0.00 %)
            other :    3.3  (  0.40 %)

This MJCF model works adequately for policy training, but the simulation speed is considerably slower, which hinders our process. The terrain map consists of 1600 x 1600 points. Are there any suggestions on improving the performance for this specific scenario?

Thank you for your time and consideration. Your insights and assistance are greatly valued!

saran-t commented 1 year ago

You indicated that you built from source. As with all questions relating to performance, I'd strongly urge you to get baseline numbers from our official binary build first.

EDIT: Looks like @yuvaltassa just posted an actual answer, so please ignore me. (Although I just ran the official build of testspeed on my own 5950x machine and got a bit higher throughput than what you posted. I suspect you're missing the INTERPROCEDURAL_OPTIMIZATION flag.)

yuvaltassa commented 1 year ago

Hi, thanks for your questions!

Question 1: I apologize, I believe the line in the paper is a (rather serious) mistake. "on a single CPU thread" should read "on a single CPU". Note however that real-time ratios scale with the timestep. The humanoid model in the MuJoCo repository has a conservative 5ms timestep, but the one used by the MJPC planner has a timestep of 15ms which leads to a total real-time ratio of 4000x on an M1 Mac and 5400x on your machine. Of course large timesteps are "dangerous" in the sense that the physics can sometimes diverge, but when used in the context of short planner trajectories (that handle divergence gracefully), this is not a problem. The humanoid environment controlled by the MJPC planner actually has the default timestep of 2ms, for high stability and fidelity. Such are the benefits of asynchronous MPC, you can choose timesteps to suit your needs 🙂

Question 2: I think I can see multiple issues in the timing breakdown.

  1. I find it curious that you have so many contacts. You say you have a quadruped on some terrain? How do you get to an average of 40 contacts / step? You have lots of geoms on the feet or something? Are you using a very dense height-field? (don't do that).
  2. I find the constraint/contact ratio curious. I think you must be using 6D contacts? These are not free...
  3. I find the 77% time spent in constraint resolution too high. Try switching solver and/or setting jacobian = "sparse".
  4. The 19% time spent in collision detection is also pretty high. This is something we can help with. In version 2.3.3 we introduced a new mid-phase pruning stage that significantly enhanced collision detection speed (credit to @quagla!). The speedup was made possible because a user sent us models that we could benchmark against. The same thing that was done for individual geoms could and should be done with heightfields: use a binary tree search instead of nested for loops. Unfortunately we ourselves do not use heightfields much and do not have example benchmark models. If you send us your model we can see if we can add tree-based pruning to heightfields. But I would really ask that you first fix whatever you did that made constraint resolution so slow.

Very generally, MuJoCo has a built in profiler that is visible in the simulate app GUI (F3 key), that will show you how different choices affect compute speed (this is printed out in testspeed). Use it!

Allow me to apologize again for the typo in the paper, it is rather embarrassing 😬

EDIT: Just realised you did attach a model. I will look more carefully tomorrow, but your 3.7MB height-field data file leads me to think that it is much denser than in needs to be.

breakds commented 1 year ago

Really appreciate your fast response!

I suspect you're missing the INTERPROCEDURAL_OPTIMIZATION flag.)

Thanks! This did boost the realtime factor from 1833 to 2200. Is this close to what you have @saran-t ?

@yuvaltassa Thanks a lot for the clarification on the MuJoCo MPC paper. The paper is very nice and the various benefits of using MuJoCo for planner rollout looks like a good fit.

On Question 2:

I find it curious that you have so many contacts. You say you have a quadruped on some terrain? How do you get to an average of 40 contacts / step? You have lots of geoms on the feet or something? Are you using a very dense height-field? (don't do that).

Unfortunately yes, we do have a very dense height field, with 1600 x 1600 vertex in 12m x 12m ground. The reason we have this is that since we are training a policy with model-free reinforcement learning, we need a relatively dense terrain to encourage the learned locomotion policy to lift feet hand enough.

I find the constraint/contact ratio curious. I think you must be using 6D contacts? These are not free...

Yes we set condim="6" on all feet based on code from mujoco-menagerie. When lower this down to "3", the simulation is indeed 20% faster. I am not sure whether this will affect sim2real at this moment.

I find the 77% time spent in constraint resolution too high. Try switching solver and/or setting jacobian = "sparse".

Using "CG" solver did help double the simulation performance. Setting "jacobian" to "sparse" does not seem to further improve the performance.

In version 2.3.3 we introduced a new mid-phase pruning stage that significantly enhanced collision detection speed (credit to @quagla!).

I noticed this when 2.3.3 was released (really appreciate your effort on actively developing it and regularly making new releases!). The algorithm was clever. I understand that currently it does not seem to work on heightfields - but in the end we just want some terrain with irregular ups and downs. Do you think it would be better if instead we create the terrain with a bunch of cubes on the ground which cannot move?

Very generally, MuJoCo has a built in profiler that is visible in the simulate app GUI (F3 key) Nice. This is very useful. Thank you!

Thanks again for your time and patiently answer the questions!

breakds commented 1 year ago

So the only unresolved problem (and the biggest) is that we have a rather dense hfield. We need this to allow the training to get a policy where the quadruped can lift its legs higher.

In terms of performance, instead of using hfield, is it going to be faster if we replace the terrain with (a huge amount of) blocks, so that the binary tree search trick can kick in?

breakds commented 1 year ago

Another trick I can think of is to place 20 robots on a single terrain (i.e. a model with 20 robots) for training. I can carefully assign the contype and conaffinity to make those robots "disjoint" for conflict detection purpose. This seems to increase the "per-robot" performance a lot, but I think this might not be recommended and can even have harmful side effects that I am not aware of.

yuvaltassa commented 1 year ago

Replacing the hfield with primitives should help, though I suspect that you might be able to get away with a somewhat less dense hfield.

I can't think of side-effects of making a multi-robot scene, this should be fine.

Closing for now but will keep following this thread.