torch_example.cpp no known converstion from at:Tensor to const btScalar&

[DuaneNielsen]

/usr/local/include/bullet/LinearMath/btVector3.h:126:20: note: no known conversion for argument 1 from ‘at::Tensor’ to ‘const btScalar& {aka const double&}’

In this example, the sync_graphics_transforms function appears to try to create a btVector3 directly using at:Tensor

  static void sync_graphics_transforms(
      const TinyMultiBody<TinyScalar, TinyConstants>* body,
      class b3RobotSimulatorClientAPI_NoDirect& viz_api) {
    for (int v = 0; v < body->m_visual_uids1.size(); v++) {
      int visual_id = body->m_visual_uids1[v];
      TinyQuaternion<TinyScalar, TinyConstants> rot;
      TinySpatialTransform<TinyScalar, TinyConstants> geom_X_world =
          body->m_base_X_world * body->m_X_visuals[v];
      btVector3 base_pos(geom_X_world.m_translation.getX(),
                         geom_X_world.m_translation.getY(),
                         geom_X_world.m_translation.getZ());
      geom_X_world.m_rotation.getRotation(rot);
      btQuaternion base_orn(rot.getX(), rot.getY(), rot.getZ(), rot.getW());
      viz_api.resetBasePositionAndOrientation(visual_id, base_pos, base_orn);
    }

I noticed that in other functions there is a wrapper function to to a conversion. Perhaps it's missing one here?

      joint.joint_origin_xyz.setValue(TinyConstants::scalar_from_double(pos.getOrigin()[0]),
                                      TinyConstants::scalar_from_double(pos.getOrigin()[1]),
                                      TinyConstants::scalar_from_double(pos.getOrigin()[2]));

here is the compilation line... I'm using c++ torch 1.6

~/PycharmProjects/tiny-differentiable-simulator/examples$ /usr/bin/c++ -DBT_USE_DOUBLE_PRECISION -DCERES_USE_CXX11 -DCPPAD -DCPPADCG -DHPIPM -DSTATIC_LINK_SPD_PLUGIN -DUSE_EIGEN -DZMQ_STATIC -I/home/duane/PycharmProjects/tiny-differentiable-simulator/third_party/eigen3 -I/home/duane/PycharmProjects/tiny-differentiable-simulator/examples/. -I/home/duane/PycharmProjects/tiny-differentiable-simulator/examples/.. -I/home/duane/PycharmProjects/tiny-differentiable-simulator/examples/../third_party/zeromq/include -I/usr/local/include/bullet -I/usr/include/python3.6m -I/usr/lib/llvm-7/include -I/usr/local/include -I/usr/include/eigen3 -isystem /home/duane/Downloads/libtorch/include -I/usr/local/include/bullet/LinearMath -isystem /home/duane/Downloads/libtorch/include/torch/csrc/api/include -fPIC -D_GLIBCXX_USE_CXX11_ABI=0 -std=gnu++1z -o CMakeFiles/torch_example.dir/torch_example.o -c /home/duane/PycharmProjects/tiny-differentiable-simulator/examples/torch_example.cpp In file included from /home/duane/PycharmProjects/tiny-differentiable-simulator/examples/tiny_system_constructor.h:20:0, from /home/duane/PycharmProjects/tiny-differentiable-simulator/examples/torch_example.cpp:15: /home/duane/PycharmProjects/tiny-differentiable-simulator/examples/pybullet_urdf_import.h: In instantiation of ‘static void PyBulletUrdfImport<TinyScalar, TinyConstants>::sync_graphics_transforms(const TinyMultiBody<TinyScalar, TinyConstants>*, b3RobotSimulatorClientAPI_NoDirect&) [with TinyScalar = at::Tensor; TinyConstants = TorchUtils]’: /home/duane/PycharmProjects/tiny-differentiable-simulator/examples/torch_example.cpp:173:40: required from here /home/duane/PycharmProjects/tiny-differentiable-simulator/examples/pybullet_urdf_import.h:153:17: error: no matching function for call to ‘btVector3::btVector3(at::Tensor, at::Tensor, at::Tensor)’ btVector3 base_pos(geom_X_world.m_translation.getX(), ^~~~ In file included from /usr/local/include/bullet/SharedMemory/b3RobotSimulatorClientAPI_NoDirect.h:9:0, from /usr/local/include/bullet/SharedMemory/b3RobotSimulatorClientAPI_NoGUI.h:4, from /home/duane/PycharmProjects/tiny-differentiable-simulator/examples/pybullet_visualizer_api.h:22, from /home/duane/PycharmProjects/tiny-differentiable-simulator/examples/torch_example.cpp:10: /usr/local/include/bullet/LinearMath/btVector3.h:126:20: note: candidate: btVector3::btVector3(const btScalar&, const btScalar&, const btScalar&) SIMD_FORCE_INLINE btVector3(const btScalar& _x, const btScalar& _y, const btScalar& _z) ^~~~~ /usr/local/include/bullet/LinearMath/btVector3.h:126:20: note: no known conversion for argument 1 from ‘at::Tensor’ to ‘const btScalar& {aka const double&}’ /usr/local/include/bullet/LinearMath/btVector3.h:117:20: note: candidate: btVector3::btVector3() SIMD_FORCE_INLINE btVector3() ^~~~~ /usr/local/include/bullet/LinearMath/btVector3.h:117:20: note: candidate expects 0 arguments, 3 provided /usr/local/include/bullet/LinearMath/btVector3.h:81:1: note: candidate: constexpr btVector3::btVector3(const btVector3&) btVector3 ^~~~~ /usr/local/include/bullet/LinearMath/btVector3.h:81:1: note: candidate expects 1 argument, 3 provided /usr/local/include/bullet/LinearMath/btVector3.h:81:1: note: candidate: constexpr btVector3::btVector3(btVector3&&) /usr/local/include/bullet/LinearMath/btVector3.h:81:1: note: candidate expects 1 argument, 3 provided

[DuaneNielsen]

Upon further review, it seems that there are 2 main problems with torch_example.cpp.

1. In some cases if the passed in type is at::tensor it's not compatible with bullet, which expects a single scalar value.
2. Torch doesn't define an overload for boolean comparison operators between tensors. This is for obvious reasons, since tensors are matrices of numbers.

Seems like in nearly all cases, this can be fixed by assuming the variable in question is a single item, and calling TinyConstants::getDouble(), then operating on a double instead.

However this required about 40 lines of code to be changed, so I'm not sure it's the intended way to solve this.

[erwincoumans]

We are doing a refactor that can make it easier to support PyTorch. See https://github.com/erwincoumans/tiny-differentiable-simulator/tree/refactor

'getDouble is not supposed to be use in the actual code, it is mainly for debugging (and for Python script bindings)

There should be no need for 'bullet' in this repo, with PyTorch. We have a new OpenGL backend and meshcat, so I try to remove/deprecate all use of Bullet in this repository.

[DuaneNielsen]

Sounds great. Can't wait for that.

And thanks for the project. I learned a whole lot about auto-diff and rigid body physics simulation reading your code.

I really feel these kind of sims are going to be foundational for robotics.

You probably already saw this paper ADD: Analytically Differentiable Dynamics for Multi-Body Systems with Frictional Contact but I figured I'd link it in case.

Have a great day!

[erwincoumans]

The refactoring of the core is done. We will move more examples with autodiff capabilities to 'examples' later. Note that since this is an active research projects with no external 'users' (unlike pybullet for example), things may change a bit faster. It is likely best to start a separate fork from the current master (which was 'refactor' before), since many files are moved around so you get likely conflicts/non-working code with the original master.

[tsbertalan]

Did the PyTorch integration ever progress? I imagine use of this package would explode if (1) I could interface it with other PyTorch-ecosystem packages, and (2) the version I could install from PyPI was up-to-date.