search

dojo-sim / Dojo.jl

A differentiable physics engine for robotics
MIT License
309 stars 27 forks source link

Shift most test from FiniteDiff to ForwardDiff #35

Closed janbruedigam closed 2 years ago

rejuvyesh commented 2 years ago

Any estimates for difference in eval time with ForwardDiff vs FiniteDiff?

janbruedigam commented 2 years ago

Haven't checked yet in detail, feels a bit faster, but I could also be completely wrong

rejuvyesh commented 2 years ago

Julia v1.8 (beta) shows timing information in tests as well. So possibly of interest.

Current master:

Integrator              |    7      7  9.7s
  FiniteDiff comparison |    7      7  9.4s
Test Summary:                         | Pass  Total     Time
Minimal Coordinates                   |  861    861  3m43.9s
  Get and set position and velocities |  602    602    22.7s
  Minimal to maximal to minimal       |   58     58    41.5s
  Jacobians                           |  201    201  2m38.1s
Test Summary:                 | Pass  Total  Time
Modified Rodrigues Parameters |    9      9  0.8s
  FiniteDiff comparison       |    5      5  0.7s
  axes pair to quaternion     |    4      4  0.1s
Test Summary:                | Pass  Total   Time
Damper                       |  120    120  51.8s
  rotational damper jacobian |  120    120  51.7s
Test Summary:              | Pass  Total     Time
Jacobian (Solution Matrix) |   32     32  5m06.5s
  Flying                   |   16     16  3m11.5s
  In contact               |   16     16  1m54.7s
Test Summary:                              | Pass  Total      Time
Data                                       |  184    184  10m41.6s
  Get and set data                         |    8      8      6.7s
  Data Jacobian without contact and limtis |   88     88   4m13.3s
  Data Jacobian with contact and limtis    |   88     88   6m17.4s
Test Summary:              | Pass  Total   Time
Collisions                 | 1566   1566  24.5s
  Collision: Sphere-sphere | 1566   1566  24.3s
Test Summary: | Pass  Total     Time
Momentum      |   73     73  4m12.8s
  Box         |    2      2     1.1s
  Pendulum    |    1      1     1.3s
  Humanoid    |    2      2    39.4s
  Atlas       |    2      2    53.4s
  Quadruped   |    2      2  1m38.1s
  Snake       |   32     32    29.4s
  Twister     |   32     32    30.0s
Test Summary: | Pass  Total     Time
Energy        |   42     42  3m15.8s
  Dice        |    1      1     0.7s
  Pendulum    |    1      1     5.9s
  Slider 1    |    3      3     1.4s
  Slider 2    |    1      1     0.6s
  Slider 3    |    1      1     2.4s
  Humanoid    |    1      1    10.8s
  Atlas       |    1      1    56.3s
  Quadruped   |    1      1    18.6s
  Snake       |   16     16    49.9s
  Twister     |   16     16    48.8s
Test Summary:      | Pass  Total   Time
Behavior           |   22     22  51.6s
  Quadruped        |    1      1  19.2s
  Box toss         |    8      8  11.0s
  Four-bar linkage |   12     12  18.3s
  Tennis racket    |    1      1   2.9s
Test Summary: | Pass  Total  Time
Joint Limits  |    1      1  3.6s
  Pendulum    |    1      1  3.5s
Test Summary:                            | Pass  Total  Time
Impulse Map                              |   20     20  7.3s
  Displacement Jacobian                  |    4      4  1.4s
  Impulse transform Jacobian: rotational |    8      8  2.9s
  Impulse map                            |    8      8  2.9s
Test Summary:      | Pass  Total  Time
Bodies             |    1      1  1.0s
  Shape convertion |    1      1  0.9s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8700
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8701
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8702
length(joint_list) = 1
length(joint_list) = 2
length(joint_list) = 3
length(joint_list) = 4
[j.name for j = joint_list] = [:neck_ay, :back_bkx, :back_bky, :back_bkz, :floating_base]
[((get_body(mechanism, j.parent_id)).name, (get_body(mechanism, j.child_id)).name) for j = joint_list] = [(:utorso, :head), (:mtorso, :utorso), (:ltorso, :mtorso), (:pelvis, :ltorso), (:origin, :pelvis)]
Test Summary:             | Pass  Total   Time
Mechanism (Miscellaneous) |   12     12  17.4s
  Pendulum                |   12     12  17.3s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8703
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8704
Test Summary: | Pass  Total  Time
Simulate      |    6      6  0.5s
  step!       |    2      2  0.0s
  Storage     |    4      4  0.5s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8705
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8706
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8707

n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    2e+0    1e+0    0e+0    2e+0    2e+0
2    3e-1    7e-3    1e+0    5e-2    3e-1    3e-1
3    8e-2    9e-4    1e+0    5e-3    7e-2    7e-2
4    7e-3    2e-4    1e+0    1e-4    7e-3    7e-3
5    4e-4    1e-5    1e+0    1e-6    4e-4    4e-4
6    2e-5    6e-7    1e+0    4e-8    2e-5    2e-5

n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    2e+0    1e+0    0e+0    2e+0    2e+0
2    5e-1    2e-1    9e-1    5e-2    4e-1    4e-1
3    3e-1    1e-1    5e-1    7e-2    2e-1    2e-1
4    2e-1    2e-2    9e-1    3e-2    2e-1    2e-1
5    4e-2    2e-4    1e+0    4e-3    3e-2    3e-2
6    4e-3    1e-4    1e+0    5e-5    4e-3    4e-3
7    2e-4    6e-6    1e+0    6e-7    2e-4    2e-4
8    1e-5    3e-7    1e+0    2e-8    1e-5    1e-5

n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    3e+0    1e+0    0e+0    3e+0    3e+0
2    5e-1    3e-1    9e-1    4e-2    4e-1    4e-1
3    3e-1    2e-1    4e-1    8e-2    2e-1    2e-1
4    4e-1    3e-2    9e-1    5e-2    3e-1    3e-1
5    6e-2    2e-4    1e+0    8e-3    6e-2    6e-2
6    8e-3    2e-4    1e+0    1e-4    8e-3    8e-3
7    4e-4    1e-5    1e+0    2e-6    4e-4    4e-4
8    2e-5    6e-7    1e+0    5e-8    2e-5    2e-5
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8708
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8709
Test Summary:       | Pass  Total   Time
Visuals             |    3      3  11.9s
  Utilities         |    1      1   2.2s
  MeshCat mechanism |    1      1   7.1s
  URDF mesh         |    1      1   2.5s
Test Summary: | Pass  Total  Time
Utilities     |    5      5  0.0s
Test Summary: | Pass  Total  Time
Mechanisms    |   21     21  8.7s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8710
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8711
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8712
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8713
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8714
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8715
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8716
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8717
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8718
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8719
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8720
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8721
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8722
Test Summary:   | Pass  Total   Time
Environments    |   26     26  52.7s
  ant           |    2      2   3.9s
  atlas         |    2      2  10.0s
  block         |    2      2   1.3s
  block2d       |    2      2   1.5s
  cartpole      |    2      2   1.2s
  halfcheetah   |    2      2   0.9s
  hopper        |    2      2   2.7s
  panda         |    2      2  12.7s
  pendulum      |    2      2   0.3s
  quadruped     |    2      2   1.4s
  raiberthopper |    2      2   6.5s
  rexhopper     |    2      2   9.5s
  walker        |    2      2   0.5s
     Testing Dojo tests passed

This PR:

Integrator               |    7      7  9.1s
  ForwardDiff comparison |    7      7  8.8s
Test Summary:                         | Pass  Total     Time
Minimal Coordinates                   |  861    861  4m22.8s
  Get and set position and velocities |  602    602    22.9s
  Minimal to maximal to minimal       |   58     58    41.6s
  Jacobians                           |  201    201  3m16.8s
Test Summary:                 | Pass  Total  Time
Modified Rodrigues Parameters |    9      9  2.6s
  FiniteDiff comparison       |    5      5  2.5s
  axes pair to quaternion     |    4      4  0.1s
Test Summary:                | Pass  Total     Time
Damper                       |  120    120  1m30.0s
  rotational damper jacobian |  120    120  1m29.9s
Test Summary:              | Pass  Total     Time
Jacobian (Solution Matrix) |   32     32  5m15.0s
  Flying                   |   16     16  3m06.1s
  In contact               |   16     16  2m08.7s
Test Summary:                              | Pass  Total      Time
Data                                       |  184    184  11m11.2s
  Get and set data                         |    8      8      6.8s
  Data Jacobian without contact and limtis |   88     88   4m25.7s
  Data Jacobian with contact and limtis    |   88     88   6m34.3s
Test Summary:              | Pass  Total   Time
Collisions                 | 1566   1566  43.4s
  Collision: Sphere-sphere | 1566   1566  43.2s
Test Summary: | Pass  Total     Time
Momentum      |   73     73  4m32.8s
  Box         |    2      2     1.1s
  Pendulum    |    1      1     1.4s
  Humanoid    |    2      2    43.1s
  Atlas       |    2      2    56.4s
  Quadruped   |    2      2  1m46.8s
  Snake       |   32     32    31.5s
  Twister     |   32     32    32.5s
Test Summary: | Pass  Total     Time
Energy        |   42     42  3m27.5s
  Dice        |    1      1     0.8s
  Pendulum    |    1      1     6.1s
  Slider 1    |    3      3     1.5s
  Slider 2    |    1      1     0.5s
  Slider 3    |    1      1     2.6s
  Humanoid    |    1      1    11.3s
  Atlas       |    1      1    59.2s
  Quadruped   |    1      1    20.0s
  Snake       |   16     16    53.0s
  Twister     |   16     16    52.0s
Test Summary:      | Pass  Total   Time
Behavior           |   22     22  54.6s
  Quadruped        |    1      1  20.6s
  Box toss         |    8      8  11.4s
  Four-bar linkage |   12     12  19.6s
  Tennis racket    |    1      1   2.8s
Test Summary: | Pass  Total  Time
Joint Limits  |    1      1  3.6s
  Pendulum    |    1      1  3.6s
Test Summary:                            | Pass  Total     Time
Impulse Map                              |   20     20  1m05.7s
  Displacement Jacobian                  |    4      4     4.0s
  Impulse transform Jacobian: rotational |    8      8    21.1s
  Impulse map                            |    8      8    40.4s
Test Summary:      | Pass  Total  Time
Bodies             |    1      1  1.1s
  Shape convertion |    1      1  1.1s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8700
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8701
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8702
length(joint_list) = 1
length(joint_list) = 2
length(joint_list) = 3
length(joint_list) = 4
[j.name for j = joint_list] = [:neck_ay, :back_bkx, :back_bky, :back_bkz, :floating_base]
[((get_body(mechanism, j.parent_id)).name, (get_body(mechanism, j.child_id)).name) for j = joint_list] = [(:utorso, :head), (:mtorso, :utorso), (:ltorso, :mtorso), (:pelvis, :ltorso), (:origin, :pelvis)]
Test Summary:             | Pass  Total   Time
Mechanism (Miscellaneous) |   12     12  17.7s
  Pendulum                |   12     12  17.7s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8703
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8704
Test Summary: | Pass  Total  Time
Simulate      |    6      6  0.2s
  step!       |    2      2  0.0s
  Storage     |    4      4  0.2s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8705
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8706
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8707

n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    2e+0    1e+0    0e+0    2e+0    2e+0
2    3e-1    7e-3    1e+0    5e-2    3e-1    3e-1
3    8e-2    9e-4    1e+0    5e-3    7e-2    7e-2
4    7e-3    2e-4    1e+0    1e-4    7e-3    7e-3
5    4e-4    1e-5    1e+0    1e-6    4e-4    4e-4
6    2e-5    6e-7    1e+0    4e-8    2e-5    2e-5

n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    2e+0    1e+0    0e+0    2e+0    2e+0
2    5e-1    2e-1    9e-1    5e-2    4e-1    4e-1
3    3e-1    1e-1    5e-1    7e-2    2e-1    2e-1
4    2e-1    2e-2    9e-1    3e-2    2e-1    2e-1
5    4e-2    2e-4    1e+0    4e-3    3e-2    3e-2
6    4e-3    1e-4    1e+0    5e-5    4e-3    4e-3
7    2e-4    6e-6    1e+0    6e-7    2e-4    2e-4
8    1e-5    3e-7    1e+0    2e-8    1e-5    1e-5

n    bvio    rvio     α       μ     |res|∞   |Δ|∞
–––––––––––––––––––––––––––––––––––––––––––––––––
1    2e+0    3e+0    1e+0    0e+0    3e+0    3e+0
2    5e-1    3e-1    9e-1    4e-2    4e-1    4e-1
3    3e-1    2e-1    4e-1    8e-2    2e-1    2e-1
4    4e-1    3e-2    9e-1    5e-2    3e-1    3e-1
5    6e-2    2e-4    1e+0    8e-3    6e-2    6e-2
6    8e-3    2e-4    1e+0    1e-4    8e-3    8e-3
7    4e-4    1e-5    1e+0    2e-6    4e-4    4e-4
8    2e-5    6e-7    1e+0    5e-8    2e-5    2e-5
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8708
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8709
Test Summary:       | Pass  Total   Time
Visuals             |    3      3  11.7s
  Utilities         |    1      1   2.3s
  MeshCat mechanism |    1      1   7.2s
  URDF mesh         |    1      1   2.2s
Test Summary: | Pass  Total  Time
Utilities     |    5      5  0.0s
Test Summary: | Pass  Total  Time
Mechanisms    |   21     21  9.7s
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8710
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8711
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8712
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8713
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8714
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8715
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8716
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8717
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8718
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8719
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8720
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8721
┌ Info: MeshCat server started. You can open the visualizer by visiting the following URL in your browser:
└ http://127.0.0.1:8722
Test Summary:   | Pass  Total   Time
Environments    |   26     26  54.4s
  ant           |    2      2   4.0s
  atlas         |    2      2  10.7s
  block         |    2      2   1.5s
  block2d       |    2      2   1.4s
  cartpole      |    2      2   1.2s
  halfcheetah   |    2      2   0.9s
  hopper        |    2      2   2.9s
  panda         |    2      2  12.9s
  pendulum      |    2      2   0.3s
  quadruped     |    2      2   1.4s
  raiberthopper |    2      2   6.5s
  rexhopper     |    2      2   9.8s
  walker        |    2      2   0.6s
     Testing Dojo tests passed
thowell commented 2 years ago

Thanks for the PR Jan!

@rejuvyesh what is the correct way to include dependencies for the tests? I was under the impression that now there should be a test/Project.toml file: https://pkgdocs.julialang.org/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.2-and-above

Also, what are your thoughts on FiniteDiff vs ForwardDiff for the tests? Ideally we would just support one.

rejuvyesh commented 2 years ago

Test dependencies

Yeah, test/Project.toml is the way to go!

FiniteDiff vs ForwardDiff for tests

Main thing I am not fully sure about is why do you have to define custom gradient/jacobian functions if AD like ForwardDiff already works? I would assume for performance/efficiency? In that case best to compare against proper AD like ForwardDiff. However, if you see that you are defining gradients for functions where AD like ForwardDiff is not going to work, then supporting FiniteDiff makes sense.

codecov-commenter commented 2 years ago

Codecov Report

Merging #35 (104bfa7) into main (1c87f5c) will decrease coverage by 0.11%. The diff coverage is 100.00%.

:exclamation: Current head 104bfa7 differs from pull request most recent head 303c208. Consider uploading reports for the commit 303c208 to get more accurate results

@@            Coverage Diff             @@
##             main      #35      +/-   ##
==========================================
- Coverage   91.73%   91.61%   -0.12%     
==========================================
  Files          87       87              
  Lines        4477     4295     -182     
==========================================
- Hits         4107     3935     -172     
+ Misses        370      360      -10
Impacted Files Coverage Δ
src/integrators/integrator.jl 97.43% <100.00%> (ø)
src/joints/rotational/minimal.jl 100.00% <100.00%> (ø)
src/mechanism/state.jl 100.00% <100.00%> (ø)
src/graph/entry.jl 75.00% <0.00%> (-12.50%) :arrow_down:
src/contacts/utilities.jl 42.85% <0.00%> (-2.60%) :arrow_down:
src/solver/line_search.jl 95.55% <0.00%> (-1.39%) :arrow_down:
src/gradients/state.jl 88.98% <0.00%> (-0.79%) :arrow_down:
src/joints/constraints.jl 89.94% <0.00%> (-0.44%) :arrow_down:
src/mechanism/traversal.jl 95.00% <0.00%> (-0.35%) :arrow_down:
src/mechanism/urdf.jl 92.05% <0.00%> (-0.32%) :arrow_down:
... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1c87f5c...303c208. Read the comment docs.

janbruedigam commented 2 years ago

Thanks for the clarification @rejuvyesh ! I've readded the test Project.toml. Moving the remaining tests to ForwardDiff will require some additional work with rotations and mrps, so I would revisit that at a later stage. I updated the test tolerances, so if tests keep failing they can be loosened again.