Added arm benchmark + improved running process of benchmarks - [merged] #574

Closed wxmerkt closed 4 years ago

In GitLab by @cmastalli on Oct 3, 2019, 10:53

Merges topic/arm-benchmark -> devel

This PR mainly tackles #259. Additionally it included the following improvements in the benchmark

Added ShootingProblem.calc and ShootingProblem.calcDiff benchmarks for all the cases.
Added pure Python benchmark for the talos arm problem

Improved the running process, now we can do it like this:

make benchmark-unicycle INPUT=100 # 100 is the number of trials
make benchmark-lqr INPUT=100 # 100 is the number of trials
make benchmark-arm-manipulator INPUT=100 # 100 is the number of trials
make benchmark-quadrupedal-gaits INPUT='100 walk' # walk is the gait, we have other options (trot, pace, bound and jump)
make benchmark-bipedal-walk INPUT=100 # 100 is the number of trials

Additionally, I don't see any breaking of the determinism (neither c++ and Python bindings code) using multithreading as reported #256. Indeed, I notice an improvement in the computation time of calcDiff. These are the results

C++:
  DDP.solve [ms]: 1.59551 (1.55648-1.94576)
  ShootingProblem.calc [ms]: 0.46114 (0.43442-0.676178)
  ShootingProblem.calcDiff [ms]: 0.560603 (0.497396-0.984347)
Python bindings:
  DDP.solve [ms]: 1.69171571732 (1.63292884827, 2.27212905884)
  ShootingProblem.calc [ms]: 0.49816608429 (0.463008880615, 0.734090805054)
  ShootingProblem.calcDiff [ms]: 0.578708648682 (0.531911849976, 1.02019309998)

and without multithreading we compute calcDiff around 2.5ms.

@wxmerkt could you run the arm-manipulation benchmark in your PC?

@proyan it's working on the c++ benchmark for quadrupedal-gaits

In GitLab by @cmastalli on Oct 3, 2019, 10:53

changed the description

In GitLab by @cmastalli on Oct 3, 2019, 10:54

enabled an automatic merge when the pipeline for f57fad10aefdd1756add9fcbf11b22af262ebf5f succeeds

In GitLab by @wxmerkt on Oct 3, 2019, 11:07

Benchmark arm manipulator:

Single core, vectorisation OFF:

C++:
  DDP.solve [ms]: 2.41596 (2.35941-3.28919)
  ShootingProblem.calc [ms]: 0.383199 (0.366688-0.688599)
  ShootingProblem.calcDiff [ms]: 1.60558 (1.51536-2.78337)
Python bindings:
  DDP.solve [ms]: 2.49390125275 (2.40015983582, 3.60989570618)
  ShootingProblem.calc [ms]: 0.420245409012 (0.399827957153, 0.806093215942)
  ShootingProblem.calcDiff [ms]: 1.60745024681 (1.53994560242, 2.4881362915)

Multi-threading (8 cores), vectorisation ON:

C++:
  DDP.solve [ms]: 1.19437 (1.16402-2.00468)
  ShootingProblem.calc [ms]: 0.371434 (0.363886-0.558771)
  ShootingProblem.calcDiff [ms]: 0.263641 (0.238182-0.519628)
Python bindings:
  DDP.solve [ms]: 1.23634848595 (1.20091438293, 2.17580795288)
  ShootingProblem.calc [ms]: 0.408774662018 (0.392913818359, 0.766038894653)
  ShootingProblem.calcDiff [ms]: 0.29428730011 (0.269889831543, 1.18613243103)

Via the Python bindings, I still see the non-determinism as reported in #256:

[DDP] Mean iter 29.1 +/- 8.757282683572571
[FDDP] Mean iter 31.2 +/- 9.88736567544662

In GitLab by @cmastalli on Oct 3, 2019, 11:14

merged

In GitLab by @cmastalli on Oct 3, 2019, 11:14

mentioned in commit e4b352fc586731f32ab9dead5a0f4b5d76006be9

In GitLab by @cmastalli on Oct 3, 2019, 12:03

This is weird, let me share my results. First I define enable CALLBACKS and increase the MAXITER to 100 in both c++ and Python code. If I run 3 times the problem, i.e.

make benchmark-arm-manipulation INPUT=3

with multithreading I got the follows:

C++:
iter     cost         stop      grad      xreg        ureg   step    feas
   0  1.28603e+00  8.98541e-04  8.86607e-03  1.00000e-02  1.00000e-02   1.0000     1
   1  1.28048e+00  7.61681e-05  5.56855e-03  1.00000e-03  1.00000e-03   1.0000     1
   2  1.22992e+00  6.95954e-05  5.18180e-02  1.00000e-04  1.00000e-04   1.0000     1
   3  9.49337e-01  3.94974e-05  3.23824e-01  1.00000e-05  1.00000e-05   1.0000     1
   4  6.36857e-01  5.62421e-06  5.05724e-01  1.00000e-06  1.00000e-06   1.0000     1
   5  5.86612e-01  8.90989e-08  6.30066e-02  1.00000e-07  1.00000e-07   1.0000     1
   6  5.73587e-01  5.20957e-08  2.46287e-01  1.00000e-07  1.00000e-07   0.5000     1
   7  3.93498e-01  1.07588e-07  5.16403e-01  1.00000e-07  1.00000e-07   0.5000     1
   8  2.84041e-01  6.78601e-08  3.23259e-01  1.00000e-08  1.00000e-08   1.0000     1
   9  2.11632e-01  2.32455e-08  1.42405e-01  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
  10  2.06883e-01  3.06708e-09  8.48569e-03  1.00000e-09  1.00000e-09   1.0000     1
  11  2.03141e-01  6.43768e-10  5.64872e-03  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
   0  1.28490e+00  9.40388e-03  9.35731e-02  1.00000e-02  1.00000e-02   1.0000     1
   1  1.27935e+00  7.61284e-05  5.56433e-03  1.00000e-03  1.00000e-03   1.0000     1
   2  1.22884e+00  6.95553e-05  5.17755e-02  1.00000e-04  1.00000e-04   1.0000     1
   3  9.48645e-01  3.94576e-05  3.23421e-01  1.00000e-05  1.00000e-05   1.0000     1
   4  6.36895e-01  5.61258e-06  5.04594e-01  1.00000e-06  1.00000e-06   1.0000     1
   5  5.86824e-01  8.88515e-08  6.28007e-02  1.00000e-07  1.00000e-07   1.0000     1
   6  5.76305e-01  5.20119e-08  2.45899e-01  1.00000e-07  1.00000e-07   0.5000     1
   7  3.95784e-01  1.08165e-07  5.18802e-01  1.00000e-07  1.00000e-07   0.5000     1
   8  2.87702e-01  6.90092e-08  3.28318e-01  1.00000e-08  1.00000e-08   1.0000     1
   9  2.12212e-01  2.53020e-08  1.49868e-01  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
  10  2.07222e-01  3.77350e-09  1.02351e-02  1.00000e-09  1.00000e-09   1.0000     1
  11  2.03666e-01  6.90398e-10  5.85228e-03  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
   0  1.28491e+00  7.67055e-05  5.59980e-04  1.00000e-02  1.00000e-02   1.0000     1
   1  1.27937e+00  7.59627e-05  5.55662e-03  1.00000e-03  1.00000e-03   1.0000     1
   2  1.22891e+00  6.94148e-05  5.17115e-02  1.00000e-04  1.00000e-04   1.0000     1
   3  9.48794e-01  3.94142e-05  3.23277e-01  1.00000e-05  1.00000e-05   1.0000     1
   4  6.36792e-01  5.61467e-06  5.04955e-01  1.00000e-06  1.00000e-06   1.0000     1
   5  5.86593e-01  8.89453e-08  6.29295e-02  1.00000e-07  1.00000e-07   1.0000     1
   6  5.72328e-01  5.20402e-08  2.46027e-01  1.00000e-07  1.00000e-07   0.5000     1
   7  3.92730e-01  1.07011e-07  5.13756e-01  1.00000e-07  1.00000e-07   0.5000     1
   8  2.86257e-01  6.76115e-08  3.22182e-01  1.00000e-08  1.00000e-08   1.0000     1
   9  2.11765e-01  2.41340e-08  1.47058e-01  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
  10  2.06750e-01  3.30319e-09  9.21792e-03  1.00000e-09  1.00000e-09   1.0000     1
  11  2.03080e-01  6.25725e-10  5.64129e-03  1.00000e-09  1.00000e-09   1.0000     1
  DDP.solve [ms]: 24.9580 (23.8039-25.8804)
  ShootingProblem.calc [ms]: 0.4721 (0.4691-0.4780)
  ShootingProblem.calcDiff [ms]: 0.6864 (0.5336-0.8559)
Python bindings:
iter     cost         stop      grad      xreg        ureg   step    feas
   0  1.28714e+00  9.47981e-03  9.42569e-02  1.00000e-02  1.00000e-02   1.0000     1
   1  1.28158e+00  7.64231e-05  5.58269e-03  1.00000e-03  1.00000e-03   1.0000     1
   2  1.23090e+00  6.98180e-05  5.19426e-02  1.00000e-04  1.00000e-04   1.0000     1
   3  9.49840e-01  3.95927e-05  3.24402e-01  1.00000e-05  1.00000e-05   1.0000     1
   4  6.36953e-01  5.63306e-06  5.06374e-01  1.00000e-06  1.00000e-06   1.0000     1
   5  5.86666e-01  8.92468e-08  6.30676e-02  1.00000e-07  1.00000e-07   1.0000     1
   6  5.76593e-01  5.22254e-08  2.46903e-01  1.00000e-07  1.00000e-07   0.5000     1
   7  3.94297e-01  1.08427e-07  5.19873e-01  1.00000e-07  1.00000e-07   0.5000     1
   8  2.84593e-01  6.84376e-08  3.25179e-01  1.00000e-08  1.00000e-08   1.0000     1
   9  2.11956e-01  2.37738e-08  1.43652e-01  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
  10  2.06932e-01  3.58002e-09  9.75465e-03  1.00000e-09  1.00000e-09   1.0000     1
  11  2.03411e-01  7.47390e-10  6.08144e-03  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
   0  1.28641e+00  4.15559e-03  4.12213e-02  1.00000e-02  1.00000e-02   1.0000     1
   1  1.28086e+00  7.62139e-05  5.57207e-03  1.00000e-03  1.00000e-03   1.0000     1
   2  1.23027e+00  6.96366e-05  5.18501e-02  1.00000e-04  1.00000e-04   1.0000     1
   3  9.49535e-01  3.95185e-05  3.24003e-01  1.00000e-05  1.00000e-05   1.0000     1
   4  6.36916e-01  5.62667e-06  5.05939e-01  1.00000e-06  1.00000e-06   1.0000     1
   5  5.86693e-01  8.91215e-08  6.29960e-02  1.00000e-07  1.00000e-07   1.0000     1
   6  5.75711e-01  5.22900e-08  2.47169e-01  1.00000e-07  1.00000e-07   0.5000     1
   7  3.93556e-01  1.08163e-07  5.18823e-01  1.00000e-07  1.00000e-07   0.5000     1
   8  2.85756e-01  6.81267e-08  3.24017e-01  1.00000e-08  1.00000e-08   1.0000     1
   9  2.12187e-01  2.42087e-08  1.45745e-01  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
  10  2.06754e-01  3.61680e-09  9.74572e-03  1.00000e-09  1.00000e-09   1.0000     1
  11  2.03314e-01  6.32650e-10  5.56694e-03  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
   0  1.28263e+00  1.22967e-02  1.22209e-01  1.00000e-02  1.00000e-02   1.0000     1
   1  1.27710e+00  7.58199e-05  5.54478e-03  1.00000e-03  1.00000e-03   1.0000     1
   2  1.22676e+00  6.92802e-05  5.15979e-02  1.00000e-04  1.00000e-04   1.0000     1
   3  9.47456e-01  3.93185e-05  3.22404e-01  1.00000e-05  1.00000e-05   1.0000     1
   4  6.36778e-01  5.59272e-06  5.02885e-01  1.00000e-06  1.00000e-06   1.0000     1
   5  5.86873e-01  8.84631e-08  6.25558e-02  1.00000e-07  1.00000e-07   1.0000     1
   6  5.74844e-01  5.18470e-08  2.45146e-01  1.00000e-07  1.00000e-07   0.5000     1
   7  3.95546e-01  1.07347e-07  5.15130e-01  1.00000e-07  1.00000e-07   0.5000     1
   8  2.89556e-01  6.86886e-08  3.27374e-01  1.00000e-08  1.00000e-08   1.0000     1
   9  2.12314e-01  2.58977e-08  1.52984e-01  1.00000e-09  1.00000e-09   1.0000     1
iter     cost         stop      grad      xreg        ureg   step    feas
  10  2.06840e-01  3.85094e-09  1.01264e-02  1.00000e-09  1.00000e-09   1.0000     1
  11  2.03282e-01  6.40254e-10  5.60457e-03  1.00000e-09  1.00000e-09   1.0000     1
  DDP.solve [ms]: 25.8808930715 (25.5718231201, 26.4348983765)
  ShootingProblem.calc [ms]: 0.78296661377 (0.75888633728, 0.81205368042)
  ShootingProblem.calcDiff [ms]: 0.946283340454 (0.884056091309, 1.060962677)

As you can see there are tiny differences in the various values for each trial and for c++ vs Python code. Note that I still don't understand why we have tiny differences but it doesn't necessary has to be a problem.

Could you run in your PC this code with the same setup?

In GitLab by @cmastalli on Oct 3, 2019, 12:03

resolved all threads

loco-3d / crocoddyl

Added arm benchmark + improved running process of benchmarks - [merged] #574