Create a standard benchmark that can be used to validate speedups

[tfoote]

Have a standard script/forumula for running the benchmark.

Repos files for each configuration

Standard table for how long sensor_msgs package takes to build, and how long overall build of up to sensor_msgs takes.

Potentially different configurations with different RMWs enabled.

In support of #17 justification.

Test cases in benchmark:

• Baseline:
• Baseline + upstream improvement proposal (expected faster justify merging upstream)
• Baseline + Protobuf current implementation (expected slower show how much)
• Baseline + Upstream improvements + Protobuf (Goal show that it's lower than baseline alone)

[tfoote]

Maybe a standalone repo with scripts that helps others to reproduce the builds.

Runs builds sequentially and parameterized options. Such as which implementation.

Disable testing for benchmarking, to avoid overhead of building and validating.

[gonzodepedro]

Get full build time Get sensor_msgs time

[tfoote]

@gonzodepedro From your gist I've created: https://github.com/tfoote/build_benchmark

-----------------                                                                                       
Benchmarking repos file: baseline.repos                                                                 
-----------------                                                                                       
Running benchmark: 1                                                                                    
Cleaning and rebuilding in /tmp/benchmark/baseline.repos-src                                            
Finished <<< sensor_msgs [46.0s]                                                                        
Summary: 106 packages finished [5min 55s]                                                               
-----------------                                                                                       
Benchmarking repos file: streamlined.repos                                                              
-----------------                                                                                       
Running benchmark: 1                                                                                    
Cleaning and rebuilding in /tmp/benchmark/streamlined.repos-src                                         
Finished <<< sensor_msgs [39.3s]                                                                        
Summary: 106 packages finished [5min 26s]                                                               
-----------------                                                                                       
Benchmarking repos file: protobuf.repos                                                                 
-----------------                                                                                       
Running benchmark: 1                                                                                    
Cleaning and rebuilding in /tmp/benchmark/protobuf.repos-src                                            
Finished <<< sensor_msgs [1min 21s]                                                                     
Summary: 110 packages finished [9min 42s]                                                               
-----------------                                                                                       
Benchmarking repos file: streamlined_protobuf.repos                                                     
-----------------   
Running benchmark: 1                                                                                    
Cleaning and rebuilding in /tmp/benchmark/streamlined_protobuf.repos-src                                
Finished <<< sensor_msgs [2min 13s]                                                                     
Summary: 110 packages finished [9min 52s]   

I'm clearly seeing the speedup for the work from https://github.com/ros2/rosidl/pull/769 but the protobuf elements are definitely slowing things down still. And with the streamlining it surprisingly goes slower. Can you double check my repos files?

[gonzodepedro]

To instrument code using time:

rosidl:

diff --git a/rosidl_cmake/cmake/rosidl_generate_interfaces.cmake b/rosidl_cmake/cmake/rosidl_generate_interfaces.cmake
index 5ced5dd..a25bacb 100644
--- a/rosidl_cmake/cmake/rosidl_generate_interfaces.cmake
+++ b/rosidl_cmake/cmake/rosidl_generate_interfaces.cmake
@@ -289,7 +289,7 @@ macro(rosidl_generate_interfaces target)
   # Generate the interface source files for all registered languages and type supports
   set(rosidl_cmake_generate_interfaces_BIN "${rosidl_cmake_DIR}/../../../lib/rosidl_cmake/rosidl_cmake_generate_interfaces")
   execute_process(
-    COMMAND ${Python3_EXECUTABLE} ${rosidl_cmake_generate_interfaces_BIN}
+    COMMAND /usr/bin/time -f %e\ %C -o /tmp/benchmark-templates -a ${Python3_EXECUTABLE} ${rosidl_cmake_generate_interfaces_BIN}
     --generator-arguments-files "${rosidl_generator_arguments_files}"
     ECHO_OUTPUT_VARIABLE
   )

rosidl_typesupport_protobuf

diff --git a/rosidl_adapter_proto/cmake/rosidl_adapt_proto_interfaces.cmake b/rosidl_adapter_proto/cmake/rosidl_adapt_proto_interfaces.cmake
index b5d5a9a..b27bdbd 100644
--- a/rosidl_adapter_proto/cmake/rosidl_adapt_proto_interfaces.cmake
+++ b/rosidl_adapter_proto/cmake/rosidl_adapt_proto_interfaces.cmake
@@ -54,7 +54,7 @@ rosidl_write_generator_arguments(
   ADDITIONAL_FILES "${_proto_include_dirs}")

 execute_process(
-  COMMAND "${PYTHON_EXECUTABLE}" "${rosidl_adapter_proto_BIN}"
+  COMMAND /usr/bin/time -f %e\ %C -o /tmp/benchmark-proto -a "${PYTHON_EXECUTABLE}" "${rosidl_adapter_proto_BIN}"
   --generator-arguments-file "${generator_arguments_file}"
   --protoc-path "${Protobuf_PROTOC_EXECUTABLE}"
   ERROR_VARIABLE error