osrf-migration
commented
5 years ago Open osrf-migration opened 5 years ago
osrf-migration
commented
5 years ago osrf-migration
commented
5 years ago Original comment by Diego Ferigo (Bitbucket: dgferigo).
Updated the cachegrind file with a longer acquisition
osrf-migration
commented
5 years ago Original comment by Michael Carroll (Bitbucket: Michael Carroll, GitHub: mjcarroll).
Thanks for reporting. We're currently in the process of doing some more detailed analysis of performance and eliminating bottlenecks, so this is useful information.
Just a few questions:
To confirm, you have disabled all sleep mechanisms in the Run loop so that it iterates as fast as possible?
For my benefit can you post the world that you are using somewhere?
Are you using any of the new level functionality currently?
One thing you can try since you are not using the GUI, you can disable the SceneBroadcaster system plugin, since that is only used for the GUI updates.
osrf-migration
commented
5 years ago Original comment by Diego Ferigo (Bitbucket: dgferigo).
Hi Michael, thanks for the prompt response! The project is open source, you can find it here: https://github.com/robotology/gym-ignition.
To confirm, you have disabled all sleep mechanisms in the Run loop so that it iterates as fast as possible?
I didn't add / remove any sleep wrt upstream. Did you mean anything in particular?
For my benefit can you post the world that you are using somewhere?
CartPole models, CartPole.world
Are you using any of the new level functionality currently?
I think I disabled it here
One thing you can try since you are not using the GUI, you can disable the SceneBroadcaster system plugin, since that is only used for the GUI updates.
Thank for the tip. I can actually enable / disable the GUI, and I thought that it only introduces overhead to report this issue. I can provide you another cachegrind analysis without it if needed.
osrf-migration
commented
5 years ago Original comment by Michael Carroll (Bitbucket: Michael Carroll, GitHub: mjcarroll).
Also, based on your longer log, it looks like you may be calling CreateComponent frequently in your plugin, causing the cached views to get invalidated (and potentially causing issues with physics). Can you post a snippet of your pre-update?
osrf-migration
commented
5 years ago Original comment by Diego Ferigo (Bitbucket: dgferigo).
You are right, I call it many times in here (function body IgnitionRobot::Impl::getOrCreateComponent), and a component is removed every step in PhysicsPrivate::UpdateSim. I saw the component cache in the codebase, and I wrongly thought that the component itself was somehow cached.
I'll try to optimize it e.g. by setting the force to 0 in the UpdateSim stage instead of removing its component. I'll report soon updates, thanks for your help so far.
osrf-migration
commented
5 years ago Original comment by Diego Ferigo (Bitbucket: dgferigo).
@mjcarroll As suggested I disabled all the GUI-related systems (broadcaster and user-commands) and commented out the GUI plugins (scene 3d, world control and world stats). The two cases represented by removing the component and zeroing its value did not differ significantly. The output of time is the following:
time column |
Removing the component | Zeroing its value |
|---|---|---|
| real | 0m13.390s |
0m12.988s |
| user | 0m6.054s |
0m5.465s |
| sys | 0m0.310s |
0m0.300s |
I did not do any statistical significant analysis, more or less every run has similar values.
However, this time the callgrind analysis is slightly better. I attached the 5% and 2% graphs. Now the CreateComponent call is not shown anymore and the cost of the physic engine became ~15.5% (data reads, cache misses, etc.). A little bit better but the overall overhead seems still significant.
osrf-migration
commented
5 years ago Original comment by Diego Ferigo (Bitbucket: dgferigo).
Added callgrind files after disabling the GUI plugins and preventing the component to be removed every step.
osrf-migration
commented
5 years ago Original comment by Diego Ferigo (Bitbucket: dgferigo).
Cropped the images
chapulina
commented
3 years ago I'd like to have a completion criterion for this issue. Performance is quite broad: in theory, things can always go faster, and Gazebo has lots of features that we could focus on. The conversation seems to be focused on physics, so I propose reducing the scope of this issue to answering:
Is the overhead added by Ignition on top of a physics engine unacceptable?
This question is hard enough to answer with minimal functionality, so I propose the following assumptions to get to test the core functionality, without optional features:
-z 100000000)Here's what shapes.sdf looks like with only the physics system using DART:
WorldForwardStep is basically just calling the physics engine's step function, which takes 0.111ms / 0.147ms. The overhead is 24.5% (0.036ms).
Here's what it looks like with TPE, which is much faster:
WorldForwardStep is much faster and takes 0.011ms / 0.066ms. The overhead is 83.3% (0.055ms).
stefanbuettner
commented
3 years ago Hey there,
recently I came across this cppcon talk Performance Matters by Emery Berger who talks about performance evaluation and something he calls "causal profiling". Since the 2021 Q2 focus is on performance and this issue (at least from the title) is about performance of the robotic libraries, I wanted to ask what you think about profiling Gazebo with coz to identify improvements possibilities.
I also wanted to ask how you plan to evaluate the changes of this Q2 roadmap item and if it might be worth considering Stabilizer to do so.
Cheers, Stefan
chapulina
commented
3 years ago causal profiling
Thank you for the tips, @stefanbuettner. A causal profiler looks really interesting to help us identify where to best focus our efforts.
We've been using Ignition Common's profiler wrapper around Remotery, that's what I used on the comment above. Our profiler API is meant to be abstract enough so we can plug other profilers, but it looks to me that the nature of a causal profiler is different enough that we shouldn't just wrap COZ_PROGRESS macros with the IGN_PROFILE macros that we already have across the libraries. We'd probably need to inject new macros into different places.
how you plan to evaluate the changes
We've been using the Remotery profiler as well as real time factor to evaluate changes as we make them. Stabilizer seems to offer a more rigorous approach that could be interesting.
francocipollone
commented
2 years ago We've been working on setting up coz profiler within ignition's workspace in order to analyze which advantages using this profiler could give us.
coz profiler is based on a different paradigm in comparison with regular profilers. This profiler instead of giving information about the time that several parts of a code take to be executed, it gives information about "which" code is needed to be speeded up in order to speed up the entire execution. In order to achieve this, because of the way that coz's api is used and how the profiler is set up, the current structure we have in ignition-common to allow multiple Profilers implementations isn't compatible with coz.
So, before embarking into an implementation that is able to live in tandem with the ign-common::Profiler machinery, the idea is to run an ignition simulation with coz and evaluate how beneficial could be this tool and also, to examine the code and understand where are the bottlenecks during a simulation.
After struggling a bit with the installation of coz and correctly linking against the coz library from ignition libraries I see value on commenting here the steps we found that are more straight forward in order to get it working.
docker run command adding CAP_SYS_ADMIN privilege. To do so you just have to add --cap-add=CAP_SYS_ADMIN as an argument to the docker run command.Note: In case it is helpful we tried this in a container using bionic.
coz into your colcon workspace where your ignition source files are.
cd colcon_ws/src
git clone git@github.com:plasma-umass/coz.gitNote: coz provides binaries that can be installed using apt install, however we abandoned that path mainly because:
master cmake support was added.Install coz dependencies:
sudo apt-get install build-essential cmake docutils-common libelfin-dev nodejs npm python3Compile coz package
colcon build --packages-select cozUse find_package(or the custom macro from ign-cmake) to find coz from another package, for example, from ign-gazebo package.
find_package(coz-profiler REQUIRED HINTS "${CMAKE_INSTALL_PREFIX}/lib/cmake")Note1: even though coz's cmake project is called coz the generated .cmake files calls it coz-profiler, that's why the package to find is coz-profiler and not coz. This also "confuses" colcon when checking the dependencies between the packages in the src folder.
Note2: A hint must be passed to find the coz-profiler-config.cmake files given that the CMakeFiles from coz doesn't install the .cmake files under install/lib/cmake/coz as expected.
DWARF error when running a simulation with the coz profiler. So tipically:
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -gdwarf-3")Note: There isn't clear information about this issue in the coz repository, but comments of people through issues where different things are suggested to fix that. It seems that it is a bug that comes from one of the coz dependencies: libelfin
coz within a compilation unit the header file should be included to the target. The coz.h header file is installed at "install/include" folder and not at "install/include/coz" so the directory that needs to be included is ${CMAKE_INSTALL_PREFIX}/include>
example:
target_include_directories(${PROJECT_LIBRARY_TARGET_NAME}
PUBLIC
...
$<BUILD_INTERFACE:${CMAKE_INSTALL_PREFIX}/include>
)
8. Now `#include <coz.h>` can be added to your code, program, etc, as well as the `coz` macros like `COZ_PROGRESS`
9. After compiling your package under analysis(like ign-gazebo). You can run coz doing:
```bash
#remember to source install to also get the coz command
source install/setup.bash
coz run --- ./<path_to_your_executable/program/simulation>
#use sudo and the entire path to coz if needed.eg.:
sudo ./install/bin/coz run --- ./<path_to_your_executable/program/simulation>a profiler.coz file is generated that can be loaded from https://plasma-umass.org/coz/ in order to visualize the output of the profiler.
Note: As you may noticed it, we haven't linked against coz because when doing that the executable is automatically run using coz and then you miss the chance to run your executable without coz.
progress points and other coz features within the ignition code. chapulina
commented
2 years ago Great work, @francocipollone ! It's unfortunate that you had to jump through so many hoops to get started using coz. Looking forward to seeing what the results can tell us about where to focus in the future!
francocipollone
commented
2 years ago Some updates on this front
coz profiler lets you select one or several "progress points", these are points in the code that you would like to improve and when you run the profiler, coz inspects the code and provides information about which line or entity you have to improve in order to get better performance at the "progress points" before selected.
To start, we set a COZ_PROGRESS macro (progress point)(defined at <coz.h> header file is mandatory) at the end of the "SimulationRunner::Step" method, in order to get information about how it performs and what should improve to smooth its execution (including <coz.h> header file). The idea was to tune or adjust where to set progress points as we got more used to using the tool.
coz and dl as recommended by documentation. (Documentation isn't accurate about the name of the modules to link to though)As coz documentation suggests, sources must be compiled as Debug, so we followed that recommendation when compiling ignition gazebo.(And its dependencies if needed)
We created an executable that in essence set up a gazebo::Server, loads a sdf and run the simulation a finite number of iterations and then the program ends. Quite simple.
Then we ran coz doing:
sudo ./install/bin/coz run --- ./install/bin/simulationWhile running, information is being dumped to a profiler.coz file, that later on, you can load as explained in item 9. in the previous post.
When we plot the dumped information no interesting results are given because no information about ignition files are provided.
It is like if coz isn't capable of traverse ignition files.
Only some references related to the cpp stl
During this run, coz logs information. Here we can see that coz/libcoz/inspect.cpp:509 is inspecting the dependencies files that will be used to perform the analysis,
Effectively, there are no ignition-related files being inspected (but some ignition math files )
I've checked the coz/libcoz/inspect.cc code in charge of including source file to the analysis and it seems that it is including files referred in the compilation unit.
So tried another thing. Instead of creating an executable that links to ignition-gazebo core library I added all the sources of ignition-gazebo to the executable and created the executable again. I know that this doesn't scale and it isn't straightforward to do, but it was worth trying in order to keep digging into the tool. And this time files were inspected:
(I show just some lines to avoid too much noise)
[/ws/src/coz/libcoz/inspect.cpp:509] Included source file /ws/src/ign-gazebo/include/ignition/gazebo/EventManager.hh
[/ws/src/coz/libcoz/inspect.cpp:509] Included source file /ws/src/ign-gazebo/include/ignition/gazebo/components/Component.hh
[/ws/src/coz/libcoz/inspect.cpp:509] Included source file /ws/src/ign-gazebo/include/ignition/gazebo/components/Factory.hh
[/ws/src/coz/libcoz/inspect.cpp:509] Included source file /ws/src/ign-gazebo/include/ignition/gazebo/components/ParentEntity.hh
[/ws/src/coz/libcoz/inspect.cpp:509] Included source file /ws/src/ign-gazebo/include/ignition/gazebo/detail/ComponentStorageBase.hh
[/ws/src/coz/libcoz/inspect.cpp:509] Included source file /ws/src/ign-gazebo/include/ignition/gazebo/detail/EntityComponentManager.hh
[/ws/src/coz/libcoz/inspect.cpp:316] Including lines from executable /ws/install/bin/simulationHowever, this run ended up aborted so the profiler.coz file wasn't created.
[libprotobuf ERROR google/protobuf/descriptor_database.cc:57] File already exists in database: peer_info.proto
[libprotobuf FATAL google/protobuf/descriptor.cc:1164] CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of 'google::protobuf::FatalException'
what(): CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
Aborted!
Inspection of files. Including all the sources when creating the executable doesn't scale. It was an alternative just to try ignition-gazebo but of course, we would like to have, e.g., rendering and physics being part of the analysis. So at this point, I wonder if there is something I am missing when building or related to coz in order to have more symbols exported that can be analyzed by coz.
coz provides. It analyses sqlite performance, and the library is compiled with the executable(Check Cmake file): The run ended up with a failure at libprotobuf, I think this is caused by coz given that coz performs the analysis by running it several times and creating deadtimes in between at different parts of the code.
I will try to dig a bit deeper into this to see if I can get a real profile.
Original report (archived issue) by Diego Ferigo (Bitbucket: dgferigo).
The original report had attachments: cachegrind.png, callgrind_2.png, callgrind_5.png
Summary
The overhead introduced by ignition gazebo seems significant. The time spent to compute physics is a small fraction of the total time.
Description
First off, thanks for all the work you're doing! Despite their early development status, ignition libraries are amazing.
I am currently developing an application based on ignition robotics. Starting by saying that it is still at a very early stage of development, I tried to evaluate its performance in order to catch initial bottlenecks. The test I did is a simple cart pole simulation with ignition configured in such a way that it tries to go as fast as it can. Currently, there's only one thread, and I can see that the CPU goes to 100% when the application is executing.
Referring to the attached cachegrind image, I am surprised to see that only the ~11% of the time passed inside the ignition libraries (the entry point is Server::Run) is actually spent in the physics engine (currently, dart). This means that the remaining 89% is computation overhead.
Is this behavior normal to your eyes? I saw that you performed in the past some profiling, maybe you have some insights on it.
Note: this is a headless simulation, I do not use the
ign-gazebo-guiprocess