search

gazebo-tooling / release-tools

8 stars 9 forks source link

Generate core dumps for tests that segfault #1030

Open azeey opened 9 months ago

azeey commented 9 months ago

This would help in debugging tests that crash on CI, but not locally. The docker flags are based on https://ddanilov.me/how-to-configure-core-dump-in-docker-container

@j-rivero @nuclearsandwich is this a potential security risk? I didn't think it was since the tests are running inside docker and AFAIK the workspace is not accessible by users that are not logged in to Jenkins, but I wanted to confirm with y'all.

j-rivero commented 8 months ago

@j-rivero @nuclearsandwich is this a potential security risk? I didn't think it was since the tests are running inside docker and AFAIK the workspace is not accessible by users that are not logged in to Jenkins, but I wanted to confirm with y'all.

Discussed in the infra meeting. The public publication of the core dump file might be a security risk given the information dumped in these files. You are right about that we protect the workspace of Jenkins to be read only by admins, buildfarmers and github osrf/employees group (to be updated) so technically it could be safe. If other external contributors need access to them, they should be manually inspected first. Said all this, we believe that the best approach would be to have a reproducible build (see #923) but there is not ETA when that will be done.

Another thing to consider from the core dumps is that they are quite big, several Gbs usually. If we enable the generation of them automatically, we could have problems with disk space in nodes. How about introducing a parameter to activate the core dump exportation?

j-rivero commented 7 months ago

Spoken with @azeey offline, we agreed that exporting the core dump file without #923 implemented would have little interested since reproducibility would be hard to obtain. A possible path forward could be to run the gdb in-the-build and export a stacktrace of it.