search

gazebosim / gazebo-classic

Gazebo classic. For the latest version, see https://github.com/gazebosim/gz-sim
http://classic.gazebosim.org/
Other
1.19k stars 480 forks source link

UNIT_ImagesView_TEST failures since pull request #921 #1021

Open osrf-migration opened 10 years ago

osrf-migration commented 10 years ago

Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

In my overnight testing, the UNIT_ImagesView_TEST is now failing about 1/3 of the time since we merged pull request #921. Last night it failed 8/25 times:

Backtrace 1

75: Test command: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST "-xml" "-o" "/tmp/gazebo_build/source/build/test_results/UNIT_ImagesView_TEST.xml"
75: Test timeout computed to be: 240
75: SharePercentChange[0.64928] ShareMaxPercentChange[1]
75: ResPercentChange[1.61581]ResMaxPercentChange[3]
75: SharePercentChange[-0.00075815] ShareMaxPercentChange[1]
75: ResPercentChange[0.201805]ResMaxPercentChange[3]
75: SharePercentChange[0.141458] ShareMaxPercentChange[1]
75: ResPercentChange[1.02971]ResMaxPercentChange[3]
75: *** glibc detected *** /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST: double free or corruption (fasttop): 0x00002b7298013940 ***
75: ======= Backtrace: =========
75: /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x2b7216e43b96]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(+0x3e5573)[0x2b721512d573]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(_ZN14QWidgetPrivate16syncBackingStoreEv+0x80)[0x2b7214f547f0]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(_ZN7QWidget5eventEP6QEvent+0xddc)[0x2b7214f63e4c]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(_ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent+0xac)[0x2b7214f13e9c]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(_ZN12QApplication6notifyEP7QObjectP6QEvent+0x13a)[0x2b7214f1830a]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN16QCoreApplication14notifyInternalEP7QObjectP6QEvent+0x8e)[0x2b7215b6c56e]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN23QCoreApplicationPrivate16sendPostedEventsEP7QObjectiP11QThreadData+0x421)[0x2b7215b703f1]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(+0x1a6a63)[0x2b7215b9aa63]
75: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_dispatch+0x135)[0x2b7219a9fab5]
75: /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x47de8)[0x2b7219a9fde8]
75: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_iteration+0x34)[0x2b7219a9fea4]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE+0x66)[0x2b7215b9abf6]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(+0x270c1e)[0x2b7214fb8c1e]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN16QCoreApplication13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE+0x2f)[0x2b7215b7089f]
75: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST(_ZN15ImagesView_TEST8SetTopicEPN6gazebo3gui10ImagesViewERKSsi+0x14b)[0x41201b]
75: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST(_ZN15ImagesView_TEST6SwitchEv+0x30e)[0x4123de]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZNK11QMetaMethod6invokeEP7QObjectN2Qt14ConnectionTypeE22QGenericReturnArgument16QGenericArgumentS5_S5_S5_S5_S5_S5_S5_S5_S5_+0x557)[0x2b7215b74987]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN11QMetaObject12invokeMethodEP7QObjectPKcN2Qt14ConnectionTypeE22QGenericReturnArgument16QGenericArgumentS7_S7_S7_S7_S7_S7_S7_S7_S7_+0x4ac)[0x2b7215b76d5c]
75: /usr/lib/x86_64-linux-gnu/libQtTest.so.4(+0xc1c2)[0x2b7214b2f1c2]
75: /usr/lib/x86_64-linux-gnu/libQtTest.so.4(_ZN5QTest5qExecEP7QObjectiPPc+0x557)[0x2b7214b30057]
75: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST(main+0x4e)[0x40e7be]
75: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x2b7216de676d]
75: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST[0x40f8e5]

Backtrace 2

75: Test command: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST "-xml" "-o" "/tmp/gazebo_build/source/build/test_results/UNIT_ImagesView_TEST.xml"
75: Test timeout computed to be: 240
75: SharePercentChange[0.64928] ShareMaxPercentChange[1]
75: ResPercentChange[1.61732]ResMaxPercentChange[3]
75: SharePercentChange[-0.0336092] ShareMaxPercentChange[1]
75: ResPercentChange[0.167748]ResMaxPercentChange[3]
75: *** glibc detected *** /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST: double free or corruption (fasttop): 0x00002b95bc016870 ***
75: ======= Backtrace: =========
75: /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x2b951e9c3b96]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(+0x3e5573)[0x2b951ccad573]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(_ZN14QWidgetPrivate16syncBackingStoreEv+0x80)[0x2b951cad47f0]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(_ZN7QWidget5eventEP6QEvent+0xddc)[0x2b951cae3e4c]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(_ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent+0xac)[0x2b951ca93e9c]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(_ZN12QApplication6notifyEP7QObjectP6QEvent+0x13a)[0x2b951ca9830a]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN16QCoreApplication14notifyInternalEP7QObjectP6QEvent+0x8e)[0x2b951d6ec56e]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN23QCoreApplicationPrivate16sendPostedEventsEP7QObjectiP11QThreadData+0x421)[0x2b951d6f03f1]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(+0x1a6a63)[0x2b951d71aa63]
75: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_dispatch+0x135)[0x2b952161fab5]
75: /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x47de8)[0x2b952161fde8]
75: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_iteration+0x34)[0x2b952161fea4]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE+0x66)[0x2b951d71abf6]
75: /usr/lib/x86_64-linux-gnu/libQtGui.so.4(+0x270c1e)[0x2b951cb38c1e]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN16QCoreApplication13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE+0x2f)[0x2b951d6f089f]
75: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST(_ZN15ImagesView_TEST6SwitchEv+0xa15)[0x412ae5]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZNK11QMetaMethod6invokeEP7QObjectN2Qt14ConnectionTypeE22QGenericReturnArgument16QGenericArgumentS5_S5_S5_S5_S5_S5_S5_S5_S5_+0x557)[0x2b951d6f4987]
75: /usr/lib/x86_64-linux-gnu/libQtCore.so.4(_ZN11QMetaObject12invokeMethodEP7QObjectPKcN2Qt14ConnectionTypeE22QGenericReturnArgument16QGenericArgumentS7_S7_S7_S7_S7_S7_S7_S7_S7_+0x4ac)[0x2b951d6f6d5c]
75: /usr/lib/x86_64-linux-gnu/libQtTest.so.4(+0xc1c2)[0x2b951c6af1c2]
75: /usr/lib/x86_64-linux-gnu/libQtTest.so.4(_ZN5QTest5qExecEP7QObjectiPPc+0x557)[0x2b951c6b0057]
75: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST(main+0x4e)[0x40e7be]
75: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x2b951e96676d]
75: /tmp/gazebo_build/source/build/gazebo/gui/viewers/UNIT_ImagesView_TEST[0x40f8e5]
osrf-migration commented 10 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

osrf-migration commented 10 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

Timeout failure on Jenkins

77: Test command: /var/lib/jenkins/workspace/gazebo-default-devel-quantal-amd64-gpu-nvidia/build/gazebo/gui/viewers/UNIT_ImagesView_TEST "-xml" "-o" "/var/lib/jenkins/workspace/gazebo-default-devel-quantal-amd64-gpu-nvidia/build/test_results/UNIT_ImagesView_TEST.xml"
77: Test timeout computed to be: 240
77: SharePercentChange[0.626495] ShareMaxPercentChange[1]
77: ResPercentChange[1.15663]ResMaxPercentChange[3]
77: SharePercentChange[-0.0350342] ShareMaxPercentChange[1]
77: ResPercentChange[0.179993]ResMaxPercentChange[3]
 77/128 Test  #77: UNIT_ImagesView_TEST ...................***Timeout 240.09 sec
osrf-migration commented 10 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

I tried valgrind --tool=memcheck and it's found some invalid memory accesses, though I'm not sure if they're connected to this test failure:

$ valgrind -v --tool=callgrind gazebo/gui/viewers/UNIT_ImagesView_TEST
...
==3683== Thread 16:
==3683== Invalid write of size 4
==3683==    at 0x8EB85C9: gazebo::transport::Publisher::SendMessage() (Publisher.cc:187)
==3683==    by 0x8EAE935: gazebo::transport::Node::ProcessPublishers() (Node.cc:152)
==3683==    by 0x8EBA702: gazebo::transport::TopicManager::ProcessNodes(bool) (TopicManager.cc:155)
==3683==    by 0x8EA63E8: gazebo::transport::ConnectionManager::RunUpdate() (ConnectionManager.cc:276)
==3683==    by 0x8EA6D15: gazebo::transport::ConnectionManager::Run() (ConnectionManager.cc:307)
==3683==    by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683==    by 0x65C2E99: start_thread (pthread_create.c:308)
==3683==    by 0x8B713FC: clone (clone.S:112)
==3683==  Address 0x3c75a754 is 36 bytes inside a block of size 40 free'd
==3683==    at 0x4C2A44B: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3683==    by 0x8EB92A7: std::_Rb_tree<unsigned int, std::pair<unsigned int const, int>, std::_Select1st<std::pair<unsigned int const, int> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, int> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<unsigned int const, int> >) (new_allocator.h:100)
==3683==    by 0x8EB824B: gazebo::transport::Publisher::OnPublishComplete(unsigned int) (stl_tree.h:783)
==3683==    by 0x8E9DBE7: boost::function1<void, unsigned int>::operator()(unsigned int) const (function_template.hpp:760)
==3683==    by 0x8EB4310: gazebo::transport::Publication::Publish(boost::shared_ptr<google::protobuf::Message>, boost::function<void ()(unsigned int)>, unsigned int) (Publication.cc:334)
==3683==    by 0x8EB85C8: gazebo::transport::Publisher::SendMessage() (Publisher.cc:187)
==3683==    by 0x8EAE935: gazebo::transport::Node::ProcessPublishers() (Node.cc:152)
==3683==    by 0x8EBA702: gazebo::transport::TopicManager::ProcessNodes(bool) (TopicManager.cc:155)
==3683==    by 0x8EA63E8: gazebo::transport::ConnectionManager::RunUpdate() (ConnectionManager.cc:276)
==3683==    by 0x8EA6D15: gazebo::transport::ConnectionManager::Run() (ConnectionManager.cc:307)
==3683==    by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683==    by 0x65C2E99: start_thread (pthread_create.c:308)
==3683== 

==3683== 
==3683== Syscall param rt_sigaction(act->sa_mask) points to uninitialised byte(s)
==3683==    at 0x65CADB5: __libc_sigaction (sigaction.c:67)
==3683==    by 0x638C0A5: gazebo::Server::Run() (Server.cc:436)
==3683==    by 0x4144A5: QTestFixture::RunServer(std::string const&, bool, bool) (QTestFixture.cc:109)
==3683==    by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683==    by 0x65C2E99: start_thread (pthread_create.c:308)
==3683==    by 0x8B713FC: clone (clone.S:112)
==3683==  Address 0x39d9cad0 is on thread 13's stack
==3683== 

==3683== 
==3683== Thread 13:
==3683== Invalid read of size 8
==3683==    at 0x81533B3: boost::thread::join() (in /usr/lib/libboost_thread.so.1.49.0)
==3683==    by 0x59C24B2: gazebo::physics::World::Stop() (World.cc:379)
==3683==    by 0x59B5569: gazebo::physics::stop_worlds() (PhysicsIface.cc:117)
==3683==    by 0x63A44C2: gazebo::shutdown() (gazebo.cc:229)
==3683==    by 0x638C17E: gazebo::Server::Run() (Server.cc:479)
==3683==    by 0x4144A5: QTestFixture::RunServer(std::string const&, bool, bool) (QTestFixture.cc:109)
==3683==    by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683==    by 0x65C2E99: start_thread (pthread_create.c:308)
==3683==    by 0x8B713FC: clone (clone.S:112)
==3683==  Address 0x3c7fbeb0 is 0 bytes inside a block of size 16 free'd
==3683==    at 0x4C2A44B: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3683==    by 0x59C24CB: gazebo::physics::World::Stop() (World.cc:380)
==3683==    by 0x59B5569: gazebo::physics::stop_worlds() (PhysicsIface.cc:117)
==3683==    by 0x63A44C2: gazebo::shutdown() (gazebo.cc:229)
==3683==    by 0x4F3A032: gazebo::gui::stop() (GuiIface.cc:280)
==3683==    by 0x4146FF: QTestFixture::cleanup() (QTestFixture.cc:126)
==3683==    by 0x7829986: QMetaMethod::invoke(QObject*, Qt::ConnectionType, QGenericReturnArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument) const (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.3)
==3683==    by 0x67E0F90: ??? (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683==    by 0x67E41FC: ??? (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683==    by 0x67E5056: QTest::qExec(QObject*, int, char**) (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683==    by 0x40F08D: main (ImagesView_TEST.cc:141)
==3683== 
==3683== Invalid read of size 8
==3683==    at 0x81533B3: boost::thread::join() (in /usr/lib/libboost_thread.so.1.49.0)
==3683==    by 0x5CFFC64: gazebo::sensors::SensorManager::SensorContainer::Stop() (SensorManager.cc:438)
==3683==    by 0x5CFFCD9: gazebo::sensors::SensorManager::Stop() (SensorManager.cc:94)
==3683==    by 0x63A44C7: gazebo::shutdown() (gazebo.cc:231)
==3683==    by 0x638C17E: gazebo::Server::Run() (Server.cc:479)
==3683==    by 0x4144A5: QTestFixture::RunServer(std::string const&, bool, bool) (QTestFixture.cc:109)
==3683==    by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683==    by 0x65C2E99: start_thread (pthread_create.c:308)
==3683==    by 0x8B713FC: clone (clone.S:112)
==3683==  Address 0x3c18e540 is 0 bytes inside a block of size 16 free'd
==3683==    at 0x4C2A44B: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3683==    by 0x5CFFC7D: gazebo::sensors::SensorManager::SensorContainer::Stop() (SensorManager.cc:439)
==3683==    by 0x5CFFCD9: gazebo::sensors::SensorManager::Stop() (SensorManager.cc:94)
==3683==    by 0x63A44C7: gazebo::shutdown() (gazebo.cc:231)
==3683==    by 0x4F3A032: gazebo::gui::stop() (GuiIface.cc:280)
==3683==    by 0x4146FF: QTestFixture::cleanup() (QTestFixture.cc:126)
==3683==    by 0x7829986: QMetaMethod::invoke(QObject*, Qt::ConnectionType, QGenericReturnArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument) const (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.3)
==3683==    by 0x67E0F90: ??? (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683==    by 0x67E41FC: ??? (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683==    by 0x67E5056: QTest::qExec(QObject*, int, char**) (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683==    by 0x40F08D: main (ImagesView_TEST.cc:141)
==3683== 

==3683== Thread 11:
==3683== Syscall param rt_sigaction(act->sa_flags) points to uninitialised byte(s)
==3683==    at 0x65CADB5: __libc_sigaction (sigaction.c:67)
==3683==    by 0x638C0A5: gazebo::Server::Run() (Server.cc:436)
==3683==    by 0x4144A5: QTestFixture::RunServer(std::string const&, bool, bool) (QTestFixture.cc:109)
==3683==    by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683==    by 0x65C2E99: start_thread (pthread_create.c:308)
==3683==    by 0x8B713FC: clone (clone.S:112)
==3683==  Address 0x36ad5ac0 is on thread 11's stack
==3683==
osrf-migration commented 10 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

osrf-migration commented 8 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

Recently, I've been seeing failures with the following console message:

UNIT_ImagesView_TEST: /usr/include/boost/thread/pthread/mutex.hpp:108:
 boost::mutex::~mutex(): Assertion `!posix::pthread_mutex_destroy(&m)' failed.

I've captured backtraces here.

osrf-migration commented 8 years ago

Original comment by Ian Chen (Bitbucket: Ian Chen, GitHub: iche033).

Not sure if it's still producing the the same backtrace as the one in the description but usually it happens if we manually call delete on a qt widget in our code. Later when Qt is shutting down, its parent widget will try to clean up and delete the qwidget (which we deleted) and this causes double deletion.

osrf-migration commented 8 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

pull request #2194

osrf-migration commented 8 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

pull request #2194 has fixed the failures on gazebo7, but it is now failing more frequently on default with backtraces that reference ImageFrame. It may be related to the changes made to ImageFrame in pull request #2112

osrf-migration commented 8 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

new backtrace on default:

it often happens in ~QImage called during ~ImageFramePrivate, which is called during ImagesView::UpdateImpl

osrf-migration commented 8 years ago

Original comment by Ian Chen (Bitbucket: Ian Chen, GitHub: iche033).

pull request #2209

osrf-migration commented 8 years ago

Original comment by Ian Chen (Bitbucket: Ian Chen, GitHub: iche033).

osrf-migration commented 8 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

I've observed a infrequent but repeatable memory corruption failure at the beginning of the test. The test still proceeds for some amount before failing:

171: [Err] [Scene.cc:2958] Light [sun] not found. Use topic ~/factory/light to spawn a new light.
171: *** Error in `gazebo/gui/viewers/UNIT_ImagesView_TEST': double free or corruption (fasttop): 0x00002b48dc017260 ***
171: [Msg] cam1 to cam6
171: [Msg] cam6 to cam1
171: [Msg] Random switch [1] of 25. Number of images [1]
171: [Msg] Random switch [2] of 25. Number of images [3]

Two separate but similar backtraces here.

osrf-migration commented 8 years ago

Original comment by Ian Chen (Bitbucket: Ian Chen, GitHub: iche033).

this is default or gazebo7 branch?

osrf-migration commented 8 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

default

osrf-migration commented 8 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

we just had a similar failure on jenkins on a branch off of gazebo7 (dea4fee84cc92bf6e34e297de391f3067f9c70bd)

https://gist.github.com/scpeters/6b64d7a8794ec6a3f1a7e31f003e14b9

osrf-migration commented 8 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

That branch lives up to its name...

osrf-migration commented 8 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).