Open osrf-migration opened 10 years ago
Original comment by Nate Koenig (Bitbucket: Nathan Koenig).
Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
77: Test command: /var/lib/jenkins/workspace/gazebo-default-devel-quantal-amd64-gpu-nvidia/build/gazebo/gui/viewers/UNIT_ImagesView_TEST "-xml" "-o" "/var/lib/jenkins/workspace/gazebo-default-devel-quantal-amd64-gpu-nvidia/build/test_results/UNIT_ImagesView_TEST.xml"
77: Test timeout computed to be: 240
77: SharePercentChange[0.626495] ShareMaxPercentChange[1]
77: ResPercentChange[1.15663]ResMaxPercentChange[3]
77: SharePercentChange[-0.0350342] ShareMaxPercentChange[1]
77: ResPercentChange[0.179993]ResMaxPercentChange[3]
77/128 Test #77: UNIT_ImagesView_TEST ...................***Timeout 240.09 sec
Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
I tried valgrind --tool=memcheck and it's found some invalid memory accesses, though I'm not sure if they're connected to this test failure:
$ valgrind -v --tool=callgrind gazebo/gui/viewers/UNIT_ImagesView_TEST
...
==3683== Thread 16:
==3683== Invalid write of size 4
==3683== at 0x8EB85C9: gazebo::transport::Publisher::SendMessage() (Publisher.cc:187)
==3683== by 0x8EAE935: gazebo::transport::Node::ProcessPublishers() (Node.cc:152)
==3683== by 0x8EBA702: gazebo::transport::TopicManager::ProcessNodes(bool) (TopicManager.cc:155)
==3683== by 0x8EA63E8: gazebo::transport::ConnectionManager::RunUpdate() (ConnectionManager.cc:276)
==3683== by 0x8EA6D15: gazebo::transport::ConnectionManager::Run() (ConnectionManager.cc:307)
==3683== by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683== by 0x65C2E99: start_thread (pthread_create.c:308)
==3683== by 0x8B713FC: clone (clone.S:112)
==3683== Address 0x3c75a754 is 36 bytes inside a block of size 40 free'd
==3683== at 0x4C2A44B: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3683== by 0x8EB92A7: std::_Rb_tree<unsigned int, std::pair<unsigned int const, int>, std::_Select1st<std::pair<unsigned int const, int> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, int> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<unsigned int const, int> >) (new_allocator.h:100)
==3683== by 0x8EB824B: gazebo::transport::Publisher::OnPublishComplete(unsigned int) (stl_tree.h:783)
==3683== by 0x8E9DBE7: boost::function1<void, unsigned int>::operator()(unsigned int) const (function_template.hpp:760)
==3683== by 0x8EB4310: gazebo::transport::Publication::Publish(boost::shared_ptr<google::protobuf::Message>, boost::function<void ()(unsigned int)>, unsigned int) (Publication.cc:334)
==3683== by 0x8EB85C8: gazebo::transport::Publisher::SendMessage() (Publisher.cc:187)
==3683== by 0x8EAE935: gazebo::transport::Node::ProcessPublishers() (Node.cc:152)
==3683== by 0x8EBA702: gazebo::transport::TopicManager::ProcessNodes(bool) (TopicManager.cc:155)
==3683== by 0x8EA63E8: gazebo::transport::ConnectionManager::RunUpdate() (ConnectionManager.cc:276)
==3683== by 0x8EA6D15: gazebo::transport::ConnectionManager::Run() (ConnectionManager.cc:307)
==3683== by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683== by 0x65C2E99: start_thread (pthread_create.c:308)
==3683==
==3683==
==3683== Syscall param rt_sigaction(act->sa_mask) points to uninitialised byte(s)
==3683== at 0x65CADB5: __libc_sigaction (sigaction.c:67)
==3683== by 0x638C0A5: gazebo::Server::Run() (Server.cc:436)
==3683== by 0x4144A5: QTestFixture::RunServer(std::string const&, bool, bool) (QTestFixture.cc:109)
==3683== by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683== by 0x65C2E99: start_thread (pthread_create.c:308)
==3683== by 0x8B713FC: clone (clone.S:112)
==3683== Address 0x39d9cad0 is on thread 13's stack
==3683==
==3683==
==3683== Thread 13:
==3683== Invalid read of size 8
==3683== at 0x81533B3: boost::thread::join() (in /usr/lib/libboost_thread.so.1.49.0)
==3683== by 0x59C24B2: gazebo::physics::World::Stop() (World.cc:379)
==3683== by 0x59B5569: gazebo::physics::stop_worlds() (PhysicsIface.cc:117)
==3683== by 0x63A44C2: gazebo::shutdown() (gazebo.cc:229)
==3683== by 0x638C17E: gazebo::Server::Run() (Server.cc:479)
==3683== by 0x4144A5: QTestFixture::RunServer(std::string const&, bool, bool) (QTestFixture.cc:109)
==3683== by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683== by 0x65C2E99: start_thread (pthread_create.c:308)
==3683== by 0x8B713FC: clone (clone.S:112)
==3683== Address 0x3c7fbeb0 is 0 bytes inside a block of size 16 free'd
==3683== at 0x4C2A44B: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3683== by 0x59C24CB: gazebo::physics::World::Stop() (World.cc:380)
==3683== by 0x59B5569: gazebo::physics::stop_worlds() (PhysicsIface.cc:117)
==3683== by 0x63A44C2: gazebo::shutdown() (gazebo.cc:229)
==3683== by 0x4F3A032: gazebo::gui::stop() (GuiIface.cc:280)
==3683== by 0x4146FF: QTestFixture::cleanup() (QTestFixture.cc:126)
==3683== by 0x7829986: QMetaMethod::invoke(QObject*, Qt::ConnectionType, QGenericReturnArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument) const (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.3)
==3683== by 0x67E0F90: ??? (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683== by 0x67E41FC: ??? (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683== by 0x67E5056: QTest::qExec(QObject*, int, char**) (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683== by 0x40F08D: main (ImagesView_TEST.cc:141)
==3683==
==3683== Invalid read of size 8
==3683== at 0x81533B3: boost::thread::join() (in /usr/lib/libboost_thread.so.1.49.0)
==3683== by 0x5CFFC64: gazebo::sensors::SensorManager::SensorContainer::Stop() (SensorManager.cc:438)
==3683== by 0x5CFFCD9: gazebo::sensors::SensorManager::Stop() (SensorManager.cc:94)
==3683== by 0x63A44C7: gazebo::shutdown() (gazebo.cc:231)
==3683== by 0x638C17E: gazebo::Server::Run() (Server.cc:479)
==3683== by 0x4144A5: QTestFixture::RunServer(std::string const&, bool, bool) (QTestFixture.cc:109)
==3683== by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683== by 0x65C2E99: start_thread (pthread_create.c:308)
==3683== by 0x8B713FC: clone (clone.S:112)
==3683== Address 0x3c18e540 is 0 bytes inside a block of size 16 free'd
==3683== at 0x4C2A44B: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3683== by 0x5CFFC7D: gazebo::sensors::SensorManager::SensorContainer::Stop() (SensorManager.cc:439)
==3683== by 0x5CFFCD9: gazebo::sensors::SensorManager::Stop() (SensorManager.cc:94)
==3683== by 0x63A44C7: gazebo::shutdown() (gazebo.cc:231)
==3683== by 0x4F3A032: gazebo::gui::stop() (GuiIface.cc:280)
==3683== by 0x4146FF: QTestFixture::cleanup() (QTestFixture.cc:126)
==3683== by 0x7829986: QMetaMethod::invoke(QObject*, Qt::ConnectionType, QGenericReturnArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument, QGenericArgument) const (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.3)
==3683== by 0x67E0F90: ??? (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683== by 0x67E41FC: ??? (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683== by 0x67E5056: QTest::qExec(QObject*, int, char**) (in /usr/lib/x86_64-linux-gnu/libQtTest.so.4.8.3)
==3683== by 0x40F08D: main (ImagesView_TEST.cc:141)
==3683==
==3683== Thread 11:
==3683== Syscall param rt_sigaction(act->sa_flags) points to uninitialised byte(s)
==3683== at 0x65CADB5: __libc_sigaction (sigaction.c:67)
==3683== by 0x638C0A5: gazebo::Server::Run() (Server.cc:436)
==3683== by 0x4144A5: QTestFixture::RunServer(std::string const&, bool, bool) (QTestFixture.cc:109)
==3683== by 0x81526C8: ??? (in /usr/lib/libboost_thread.so.1.49.0)
==3683== by 0x65C2E99: start_thread (pthread_create.c:308)
==3683== by 0x8B713FC: clone (clone.S:112)
==3683== Address 0x36ad5ac0 is on thread 11's stack
==3683==
Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
Recently, I've been seeing failures with the following console message:
UNIT_ImagesView_TEST: /usr/include/boost/thread/pthread/mutex.hpp:108:
boost::mutex::~mutex(): Assertion `!posix::pthread_mutex_destroy(&m)' failed.
I've captured backtraces here.
Original comment by Ian Chen (Bitbucket: Ian Chen, GitHub: iche033).
Not sure if it's still producing the the same backtrace as the one in the description but usually it happens if we manually call delete
on a qt widget in our code. Later when Qt is shutting down, its parent widget will try to clean up and delete the qwidget (which we deleted) and this causes double deletion.
Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
pull request #2194 has fixed the failures on gazebo7, but it is now failing more frequently on default with backtraces that reference ImageFrame
. It may be related to the changes made to ImageFrame
in pull request #2112
Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
new backtrace on default:
it often happens in ~QImage
called during ~ImageFramePrivate
, which is called during ImagesView::UpdateImpl
Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
I've observed a infrequent but repeatable memory corruption failure at the beginning of the test. The test still proceeds for some amount before failing:
171: [Err] [Scene.cc:2958] Light [sun] not found. Use topic ~/factory/light to spawn a new light.
171: *** Error in `gazebo/gui/viewers/UNIT_ImagesView_TEST': double free or corruption (fasttop): 0x00002b48dc017260 ***
171: [Msg] cam1 to cam6
171: [Msg] cam6 to cam1
171: [Msg] Random switch [1] of 25. Number of images [1]
171: [Msg] Random switch [2] of 25. Number of images [3]
Two separate but similar backtraces here.
Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
we just had a similar failure on jenkins on a branch off of gazebo7
(dea4fee84cc92bf6e34e297de391f3067f9c70bd)
https://gist.github.com/scpeters/6b64d7a8794ec6a3f1a7e31f003e14b9
Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
In my overnight testing, the UNIT_ImagesView_TEST is now failing about 1/3 of the time since we merged pull request #921. Last night it failed 8/25 times:
double free or corruption
errors, 2 of which gave memory map and backtracesBacktrace 1
Backtrace 2