TimMooreAu
commented
10 months ago I'm getting what I think is the same issue. In my case, I'm not using a GUI, the issue seems to occur when calling DataReader.listener() when the the reader and writer are within the same process, and the writer has already written a message when the reader is created (using Durable and TransientLocal QoS).
The issue seems to be due to a ScopedObjectLock which is locked in listener() (frame 18 of stack trace below). Before this lock is unlocked dds_reader_invoke_cbs_for_pending_events() is called, calling the listener callback. If the listener callback does a take(), a ScopedObjectLock on the same object tries to lock (frame 6 of stack trace).
To resolve this, perhaps the original lock needs to be unlocked before pending events are processed?
Stack trace:
#0 0x00007ffe8e0b0a44 in ntdll!ZwWaitForAlertByThreadId () from C:\\WINDOWS\\SYSTEM32\tdll.dll"
#1 0x00007ffe8e039205 in ntdll!RtlAcquireSRWLockExclusive () from C:\\WINDOWS\\SYSTEM32\tdll.dll"
#2 0x00007ffe1a3fab5d in ddsrt_mutex_lock (mutex=0x1826d38af00) at C:/Users/intim/Documents/cyclonedds/cyclonedds-master/src/ddsrt/src/sync/windows/sync.c:31"
#3 0x00007ffe0df76760 in org::eclipse::cyclonedds::core::Mutex::lock (this=0x1826d38b780) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/src/org/eclipse/cyclonedds/core/Mutex.cpp:33"
#4 0x00007ffe0df7696d in org::eclipse::cyclonedds::core::ObjectDelegate::lock (this=0x1826d38b778) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/src/org/eclipse/cyclonedds/core/ObjectDelegate.cpp:36"
#5 0x00007ffe0dfb6911 in org::eclipse::cyclonedds::core::ScopedLock<org::eclipse::cyclonedds::core::ObjectDelegate>::ScopedLock (this=0x18ef3fefe0, obj=..., lock=true) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/include/org/eclipse/cyclonedds/core/ScopedLock.hpp:41"
#6 0x00007ffe0df91615 in org::eclipse::cyclonedds::sub::AnyDataReaderDelegate::loaned_take (this=0x1826d38b5b0, reader=2077064500, mask=..., samples=..., requested_max_samples=4294967295) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/src/org/eclipse/cyclonedds/sub/AnyDataReaderDelegate.cpp:239"
#7 0x00007ff75c64abe7 in dds::sub::detail::DataReader<HelloWorldData::Msg>::take (this=0x1826d38b5b0) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/include/dds/sub/detail/TDataReaderImpl.hpp:613"
#8 0x00007ff75c649727 in dds::sub::DataReader<HelloWorldData::Msg, dds::sub::detail::DataReader>::take (this=0x18ef3ff170) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/include/dds/sub/detail/TDataReaderImpl.hpp:349"
#9 0x00007ff75c6473d0 in TakeListener::on_data_available (this=0x18ef3ff4f8, reader=...) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/examples/helloworld/publisher_modified.cpp:29"
#10 0x00007ff75c64a261 in dds::sub::detail::DataReader<HelloWorldData::Msg>::on_data_available (this=0x1826d38b5b0) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/include/dds/sub/detail/TDataReaderImpl.hpp:1380"
#11 0x00007ffe0df79e39 in callback_on_data_available (reader=2077064500, arg=0x1826d38ba30) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/src/org/eclipse/cyclonedds/core/ListenerDispatcher.cpp:177"
#12 0x00007ffe1a3b2c48 in da_or_dor_cb_invoke (rd=0x1826d38bdf0, lst=0x1826d38be90, status_and_mask=359929088, async=false) at C:/Users/intim/Documents/cyclonedds/cyclonedds-master/src/core/ddsc/src/dds_reader.c:215"
#13 0x00007ffe1a3b42db in dds_reader_invoke_cbs_for_pending_events (e=0x1826d38bdf0, status=5376) at C:/Users/intim/Documents/cyclonedds/cyclonedds-master/src/core/ddsc/src/dds_reader.c:450"
#14 0x00007ffe1a3c66ec in dds_entity_deriver_invoke_cbs_for_pending_events (e=0x1826d38bdf0, status=5376) at C:/Users/intim/Documents/cyclonedds/cyclonedds-master/src/core/ddsc/src/dds__types.h:261"
#15 0x00007ffe1a3c870f in dds_set_listener (entity=2077064500, listener=0x1826d38d300) at C:/Users/intim/Documents/cyclonedds/cyclonedds-master/src/core/ddsc/src/dds_entity.c:1083"
#16 0x00007ffe0df786b8 in org::eclipse::cyclonedds::core::EntityDelegate::listener_set (this=0x1826d38b5b0, _listener=0x18ef3ff4f8, mask=..., reset_on_invoke=true) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/src/org/eclipse/cyclonedds/core/EntityDelegate.cpp:226"
#17 0x00007ff75c64aeb2 in dds::sub::detail::DataReader<HelloWorldData::Msg>::listener (this=0x1826d38b5b0, l=0x18ef3ff4f8, event_mask=...) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/include/dds/sub/detail/TDataReaderImpl.hpp:744"
#18 0x00007ff75c649782 in dds::sub::DataReader<HelloWorldData::Msg, dds::sub::detail::DataReader>::listener (this=0x18ef3ff500, listener=0x18ef3ff4f8, event_mask=...) at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/src/ddscxx/include/dds/sub/detail/TDataReaderImpl.hpp:420"
#19 0x00007ff75c641a2e in main () at C:/Users/intim/Documents/cyclonedds/cyclonedds-cxx-master/examples/helloworld/publisher_modified.cpp:78"
I've included a modified version of the helloWorldPublisher application which reproduces the problem. Note: I've only tested this on windows.
#include <cstdlib>
#include <iostream>
#include <chrono>
#include <thread>
/* Include the C++ DDS API. */
#include "dds/dds.hpp"
/* Include data type and specific traits to be used with the C++ DDS API. */
#include "HelloWorldData.hpp"
using namespace org::eclipse::cyclonedds;
class TakeListener : public dds::sub::NoOpDataReaderListener<HelloWorldData::Msg>
{
void on_data_available(dds::sub::DataReader<HelloWorldData::Msg>& reader) override
{
reader.take();
std::cout << "=== Took data." << std::endl;
}
};
int main() {
std::cout << "=== [Publisher] Create writer." << std::endl;
/* First, a domain participant is needed.
* Create one on the default domain. */
dds::domain::DomainParticipant participant(domain::default_id());
/* To publish something, a topic is needed. */
dds::topic::Topic<HelloWorldData::Msg> topic(participant, "HelloWorldData_Msg");
/* A writer also needs a publisher. */
dds::pub::Publisher publisher(participant);
auto data_writer_qos = publisher.default_datawriter_qos();
data_writer_qos << dds::core::policy::Reliability::Reliable();
data_writer_qos << dds::core::policy::Durability::TransientLocal();
data_writer_qos << dds::core::policy::History::KeepLast(1);
/* Now, the writer can be created to publish a HelloWorld message. */
dds::pub::DataWriter<HelloWorldData::Msg> writer(publisher, topic, data_writer_qos);
/* Create a message to write. */
HelloWorldData::Msg msg(1, "Hello World");
/* Write the message. */
std::cout << "=== [Publisher] Write sample." << std::endl;
writer.write(msg);
std::cout << "=== [Publisher] Done." << std::endl;
std::cout << "=== [Subscriber] Create reader." << std::endl;
/* A reader also needs a subscriber. */
dds::sub::Subscriber subscriber(participant);
auto data_reader_qos = subscriber.default_datareader_qos();
data_reader_qos << dds::core::policy::Reliability::Reliable();
data_reader_qos << dds::core::policy::Durability::TransientLocal();
data_reader_qos << dds::core::policy::History::KeepLast(1);
/* Now, the reader can be created to subscribe to a HelloWorld message. */
dds::sub::DataReader<HelloWorldData::Msg> reader(subscriber, topic, data_reader_qos);
TakeListener listener;
reader.listener(&listener, dds::core::status::StatusMask::data_available());
std::cout << "=== Added listener." << std::endl;
while (true)
{
// Wait for data.
}
return EXIT_SUCCESS;
}
eboasson
e-hndrks
Greetings, Cyclone team. I've hit an issue trying to use CycloneDDS within a GUI application.
I have a basic "hello world" reader and writer working between two processes, and I've been trying to add a basic GUI to the reader as a proof of concept. In the eventual application, I want late-joining clients to be able to pick up all the existing instances and pass the live ones to a GUI. I think that means that I have to create a Listener, connect it up to my GUI objects, and then register the Listener with a DataReader, so that the GUI doesn't miss anything from that first callback. I do realize that I need to handle threading boundaries and memory ownership between the callback and the GUI event loop. But I haven't gotten that far yet...
As soon as I add even a basic GUI object AND have samples present (using Durable and TransientLocal QoS, with the writer process running), the call to register the Listener hangs. If there are no samples, the application starts up fine. Likewise, if I remove the GUI object and have samples present, the application also starts up fine.
At first I assumed it was something funny with my environment, but I've now tried the basic concept with multiple OSes and GUI toolkits, and I keep getting the same results. I've tried Qt6 on MacOS using both Homebrew and Qt's binary installer. I've tried Qt6 and Qt5 on Ubuntu using the system packages. I started thinking it was a Qt issue, but then I was able to also reproduce the problem with wxWidgets on Linux.
In all these cases, the reader process gets hung on a mutex lock, and the call to listener(...) never returns. The callback does get called, however.
I have some minimal examples that I can commit, if that's easier than posting entire sample applications here.
The CycloneDDS and CycloneDDS-CXX unit tests are passing on my builds.