PointCloud2 subscribe problem still exist

[whirwind]

Hello @EricVoll

Thanks for your work, and the discussion about PointCloud2 message before. I have updated my Hololens2 project and tested the result. But unfortunately, the subscriber of PointCloud2 doesn't work in my project, either.

Some results I found in my project: 1)In Unity, PointCloud2 and PointCloud message can be received correctly. 2)In Emulator or HL2 device, the 'ReceiveMessage' callback function can not be triggerred, 'isMessageReceived' variable is always False. 3)No any error notification on both HL2 client and server in Linux. 4)After the HL2 device(or emulator) connects to the server, the server appears to be running well, but actually doesn't send or receive data any more even if I connect to it using Unity. 5)I also subsribe Image, Pose messages, and publish Pose, Twist, PointCloud messages. They work well in both Unity and Emulator(or device). Espeicially I can publish PointCloud message but can't subscribe it. 6)I tried to reduce the number of points in PointCloud2 and PointCloud messages, thousands, hundreds, then only 10 points. But problem still exists.

A little problem I want to make sure: 1)For some reason I have to create and delete 'RosConnector.cs' during runtime by code. But the value of Serializer will be 'microsoft'. So I set the value of Serializer by habd in code and let it be the same to your Unitysample. Does this matter? public RosSocket.SerializerEnum Serializer= RosSocket.SerializerEnum.Newtonsoft_JSON;

Best, whirwind

[EricVoll]

Hi @whirwind

ok so the problem that something works in the editor but not in the UWP compiled project was something I repeatedly encountered during the development within the last two weeks. In my case there were two possible problems here: Either the json deserialization did not work, or once (we believe/assume) a compiler optimization optimized the code away.

But since for you subscribing to other message types works in the UWP compiled apps, it is unlikely that the problem is the json deserialization. I did notice, that messages are not "received" if the RosMsgName property of the class does not equal to the one how it was sent from the publisher even though the class matches, which makes sense of course. But then again, the fact, that it works in Editor suggests, that this is not the problem.

Your 4th point is what confuses me a bit. If the server doesn't send anything, then the UWP app can of course not receive anything. Have you tried debugging your server/ubuntu side application to find out why it doesn't send anything anymore? Does maybe the RosBridge crash or something like that?

About having to remove RosConnector: That seems weird. In my case (Both on HL2 and in Editor Mode) I did not have to do that. If you remove the RosConnector from a GameObject, then its instance "dies". The subscriber only subscribes in the public void Start() method, which is of course not called again if the RosConnector is added, since they are different comonents. So what could happen is, that at the start of the app the RosConnector starts, connects to the WebSocket, then the Subscriber subscribes, then you remove the RosConnector component, then the subscriber does not receive any messages anymore, then you add the RosConnector again (which might startup correctly) but the Subscribers don't re-subscribe anymore. Could you check if that might be the issue?

Eric 👋

[whirwind]

Hi @EricVoll

I am really sorry that I made a big mistake in the 4th point. After doing more tests, the truth is : "server doesn't work only happens after emulator. It works fine after connection from HL2 device and Unity. " I guess it is because I just simply close emulator after tests instead of closing the application. I can still connect using Unity after connecting using HL2 device.

About having to remove RosConnector: I want to 'connect' and 'disconnect' a server during running time. In order to avoid interference with subscribers and publishers scripts, my method is to delete all scripts while disconnecting and create them again while connecting. (I know it is a bit violent). This works I can connect and disconnect multiple times.

I found some more results or problems: 1.Even if callback function can't be triggered on HL2 device, server window still shows the PointCloud2 message is subscribed 2.All publishers work fine in Unity. However, Errors 'Expect a topic field' appears in server while using HL2 device. 3.If I publish a PointCloud(not 2) message without any point, callback function can be triggered in HL2.

Best, whirwind

[EricVoll]

Hi @whirwind

ok thanks for the clarification.

About publishing in Ros# UWP: good point! Might sound stupid but I actually didn't test it. I'll do that today. There probably is a msitake in the library somewhere. The error message in the console looks similar to one I had while working on the subscriptions. The json serialization does not work in some configurations (probably due to compiler optimization stuff - at least, thats our current theory).

@3: Interesting. Have you checked exectly how the PointCloud2 message (which is being published on linux side) looks like, exactly how each point class looks like and how their RosMessageNames are defined? The problem might be, that the Json Deserializer tries to deserialize it, but fails since the classes don't match perfectly and then doesn't "accept the message". This could explain why it works without any point, because then their deserialization can of course also not fail if there aren't any.

[whirwind]

Hi @EricVoll

I extracted one frame from my PointCloud2 message(except "data", that's too long) and compared this with your source code, not finding any obvious contradiction. If your assumption is right, maybe it is because parameter type can not match between C++ and Ros (e.g. Byte ,uint32, uint8)? But for PointCloud message there are only float for C++ and float32 for Ros, it doesn't make too much sense that these two types can not match.

topicname: "husky1/lidar_points"

header:
seq: 416
stamp: 
  secs: 1719
  nsecs: 792000000
frame_id: "husky1/lidar_points"

height:1
width:32768

fields:
[name: "x"
offset: 0
datatype: 7
count: 1, name: "y"
offset: 4
datatype: 7
count: 1, name: "z"
offset: 8
datatype: 7
count: 1]

is_bigendian:False
point_step:12
row_step:393216
is_dense:True

[EricVoll]

hi @whirwind

so I found the problem with publishing data. The Advertisement did not work, because Unity's IL2CPP compiler stripped away the properties of the Publisher class. Adding a link.xml file as explained here did the trick. I'll push the necessary files soon. This might also solve your issue with the pointcloud2 subscriber.

[EricVoll]

I updated the Readme, added a short description and added the required files to the ProjectSetup and Unity3D folder of RosSharp. Please try out your project with this link.xml file and tell me if it worked now. While debugging the code I also received the same error msg on the ROS side as you did. That was a result of the IL2CPP compiler stripping away code, which was then not serialized, and thus the RosMaster never received any Topic information on which something was going to be published. Should be fixed now. I guess that a similar issue might be the cause of your problem with the PointCloud2 subscriber.

[whirwind]

Yes, it works. both problems have been sovled. Thank you very much for your help all the way. And I guess I might need your help for more problems in the future, too. Thanks. (In fact when I talked about the problems with my instructor, he mentioned you surprisingly and said that there is a person who is an intern at Microsoft and is very good at this field)

[EricVoll]

Very glad to hear @whirlwind happy to help out. Indeed very interesting. Might I ask where you are studying/working and who your supervisor is?

[whirwind]

I am a master student in NYU, my supervisor is Prof. Giuseppe Loianno from The agile robotics and perception lab. In fact I made some omissions. Prof Loianno has a good relationship with Mr. Jeffrey Delmerico in Microsoft, who give us guides and suggestions about Hololens developemt. It is Mr. Jeff who mentioned you and said you might be able to help us.

[EricVoll]

Ah yes the world is tiny sometimes. Anyways, you know now where you can find me 🙂