search

siemens / ros-sharp

ROS# is a set of open source software libraries and tools in C# for communicating with ROS from .NET applications, in particular Unity3D
Apache License 2.0
963 stars 367 forks source link

Hololens 1 connecting on Rosbridge but not communicating both on topics and actions #278

Closed aioniosfoititis closed 4 years ago

aioniosfoititis commented 4 years ago

3 ros_connector 2

I found a bug!

aioniosfoititis commented 4 years ago

I'm trying to communicate Hololens 1 with ROS. When I use the unity play everything is occuring as expected.When the project is build although the hololens are connected but there is no no communication,tried both topics and action messages.Both tried in build settings .NET 2.0 and 4.x as well as VS2017 and VS2019.

I found a bug!

win 10 and VMware 2019.2.11f1 but also tried 2018.4 ROS Melodic 18.04 Hololens 1

berkayalpcakal commented 4 years ago

Hello @aioniosfoititis,

Have you tried the repo from @dwhit? Here is the link for UWP supported version for ROS#. Currently I am able to do two way communication of simple messages and action messages between ROS and a Unity application that is deployed to Hololens 1.

aioniosfoititis commented 4 years ago

@berkayalpcakal Thanks for the quick reply.No I haven't noticed that that exists.Huge relief that there is actually support for that. I will test that as soon as possible and come again to report if everything works fine.

aioniosfoititis commented 4 years ago

@berkayalpcakal Still the same result.Connection but no communication. Is there any special step I should have done in order to make the system connect? Im inserting Mixed Reality toolkit ,placing Ros Connector and eg fibonacci server and adding action name and ip address.

With the play button everything occurs with success.On the other hand when deployed in the hololens I can only notice a connection to the rosbridge but no subscribing to the fibonacci topics.

berkayalpcakal commented 4 years ago

Hi again @aioniosfoititis,

Sadly I am not able to reproduce the issues that you have. The only recommendation I can make is that maybe you should check the "Project Settings/Player/Capabilities" ( I am not sure if I remember the path correctly, you can go ahead and check this link), and enable network related fields. Maybe there is an unchecked field which prevents the Unity app that is deployed to Hololens from having a proper communication to ROS installed PC.

aioniosfoititis commented 4 years ago

To anyone that listens, As for the capabilities I'm using InternetClient, InternetClientServer and privateClientServer.

I also tried to not use the virtual machine and use an actual PC, but the problem still occurs.I can see message "one client connected" but no subscribing to any topics happens. After that I tried the Export commands of ROS ( ip,hostname ,master uri ) but it didnt real overcome the problem. Also tried the UWP version of the websocket in the RosConnector . Anyone that has exei more suggestions please feel free to respond and I will give it a try.

helenol commented 4 years ago

To revive this a bit, ran into the same issues and was able to get around them by following this thread: https://github.com/dwhit/ros-sharp/issues/3

Solution was to use @dwhit's fork and replace the Netwonsoft.json dll with the AOT one from the asset store here: https://assetstore.unity.com/packages/tools/input-management/json-net-for-unity-11347

aioniosfoititis commented 4 years ago

I can also confirm the mentioned fix!

aoboaten1 commented 4 years ago

@helenol I tried exactly what you said and it does not work for me.

aioniosfoititis commented 4 years ago

@aoboaten1 Can u name what are u using,unity version ,ros# version, ubuntu version?

aoboaten1 commented 4 years ago

@aoboaten1 Can u name what are u using,unity version ,ros# version, ubuntu version? OS: WINDOWS 10 Unity Version : Unity 2019.2.20f1 (64-bit) ROS Distribution : ROS Kinetic I use use the @dwhit's fork and followed all instructions

helenol commented 4 years ago

Try 2019.3.1 for Unity as first debug step. I'm also using Kinetic. (Using Hololens 2 though).

On Fri, Feb 28, 2020, 08:41 aoboaten1 notifications@github.com wrote:

@aoboaten1 https://github.com/aoboaten1 Can u name what are u using,unity version ,ros# version, ubuntu version? OS: WINDOWS 10 Unity Version : Unity 2019.2.20f1 (64-bit) ROS Distribution : ROS Kinetic I use use the @dwhit https://github.com/dwhit's fork and followed all instructions

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/siemens/ros-sharp/issues/278?email_source=notifications&email_token=ABK3GCAA5PJ5YBOAREYTTX3RFC54LA5CNFSM4KFKBPU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENHSSXI#issuecomment-592390493, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABK3GCCKR64UOLX4PKTCMT3RFC54LANCNFSM4KFKBPUQ .

aoboaten1 commented 4 years ago

Try 2019.3.1 for Unity as first debug step. I'm also using Kinetic. (Using Hololens 2 though). Yeah. I have used 2019.3.1 as well but it didnt work. I use hololens. I went back to an old version of the Ros# I had downloaded and built with .net backend instead of il2cpp and it worked

aioniosfoititis commented 4 years ago

@aoboaten1 So if you are using .net backend you have probably used 2018.x version of unity as 2019.x does not support .net anymore ,If I recall correctly.

aoboaten1 commented 4 years ago

@aioniosfoititis Yes. I have about 4 versions on of unity on my laptop. I downloaded the newer version of the ros# and it wouldn't work so I went to my old laptop to use the old ros# i had and used 2018 unity version.

rdmerillat commented 4 years ago

Hello, I am currently trying to do something like what has been discussed in this thread, however I have been having issues getting it to run properly. For reference I currently have my ROS nodes and topics running on a ubuntu computer connected to my network and running/building my unity in windows. Same thing as above, I get rosbridge connection, but no subcribing to topics. I'm running on windows 10 with Unity 2019.3.14f1 using ROS Melodic.

Thus far I have done the following:

In the MRTK build window I am using:

I am able to use the build and install method, However as stated previously, I get a rosbridge connection, but it does not subscribe to my image topic, and when run in Unity it works just fine. Any assistance would be greatly appreciated.

berkayalpcakal commented 4 years ago

Hello @BloodRaine, just asking for curiosity, have you tried .NET Serializer (preferably with WebSocketSharp protocol) instead of Newtonsoft serializer? You can see the corresponding settings in the RosConnector component.

rdmerillat commented 4 years ago

@berkayalpcakal I have not, Any resources/ways to set that up? I do not see .NET as a serializer option in the connector.

berkayalpcakal commented 4 years ago

I think the fork by @dwhit is old, so you need to adjust the new changes in the serialization interface (please see #292) to your code. Or, you can adjust the modifications done in that fork to the current ROS#, see my comment here. I hope this helps. I would be glad if you can comment your experiences regarding .NET Serialization tool with HoloLens here.

rdmerillat commented 4 years ago

Just so that I am making sure I'm understanding you correcltly @berkayalpcakal, I can:

rdmerillat commented 4 years ago

@berkayalpcakal after updating to the latest ros-sharp, and updating subscriber scripts to follow newer syntax for Messages -> MessageTypes and Subscriber -> UnitySubscriber, I get the error "ReflectionTypeLoadException ... etc" is this something that you have seen before?

Jntzko commented 4 years ago

This could help with the Reflection issue: https://github.com/siemens/ros-sharp/issues/201#issuecomment-646590687 I'm interested in your branch, could you upload it?

rdmerillat commented 4 years ago

Yeah thats very true, after taking steps that were defined in some of the linked comment threads above (same as the one you linked), I managed to get around that issue using this fork of ros-sharp, however, I am running into some issues of certain namespaces not being found by the Newtonsoft.Json such as JsonConvert. @Jntzko once I have a working system I plan to create my own fork/version, unless it would help in debugging what issues I am running into.

rdmerillat commented 4 years ago

@Jntzko, have you managed to get a working connection with topic subscription/publishing? I think I have my dll's set properly, however, I am having other libraries throw issues. That or I get 'JsonConvert does not exist in Newtonsoft.Json'. @berkayalpcakal Any thoughts on this? For my project I am using OpenCVForUnity and that seems to be having issues when my dll's are set properly. I haven't yet gone as far as to take ros-sharp and re-remove the reflection code as @EricVoll had previously done.

Jntzko commented 4 years ago

I didn't have it running yet, I use the current ros-sharp master branch and applied the changes from dwhits fork, but I can't connect to topics, even with the json-net-for-unity dll, independent of the serializer. I didn't applied the reflection fix yet, maybe that helps.

EricVoll commented 4 years ago

Hi guys, I'll comment in 1-2 hours and give you some links. On mobile atm... Maybe I can help. In the meantime you can have a look here: ARbotics We have a full (tested) "How to get started" section. It should work right out of the box after cloning the master branch. It is possible to run all of that on the HL1 & 2. We have a stably running app The framework might even fit all your requirements to be used as is. We will actively continue to develop the framework.

The changes I had to make to get everything running on HL1: First of all: If the "game" runs (and works as expected, connects successfully etc.) in the Unity Editor mode, successfully builds for UWP apps, but does not work on the HoloLens it is most probably an AOT issue. If it runs in Editor Mode, but does not build for UWP apps, its most probably an issue with asmdef files or the Unity.Editor namespace (which Ros# uses extensivley)

"The reflection fix": The main thread on my RosSharp fork is heavily modified (especially the Unity Part) such that it does not have a lot to do with the "standard" Ros# anymore. But there is another thread I quickly created for another guy ReflectionFix that removes all reflection code from Ros#, which was necessary for me (and him) to get it running on the HL1.

The main thread of my fork removed all of the Unity.Edtior namespaces from Ros#, such that the robot generation from urdf files also works during runtime, which the "vanilla" Ros# cannot do in buit apps, due to its use of the Unity.Editor namespace. There are a few caviats with that though. For example, the robot has to be imported using the "vanilla Ros#" once before, since you cannot make prefabs out of stl files in runtime directly. We have solutions ready but did not have the time yet to implement them.

"Json Serializer": After failing for many, many hours, I threw away the NewtonSoft.json.dll's from the Ros# repository, and found a working workaround using a AOT-compilable-Version of NewtonSoft with the exact same API, which allowed a dll-replacement. Now, I had to leave the NewtonSoft.dll files in the Ros# Plugins folder for Unity to compile, but I configured there build process in a way, that they are not built for any platform. Then I added the "AOT-compilable" Newtonsoft and configured it to build for all platforms. This way you can "trick" unity into building (and working) during Editor time but also during built-application-time - even on the HoloLens. There is a comment with the links somewhere (I'll look for them in a sec) (Edit: Here is the comment and the issue)

Ros-Sharp and its 2 components: Ros-Sharp has two components. The actual ROS# communication/urdf-parsing dll (from now on I'll call it "library"), and the Unity-Scripts used to generate Robot-GameObjects. Using the library can be used on HL (UWP builds) if all the reflection code is removed, which can be done easily (see my fork with the second branch above), if you also utilize the NewtonSoft.json.dll trick I mentioned above. The Unity Ros#-Scripts cannot be used during runtime in any built application, due to the use of the Unity.Editor namespace, which is not available in built applications. Actually, the building process will already fail if you try to use it. That's why I rewrote a huge part of these Scripts for our project.

Lastly some comments:

I hope there is some information in there for you! Regards, Eric

Jntzko commented 4 years ago

Thanks a lot for your detailed description, this definitely cleared up some things. ARbotics looks very interesting, I didn't new about that work. Unfortunately, I can start integrating your suggestions in 2 weeks, but I'm pretty sure they will help a lot!

Jntzko commented 4 years ago

You were right @EricVoll, thanks a lot. I was able to publish to a topic from the HoloLens 2 emulator, I just copied the RosSharp and the plugin folder from the Assets of ARbotics. I don't have access to the real hardware right now.

My setup right now is: Unity 2019.3.15f and MRTK 2.4.0

EricVoll commented 4 years ago

Happy to hear that everything worked out so far. I exchanged a few messages today with @MartinBischoff who will probably/maybe add a link to the specific fork on the Ros# Readme under "Ros# for HoloLens" (or something similar)

rdmerillat commented 4 years ago

I am still running into issues when trying to do the same. I am currently using the OpenCVForUnity Package and when I try the same method that @Jntzko tried, I get a sort of namespace or package error - 'Utils' does not contain defenition for (function name here) for multiple methods. I imagine it has something to do with dll's or plugins or something but I'm not certain.

EricVoll commented 4 years ago

I am still running into issues when trying to do the same. I am currently using the OpenCVForUnity Package and when I try the same method that @Jntzko tried, I get a sort of namespace or package error - 'Utils' does not contain defenition for (function name here) for multiple methods. I imagine it has something to do with dll's or plugins or something but I'm not certain.

I know the error. I guess you used my wrong ros-sharp fork branch by mistake. In the meantime I updated the repository and also added a demo project that is ready to deploy. I'll edit the readme today with a full tutorial on how to start rosSharp on HoloLens. Use the new main branch from www.github.com/EricVoll/ros-sharp

rdmerillat commented 4 years ago

Oh thats perfect. My bad on using the wrong one. I got a bit confused with there being 2 in the Repo. Thanks for updating. I'll be sure to update to the new main. Thanks for the assistance!

rdmerillat commented 4 years ago

Ok, I think it's mostly working. It is now successfully subscribing which is awesome progress! I'm hitting a Nullreference error that an object isnt instantiated, considering everything I have, and the amount of incoming nullReference messages, it is probably something in the subscriber that is trying to be accessed when the message is recieved. Have you run into something like this and have any tips for tracking it down further? Long shot on that one I know, just checking before I go digging for too terribly long.

EricVoll commented 4 years ago

Yes I experienced that as well but I forgot why. Generally it's easy to find out that problem. Simply attach Visual Studio to Unity and start the Debugging Mode. Or in the Unity-Console you'll see the NullReferenceException and double click it to get where the exception is thrown. It probably is obvious then.

rdmerillat commented 4 years ago

I may be running it improperly as when I run it in debuggin gmode from VS on the device, I dont get any specifics like what object is Null or a line number, and Unity hasn't been giving me any errors when it is run.

EricVoll commented 4 years ago

So do you get the exceptions if the "game" is running on the HoloLens, or while it is running in Editor Mode? These NullReferenceExceptions usually also occur in Editor Mode. I had them for some time somewhere in a Subscriber if I remember correctly.

What I did for another project was to do a "sneaky" thing: You can subscribe to the Unity Log-Output and add a string publisher that publishes the Log via a ROS-topic. Investing a bit of time upfront in things like this made our life much easier down the road while developing the app. See the publisher and here is how to use it.

That way you can get Unity's Debug Logs while the app is deployed on the HoloLens. Theoretically it is possible to attatch the VisualStudio Debugger to a running instance on HoloLens. But I keep forgetting how to do that - you might want to google that for a bit. Its easy once you found out how to...

rdmerillat commented 4 years ago

So big picture I am using ROS to package up a Pointcloud as an image and send it to unity then render it, we need to do custom stuff with the point cloud so we cant just use the built in spacialization stuff. The app still works fin ine the Editor, and when I run it with "Start Debugging" in VS on the device, so on the hololense, In the bottom left here is a debug window. That was telling me there was a NullReference. But it still ran fine in the Editor. Subscribing to the Unity log might also be a useful thing for debugging for sure.

Previously what I had tried was to add your versions, files, and plugins to my project. Now I'm trying out your sample project with my libraries and scripts added and seeing if that makes a difference. Your sample proj doesnt use MRTK and I think that's the main difference.

belowis the exact error message I get when using the demo project in your demo project @EricVoll. A few things I notice, There is some network error that wasnt happening before, it also is only calling Null Reference on a sharer or something of the sort, when in the other project, there would be a continuous stream of NullReference errors, which made me thing it was something in the Subscriber.

20200813_161528_HoloLensh

EricVoll commented 4 years ago

mhh that seems weird. But its a bit hard to help without the opportunity to look at the stuff myself. I assume you thought of adjusting the IP-Adresses on the ROS-Connector to whatever your IP-adresses are right?

As I wrote above, try this: https://docs.microsoft.com/en-us/windows/mixed-reality/using-visual-studio#:~:text=Pairing%20your%20device,-The%20first%20time&text=On%20the%20HoloLens%2C%20generate%20a,HoloLens%20to%20dismiss%20the%20dialog. With the steps explained in this article you can pair your HL with Visual Studio and live-debug with breakpoints etc.

rdmerillat commented 4 years ago

Yep, I double checked the IP, I'll definately take a look at setting up the HL-VS debug with breakpoints Thanks for reminding me of the link to how to do that. I'll let you know the results

rdmerillat commented 4 years ago

I was able to get my application working! The issue was that a shader I was using was registerred as Null. In my renderer, I attempt a Shader.find and that was coming back with nothing as I guess it may not be included in the build if nothing references it directly. The solution was to attach the shader to a material in the Resources folder. Thanks for the help. Looks like the data stream is a little slow so my next task will be to try to find a way to speed that up.

EricVoll commented 4 years ago

Great to hear! So in that case the issue wasn't Ros#. If you have any further issues with ROS# please create issues over on the UWP fork.