NetMQ + Unity3D, am I wasting my time?

[Sohojoe]

I have been trying to implement a basic REQ/REP model with Unity3D (Mac OS X) acting as the server and python as the client. It works OK as a standalone Unity program. However Unity will soft-lock after 10-40 frames if i run with the Mono Develop debugger attached (requiring a Force Quit)

I have read through the numerous open and closed issues with regards to problems with NetMQ and Unity3D;

1) Is there a fundamental problem with Unity3D + NetMQ (i.e. I wasting my time with NetMQ and should look for an alternative)

2) or, is there a localized problem only when attaching Mono Develop debugger to Unity3D + NetMQ (i.e. NetMQ is good for production but I should look for another solution for debugging)

3) or, it's something hookey with what I'm trying to do (so either we debug that or I change my architecture)

Things I have tried

• building NetMQ and AsyncIO from master
• adding AsyncIO.ForceDotNet.Force ();
• adding NetMQConfig.ManualTerminationTakeOver();
• adding NetMQConfig.ContextCreate(true);
• using RouterSocket instead of ResponseSocket
• running server code on a seperate thread (per one of the example in the Issues

Environment

NetMQ Version:

• NetMQ 4.0.0-rc5
• AsyncIO 0.1.26 (Note: I started with the Nuget version of NetMQ and AsyncIO; then I built locally from git.Master)

Operating System:

• MacOS 10.12.1

.NET Version:

• Unity 5.4.3f1
• Python 2.7 (for the client)

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

[Supergeek]

I have a lot of the same concerns. I'm just learning NetMQ/ZMQ and it feels very fragile on Unity. Some guidelines and best practices to avoid editor freezes and crashes would be very helpful.

[dogagithub]

Same problem to me.Unity will stuck.

[flisky1]

Try closing all sockets and adding NetMQConfig.ContextTerminate(); to the OnDestroy() event of the MonoBehaviour. This fixed all of my freezing issues. I am using AsyncIO.ForceDotNet.Force(); NetMQConfig.ManualTerminationTakeOver(); NetMQConfig.ContextCreate(true); and a single RequestSocket for my development scenario under Unity 5.4.1.

[tobi-tobsen]

Most of the issues of using netmq with Unity is stemming from not properly initializing it (using AsyncIO.ForceDotNet.Force();) and not properly shutting down / disposing the netmq sockets and context (e.g. in OnDisable() or OnApplicationQuit() methods of the MonoBehaviour).

So, in my opinion you are not wasting your time and people are successfully using netmq with unity. You could try what others have been using.

Are there any additional things which could help you getting started with unity? Maybe post your solution or add documentation to help others?

[Sohojoe]

@tobi-tobsen, @flisky1

• I have created a sample project that only has AsyncIO, NetMQ.
• I created a Req/Rep example based on @tobi-tobsen's link
• It runs OK when running from Unity
• When I attach MonoDebug, and run from Unity, it locks after 5 or 6 cycles
• This is consistent with what I've seen in every test I have done
• Can you check if you get the same problem (if so, are you on Mac or PC?)

Code - https://github.com/Sohojoe/UnityMQ

[tobi-tobsen]

@Sohojoe I quickly skimmed through the code and as far as I can tell it should work that way. Since I have neither unity nor a mac at my hands, I cannot reproduce the behaviour. I am not convinced that this is related to netmq but may be related to unity + monodebug on mac since others are still having issues debugging (and here) without netmq involved.

[Sohojoe]

@tobi-tobsen I didn't get the sense that this problem was related to those other MonoDev issues. I all I do is click run which the debugger attached; I'm not stepping through code or actually 'debugging'

These are my ideas to move this forward, do you have other ideas?

• [x] Step through code to find where it locks
• [x] Check if bug presents on windows
• [x] Submit bug to Unity (I think we should at least step through the code first)

I will try and get some more time on this over the holidays

[Sohojoe]

1) I have Reproduced on Windows (when debugging with VisualStudio) 2) debugged and I can confirmed code locks within NetMQ 3) I have submitted a bug report to Unity

I also implemented the ReliablePubSub patterns; which also soft locks when using the debugger.

[Mystfit]

@Sohojoe The included UnityMQ example locks up on second run, which I believe happens when you call NetMQConfig.ManualTerminationTakeOver().

This is behaviour that I experience when I have more than two sockets active. When I call close on the sockets, the actual low level socket object never receives a shutdown message, and so the context will never terminate as it awaits a successful shutdown message from each socket. Since you have the cleanup code at the start, it doesn't lock when you stop the editor (which triggers OnApplicationQuit()), but it will lock when the old context has yet to close due to the frozen sockets.

Tested using Unity 5.5.0f3 on WIndows 10.

[valkjsaaa]

I just went a long way to figure this problem out.

Check this

The key is to have AsyncIO.ForceDotNet.Force(); before you start and NetMQConfig.Cleanup(); before you end.

[erdalpekel]

@Sohojoe Can you please tell me how you actually installed netmq on Mac OS X for use with Unity. And how you included it in the project.

Thanks

[Sohojoe]

@erdalpekel I had created a repro. I gave up on NetMQ for the problem I was hoping to solve, but I did update the repro with Unity 2017.1 / .Net4.6 - I think it is working now, but I didn't do deep testing

https://github.com/Sohojoe/UnityMQ

[overthrowrobotics]

I need to publish from Unity -> Python and also publish from Python -> Unity. I used @valkjsaaa code and it worked great for the Unity client and I can publish from Python. I haven't been able to figure out how to publish from Unity though, the server example is for req/res, not pub/sub.

@Sohojoe I tried your code and the sample where it publishes and subscribes in a single script works fine. When I use my python script to try to listen to that IP and port it doesn't work. Is reliable server/client somehow different than pub/sub?

[Sohojoe]

@overthrowrobotics I gave up NetMQ and went with a mem cache solution (Redis) which is complex but well suited to my use case (Reinforcement Learning where I need to sync at up 1000 per seconds)

What are you trying to do? There are a few different approaches. If you are not too worried about latency (and dont need to transfer too much data) then PubNub is super easy to set up. If you need something faster then it maybe worth looking at Unity's ML which uses sockets to comunicate between Unity and Python (it came out after I had figure out my path so I have not tried it but I belive it is open source). Initially I used HTTP / Rest which is pretty simple but not great for performance.

[overthrowrobotics]

Building a boxing robot. Here's an older version of it. https://www.youtube.com/watch?v=Bwv_wJbEbVE

I have 5 Teensy (32-bit arduino clone) that I'm using for various encoders, pressure sensors, IMUs, LEDs, dollar bill acceptor, etc. Standard System.IO.Ports sucks. Super unreliable and I've tried 5 other 3rd party serial libraries and they are all problematic. Pyserial works pretty well.

I managed to get mqtt/paho/mosquitto working last night. I'm only doing about 20hz with small <20 byte messages so I think it's going to be ok.

I was thinking about the Unity ML but I couldn't find any documentation on how to hijack it's sockets.

I use SARSA (similar to Q-Learning) in the robot through Aforge.net library since it was long before they released their ML stuff. It's not deep RL but for basic stuff it works. The problem is that there are a lot of new RL algorithms (even outside of deep) like contextual bandits that might work well and Aforge isn't being updated.

[jwvanderbeck]

I had a fully working setup using AsyncIO.ForceDotNet.Force() paired with NetMQConfig.Cleanup(false) in 2017.1 and 2017.2

However I recently did some tests in 2017.3f3 and with the same setup the freezing upon Unity close or Assembly Reload has returned :(

edit I think I jumped the gun on this. Reverted to 2017.2 and I am seeing the same problem. Which is very confusing as this was working fine a few months ago. The only other major change I am aware of (besides of course other work in Unity) was an upgrade in OS to Windows 10.

[jwvanderbeck]

After some more digging I discovered the issue. It works when Unity is set to use .NET 4.6, and hangs when set to the older option.

[rickyviking]

Same issue found again here! Tried code from @valkjsaaa with player build settings set to .NET 4.x as suggested by @jwvanderbeck Application hangs on exit both on preview and when running from standalone build. Unity 2018.1.6f1 NetMQ 4.0.01 Any suggestions?

[samuell]

Did you find any solution to this @rickyviking ?

I'm currently investigating using NetMQ with @valkjsaaa's tricks to talk to some external processes like Python, C++ etc, and is interested to know if there are still serious problems with this.

(Will try to update my own experiences, if I continue on this route).

[rickyviking]

@samuell I didn't find a fix/workaround to make it work with that version of Unity. I had to drop NetMQ e use basic socket instead. Good luck!

[samuell]

@rickyviking Thanks for the info, good to know!

[samuell]

I'm having a suspicion now ... don't you need to do the NetMQConfig.Clean() in the OnDestroy method on behaviours as well (such as here in @valkjsaaa 's code)?

I'm trying it now, but it results in Unity blocking ... which, according to the NetMQ cleanup docs is what happens when there are undisposed stuff left:

The most important thing to know about cleanup is that you must call Dispose on all sockets before calling Cleanup. Also make sure to cleanup any other resource from NetMQ library, like NetMQPoller, NetMQQueue and etc... If socket is not get disposed the NetMQConfig.Cleanup will block forever.

... so it seems the issue is doing more proper cleanup with NetMQ?

[jwvanderbeck]

Hey guys sorry I wasn't following this conversation. If it would still help, I can try and post the relevant pieces of our code. Everything has been working rather solid for almost the last year for us, the main key was using .NET 4.6 in Unity. 3.5 caused the locks.

[samuell]

Thanks for the reply @jwvanderbeck ! I actually also just got it working, with .Net 4.x and a slightly customised (for our needs) version of @valkjsaaa 's code :) (Thanks @valkjsaaa and all who helped out resolving this!).

I hope to find time to set up a proof-of-concept repo for what we did. Basically I think @valkjsaaa 's code works fine, but I implemented a component that does push/pull for exporting data too, as well as added the NetMQConfig.Clean() call in the Stop() method of the NetMqPublisher, to be called from OnDestroy() of the MonoBehavior object.

[offchan42]

What's the conclusion of the most bug-free way for communicating between Unity and other processes (e.g. python)? Is it still NetMQ?

[overthrowrobotics]

I gave up on zeromq a while ago and went to using websockets.

On Tue, Aug 13, 2019, 7:37 AM Chanchana Sornsoontorn < notifications@github.com> wrote:

What's the conclusion of the most bug-free way for communicating between Unity and other processes (e.g. python)? Is it still NetMQ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zeromq/netmq/issues/631?email_source=notifications&email_token=AHUKFBNKNGKFFLWJMEX6B43QELBJDA5CNFSM4CYNUGJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4F3TIQ#issuecomment-520862114, or mute the thread https://github.com/notifications/unsubscribe-auth/AHUKFBKLRF6K3CQZO4Q43GLQELBJDANCNFSM4CYNUGJA .

[jwvanderbeck]

We switched to RabbitMQ.

[offchan42]

I am able to use NetMQ on Unity as a request client to connect to the python server fine. But I found 2 issues with it: 1) If I create more than 1 client, the Unity editor would freeze. So If there is a way to fix this ZeroMQ would work fine. 2) Another issue is about the AsyncIO.ForceDotNet.Force(); and NetMQConfig.Cleanup(); thing. I don't even know what exactly they do and where to put them. So this becomes the mysterious piece of code that I don't trust lying around in my request client.

With these 2 issues in mind, it's making NetMQ unreliable to use in production for me. So I'm considering other inter-process communication methods like a named pipe, or whatever that works in multiple programming languages. (At least python, C++, and C#)

@jwvanderbeck Can you make RabbitMQ work with Unity and python? And is there a nagging issue like something I mentioned above or something you are not satisfied with it?

[Oneiros90]

Thank you guys for the suggestions, I was really getting crazy about this freeze problem. At the end, in my case, NetMQSocket.ReceiveFrameString() was the guilty. Calling it in a task is a big no no since it is blocking! Call the non-blocking version instead: NetMQSocket.TryReceiveFrameString()

[dparker2]

@Oneiros90 You can still use ReceiveFrameString() by marking the Thread it is run in as IsBackground = true so it is automatically killed when the main thread exits

[jBachalo]

Whats the latest update to this issue? I am having both editor freeze and memory leak issues using this code as a starting point https://github.com/sye8/Python-Unity-ZMQ

[xaedes]

Also had the freeze issues. For me it was NetMQConfig.Cleanup which was blocking. Solution: NetMQConfig.Cleanup(false);

[shanecelis]

NetMQ seems great, was easy to connect my dotnet with Unity project. However, as soon as I wanted it to work in the editor rather than a build, it became a lot tougher for me to get it running smoothly. I thought I'd offer my code that works with Unity 2022.1 and NetMQ 4.0.1.6 as another example for others who come here with similar issues.

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using NetMQ;
using NetMQ.Sockets;
using UnityEngine;

public class NetMQAsyncServer : MonoBehaviour {
  public int port = 5555;
  public int timeout = 10; // seconds
  private Queue<(string, Action<string>)> requests = new Queue<(string, Action<string>)>();
  private CancellationTokenSource tokenSource;

  void Start() {
    tokenSource = new CancellationTokenSource();

    // Since we're using async, log any errant exceptions.
    TaskScheduler.UnobservedTaskException -= UnobservedException;
    TaskScheduler.UnobservedTaskException += UnobservedException;

    Task.Run(() => {
      try {
        // Do the AsyncIO force on the thread that's going to use NetMQ.
        AsyncIO.ForceDotNet.Force();
        var token = tokenSource.Token;
        using var runtime = new NetMQRuntime();
        runtime.Run(token, ServerAsync(token));
      } catch (Exception e) {
        Debug.LogException(e);
      } finally {
        NetMQConfig.Cleanup(false);
      }
    });
  }

  private static void UnobservedException(object sender, UnobservedTaskExceptionEventArgs e) {
    e.SetObserved();
    Debug.LogException(e.Exception);
  }

  async Task ServerAsync(CancellationToken token) {
    using var responseSocket = new ResponseSocket($"@tcp://*:{port}");
    Debug.Log($"Listening on port {port}.");
    while (! token.IsCancellationRequested) {
      // We create a new source so that our primary token doesn't accrue garbage.
      using (var source = CancellationTokenSource.CreateLinkedTokenSource(token)) {
        var ourToken = source.Token;
        /* We don't give ReceiveFrame our primary token because it Registers to cancel its own
           promise, which can cause an exception after it's called. Suggest changing this
           in AsyncReceiveExtensions.cs from this:

       cancellationToken.Register(() => source.SetCanceled());
       ```
       to
       ```
       var disposeMe = cancellationToken.Register(() => source.SetCanceled());
       // later
       disposeMe.Dispose();
       ```
       Looks like this is handled better in the latest sources of NetMQ. I'm using
       NetMQ 4.0.1.6.

       https://github.com/zeromq/netmq/blob/ea0a5a7e1b77a1ade9311f187f4ff37a20d5d964/src/NetMQ/AsyncReceiveExtensions.cs#L172
    */
    var (message, _) = await responseSocket.ReceiveFrameStringAsync(ourToken);
    Debug.Log($"responseSocket : Server Received '{message}'");
    var responsePromise = new TaskCompletionSource<string>();
    requests.Enqueue((message, result => responsePromise.TrySetResult(result)));
    using (var registeredAction = ourToken.Register(() => responsePromise.TrySetCanceled(ourToken))) {
      var response = await responsePromise.Task;
      responseSocket.SendFrame(response);
    }
  }
}

}

/* Handle the requests on the main thread. / void Update() { if (requests.Count > 0) { var (request, provideResult) = requests.Dequeue(); / Just echo the request, basically. / provideResult($"I see: '{request}'."); } }

private void OnDisable() { if (tokenSource != null && ! tokenSource.IsCancellationRequested) { tokenSource.Cancel(); } // Don't do NetMQConfig.Cleanup(false); here. It'll hang in the editor.

// Don't cancel your token source after cleanup. It'll throw an exception.

}

private void OnDestroy() { tokenSource?.Dispose(); } }

[sapsari]

For those who wants to use NetMQ for Unity Editor extensions, domain reloads may mess with your sockets. You need to disable socket inheritance to close them properly.

That's how Microsoft/Unity handled it in package com.unity.ide.visualstudio https://github.com/needle-mirror/com.unity.ide.visualstudio/blob/master/Editor/Messaging/Messenger.cs

#if UNITY_EDITOR_WIN
        [System.Runtime.InteropServices.DllImport("kernel32.dll", SetLastError = true)]
        private static extern bool SetHandleInformation(IntPtr hObject, HandleFlags dwMask, HandleFlags dwFlags);

        [Flags]
        private enum HandleFlags: uint
        {
            None = 0,
            Inherit = 1,
            ProtectFromClose = 2
        }
#endif

        protected Messager(int port)
        {
            _socket = new UdpSocket();
            _socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ExclusiveAddressUse, false);
            _socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);

#if UNITY_EDITOR_WIN
            // Explicitely disable inheritance for our UDP socket handle 
            // We found that Unity is creating a fork when importing new assets that can clone our socket
            SetHandleInformation(_socket.Handle, HandleFlags.Inherit, HandleFlags.None);
#endif

I modified AsyncIO to reflect these changes here So in short, just replace AsyncIO.dll with this to make NetMQ work in Unity Editor extensions properly.

[spebern]

This issue hasn't received any update for quite a while, yet I want to share our recent experiences. We were using NetMQ 4.0.1.12, together with unity 2022.1.14.f1 and experienced sporadic message loss. This happened with the REQ as well as PUB sockets. Interestingly, when we caused a lot of traffic with the PUB socket from within unity our external subscribers lost messages roughly every 30s, but the messages that were lost were not the same.

We did not have more time to look into any underlying issue, but replaced our usage of NetMQ with simple sockets, which worked flawlessly.