Removing a socket from a poller won't guarantee that their callbacks won't be called

Environment

NetMQ Version:    4.0.1.2-pre
Operating System: Windows 10
.NET Version:     .NET Core 3.1

Expected behaviour

Removing a socket should disallow any further triggering of its callbacks.

Actual behaviour

The opposite.

Steps to reproduce the behaviour

Run the last example in this Lithe commit using the UI. What happens is that I get a queue empty exception which should not be happening had this operation been truly thread safe.

I am going the ZeroMQ guide example by example, and this is the clustering one.

This issue is similar to #900. I am not really sure if what I am reporting are bugs, or if the library should be like this.

One of the main differences between ZeroMQ and NetMQ are how the two do pooling. NetMQ tried for a more idiomatic experience with callbacks, but while it might have advantages, this way of doing things is actually less expressive than how ZeroMQ does it. If I had access to the full polling loop it would be easy for me to guarantee that the code I have written does not have any races in this example, but here that is obviously not the case.

This example is interesting as the workers get stored in a queue and the polling on the frontend receivers is only done when at least one of the workers is free.

Since the guide docs assured me that adding and removing sockets from the poller is thread safe what I did was create a little reactive switch like...

let switch_frontend =
    let remove () = poller.Remove(frontend_cloud); poller.Remove(frontend_local)
    let add () = poller.Add(frontend_local); poller.Add(frontend_cloud)
    remove ()
    let mutable old = false
    fun x -> if old <> x then (if x then add() else remove()); old <- x

...and hit this after every Enqueue and Dequeue. Nope. Bad idea.

For this example, and for the load balancing REQUEST-REQUEST one, how should they be implemented in NetMQ?

zeromq / netmq