asyncio coroutines support

[rooterkyberian]

Hey, I'm wondering if I there is any plan to support exposing async coroutines? so that I can do something like this:

@expose
class Thingy(object):
   async def message(self, arg):
        print("Message received:", arg)
        return "Roger!"

I think it would fit nicely with Pyro5 being Python 3.5+ and all.

[irmen]

Maybe.

What you're showing is server-side async. I'm not sure yet how to combine that with the Pyro daemon and socket servers as they are now.

The first thing that has to be done to give async a chance is decoupling the protocol logic from the socket i/o. Some steps have already been taken but work is not complete, see for instance https://github.com/irmen/Pyro5/blob/master/Pyro5/protocol.py#L183

[tbhkuiper]

Has some progress been made on this? Asynchronous Pyro would solve a software design problem that is too much to explain here. I'll attach some pages of my work notes with details. RATworklog-227-228.pdf

[irmen]

Are you referring to an async server, or async pyro calls from a client? There's no async event loop integration at this time. For async client calls maybe you could wrap those yourself

[tbhkuiper]

On 8/16/20 5:15 AM, Irmen de Jong wrote:

Are you referring to an async server, or async pyro calls from a client? There's no async event loop integration at this time. For async client calls maybe you could wrap those yourself

I hope you haven't thought me rude in not acknowledging this.  In fact, I've been trying to make it work.

The problem that I'm encountering is that the server owns four threads that are running continuously.  The client sends a request to the server along with a callback.  This request causes the server to assign tasks to the threads.  Each thread wants to return data incrementally to the client until its task is done.  I tried to do that by having the thread invoke the callback given to its parent the server.  I think I attached a file to my original post with a figure that shows this.

So, of course, I ran into proxy ownership problem.  I found an example of claiming proxy ownership.  This would be a nice solution since a thread needs ownership for a little while and can then give it back.  My question is if that necessary or can all threads be co-owners?  If not, how do I give ownership back?

Maybe I'll just try the first (co-ownership) and see what happens.

My project, still in its early stages, is found at https://github.com/SDRAST and documented at https://sdrast.github.io/index.html. My questions concern the MonitorControl elements.

Hartelijk bedankt en beste wensen,

Tom

[tbhkuiper]

On 8/23/20 5:33 PM, Tom Kuiper wrote:

My question is if that necessary or can all threads be co-owners?  If not, how do I give ownership back?

Maybe I'll just try the first (co-ownership) and see what happens.

Never mind that.  I found the documentation which covers that. Thanks.  Very elegant and powerful package, by the way.

[irmen]

Bedankt!

Instead of claiming ownership and returning it, of a single proxy, you can copy the proxy and hand out a unique copy to each thread. This does move the problem to the server, possibly, where you will have to be able to handle concurrent calls from the client threads.

[tbhkuiper]

On 8/24/20 1:23 PM, Irmen de Jong wrote:

Instead of claiming ownership and returning it, of a single proxy, you can copy the proxy and hand out a unique copy to each thread. This does move the problem to the server, possibly, where you will have to be able to handle concurrent calls from the client threads.

I have the opposite situation.  The server has four threads trying to use the same callback, the one given to the main server thread, and tripping over each other.  I don't suppose there is a way for a thread to see if the callback is busy, and wait if necessary?

The idea of copying the proxy is interesting.  Could the main server thread give a copy of the callback to each of the child threads?  The client doesn't care about the order in which the data come back.  They all go into a queue that feeds a combiner.

I've updated the diagram and attached it.  The big picture is that there is a user with a browser at the left wanting to see spectra from N ROACH signal processors (on the right) handling parallel signals.

I'm curious.  If you are chatting or corresponding about this kind of technical stuff with another Hollander, what language do you use?  My astronomer colleagues tend to switch to English.

Ciao

Tom

[tbhkuiper]

On 8/24/20 4:56 PM, Tom Kuiper wrote:

Could the main server thread give a copy of the callback to each of the child threads?

That doesn't work on the server side.  There are no error messages from the server, but the client sees nothing on the queue of the callback object.

Met vriendelijke groet

Tom

[irmen]

If you are chatting or corresponding about this kind of technical stuff with another Hollander, what language do you use? My astronomer colleagues tend to switch to English.

Because this is on github as an open discussion (as in: free to read/participate in) I stick to English ! :)

[gkaissis]

In line with this issue‘s main question , is it possible to use gevent to just patch the stdlib sockets and achieve concurrency without threads?

[irmen]

Gevent won't magically turn a synchronous program into an asynch system.

I'm closing this issue - I won't be changing Pyro5 internally myself to support this. I could see a PR with a new additional socketserver implementation based on an async event loop mechanism, but I'm not going to write that myself in the forseeable future

[gkaissis]

Thank you for the insight!