Migrate IPC to tornado/asyncio

[thatch45]

The thing to do is migrate all of the IPC calls to tornado, now that tornado is a hard dep. The reformatting suggests that we can make IPC Mixins to handle much of this and refactor @jacksontj's new transports code to handle it.

I am making notes here as to how this can be done.

Also, we need a standard interface class for all of the IPC that will create sockets or localhost tcp connections or mailslots so that it works on windows. My guess is that localhost tcp sockets will be easier then mailslots

[thatch45]

On the PubServer: Only the push and pull interfaces need to be changed out, this should be a simple change to uxd stream sockets. All of the code that needs changing appears to be inside the tcp PubServer classes

[thatch45]

Event Bus This one is more major, the entire event bus code needs to be rewritten with tornado ipc socket. This means that salt/utils/events.py just needs to be rewritten. This SHOULD be the only code that needs changing, since all interfaces to the event bus should be using these libs. if there are libs that are not using it then we need to make them use it.

Also, we may be able to make this not fork an event process and manage all events in the main router process like it is currently done with raet

[thatch45]

https://github.com/saltstack/salt/blob/develop/salt/master.py#L489 Is this code ever called?

[jacksontj]

@thatch45 yes, 2 lines after the function (https://github.com/saltstack/salt/blob/develop/salt/master.py#L496) ;)

[jacksontj]

One thing to keep in mind is that my initial PR is just the beginning. It was solely focused on abstracting transport, so we still need to remove the rest of zmq and expose coroutine's to everything else. Sadly githubs issue tracker doesn't really do projects well, sine all of those tasks are more or less independent. On Apr 14, 2015 8:22 PM, "Thomas S Hatch" notifications@github.com wrote:

https://github.com/saltstack/salt/blob/develop/salt/master.py#L489 Is this code ever called?

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/22694#issuecomment-93173535.

[thatch45]

Right, I am just trying to figure out the particulars about how it all works and what exactly needs to be done, so we can help. And so I know better on how to get raet in there :)

[thatch45]

but yes, we should be able to rewrite the whole event system and it should work seemlessly with zmq tcp and raet

[jacksontj]

Do we want to make IPC pluggable? Or just make a TCP implementation? Different amounts of work ;) I was planning on just a TCP one since its local there should be no reason for raet (UDP reassembly in python for a domain socket seems like overkill) but we can if we want to. On Apr 14, 2015 10:16 PM, "Thomas S Hatch" notifications@github.com wrote:

but yes, we should be able to rewrite the whole event system and it should work seemlessly with zmq tcp and raet

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/22694#issuecomment-93192244.

[thatch45]

I don't think we need IPC to be pluggable, just something that will work on *NIX and windows. I think that if all transports could use the same IPC it would be the best.

So making a single event system with tornado would be fantastic for all transports to use.

[jacksontj]

Agreed :)

On Wed, Apr 15, 2015 at 7:36 AM, Thomas S Hatch notifications@github.com wrote:

I don't think we need IPC to be pluggable, just something that will work on *NIX and windows. I think that if all transports could use the same IPC it would be the best.

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/22694#issuecomment-93425288.

[thatch45]

Awesome, if we want to do that then we should wait on raet until that is in place but it would make it all work much more cleanly.

[jacksontj]

IMO rate is more complicated than we want/need for just IPC, we can do something with TCP/domain_sockets and it should be fine On Apr 15, 2015 8:50 AM, "Thomas S Hatch" notifications@github.com wrote:

Awesome, if we want to do that then we should wait on raet until that is in place but it would make it all work much more cleanly.

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/22694#issuecomment-93456874.

[thatch45]

right, no raet for ipc

[twangboy]

@thatch45 @jacksontj Did #25458 resolve this issue?

[thatch45]

Yes