Add support for a synchronous client

tobias-urdin commented 1 year ago

This issue is here to track the possible work of implementing a synchronous client in nats.py that can be used instead of the asyncio client that this library supports today.

Some projects might do their own worker implementations and need a client library that does not do asyncio and only does blocking operations.

Based on a older version of nats.py I've forked it and implemented a simple PoC for that functionality here https://github.com/tobias-urdin/nats.py/commits/sync-client that is missing a lot of error checking and testing.

Moving common code, constants, abstract classes etc from the aio client to a new nats.common module
Implement a new synchronous TcpTransport
Implement a copy of the Parser as SyncParser since it needs to call functions and not await coroutines
Implementing a new nats.sync module that implements SyncClient class that does blocking operations instead of using asyncio
Some caveats is that the PoC does a lot of code duplication for Msg, Subscription etc since we need to call a synchronous function in there

This fork has been used for implement a PoC oslo.messaging driver so that NATS can be used as the messaging bus for an OpenStack cloud https://review.opendev.org/c/openstack/oslo.messaging/+/878588 where the Devstack job is passing.

Looking for feedback on a approach that works for you, any potential help and anything else that might help push it forward.

bruth commented 1 year ago

Thanks @tobias-urdin. Looping in @wallyqs to provide guidance.

aqc-carlodri commented 1 year ago

I would also be very interested in this and it would unblock the migration of our project entirely towards NATS.

Lancetnik commented 1 year ago

Waiting for this changes too! Also ready to help with some parts of implementation to help release sync version soon

tobias-urdin commented 11 months ago

@wallyqs Any feedback on this? It's basically what we talked about quite a few months back now but need some direction for moving forward.

robin-coac commented 11 months ago

Hi @tobias-urdin, Sorry to bug you. I am new to writing asynchronous code.

Do you think this issue I have posted resonate with what you are saying ? https://github.com/nats-io/nats.py/issues/482

Because I am unable to run multiple worker processes in parallel with nats.

electronick commented 10 months ago

+1 on this change. Will help to user celery with nats.

AlejandroUPC commented 9 months ago

Is this still going on? It seems quite people is interested, should we organize somehow?

tobias-urdin commented 9 months ago

I'm not actively working on this because of priorities but is still something I want to do.

It would be awesome if more people would like to coordinate this work but it's my opinion that we would need some directions from @bruth and @wallyqs on how they want this implemented for it to be acceptable, we wouldn't want to do all this work and afterwards get feedback on the approach.

aqc-carlodri commented 7 months ago

Hi everyone, friendly ping 😊: is there anything we can do to get the ball rolling on this?

charbonnierg commented 5 months ago

For what it's worth, I tried to come up with a Sans-IO protocol implementation in python. This article from the author of the websocket library really rang a bell while I read this thread so I tried to come up with a similar solution for NATS.

The code is available on github: https://github.com/charbonnierg/natsproto, it's far from being robust or correct at the moment, but I believe that this is a very promising approach for supporting both a synchronous and an asynchronous client without reiventing the wheel. Any feedback would be greatly appreciated.

I also tried in this repository to write an async client without relying on asyncio, which would make it possible to run the client using trio (assuming a transport implemented on top of anyio also exists). TBH I was quite lost at the beginning, especially regarding cancellation 😅 but in the end I believe that writing code using structured concurrency rather than future-based approach + callbacks is easier to read and understand.

So to conclude there are 3 aspects in this proof of concept:

a sans IO protocol
an async client
a sync client (poorly written)

I'm willing to work on my free time on the sans IO protocol up until there is a 100% coverage and more features such as validating subjects, validating max payload size, validating URL schemes, support for verbose mode, etc..., assuming that you're interested in this @wallyqs. This could take from a few weeks to a month. Biggest effort would be porting all existing protocol tests I guess. We could then come up with a very simple sync client and refactor the existing async client to use the sans-IO protocol. Only then would I consider introducing breaking changes to the async client to use anyio instead of asyncio, if you want this.

pySilver commented 3 months ago

I think the best approach right now is to use https://github.com/airtai/faststream/ - yep its an additional layer of code but it works out of the box.

aqc-carlodri commented 3 months ago

I think the best approach right now is to use https://github.com/airtai/faststream/ - yep its an additional layer of code but it works out of the box.

does it support sync operations? It seems to me like it is still async-only.

pySilver commented 3 months ago

I think the best approach right now is to use https://github.com/airtai/faststream/ - yep its an additional layer of code but it works out of the box.

does it support sync operations? It seems to me like it is still async-only.

It works both ways

nats-io / nats.py

Add support for a synchronous client #462