gevent support

[chicagobuss]

We'd like to use aerospike at a high rate of inserting with our asynchronous code but currently it doesn't work with gevent well.

[rbotzer]

This is still being researched and discussed. See this discussion thread: https://discuss.aerospike.com/t/gevent-compatibility-or-async-api/1001

[ramrengaswamy]

Any update here on what the plan is ?

[RonRothman]

+1 for async client. We're processing many thousands of requests concurrently--are we supposed to block them all every time we call out to Aerospike?

[rbotzer]

The likely chain of events is that the C client adds async support, which is then used for the clients built around it. Currently the Python client has been measured doing 4.5Ktps per-process, and in aggregate around 200Ktps. Among your options is to use the fairly fast synchronous client, contribute to the async effort, or wait.

[ramrengaswamy]

The 4.5K TPS roughly translates to ~200us for an operation. So under what kind of a controlled setup was the 4.5K TPS achieved ? What was the variance ? We are using Aerospike on EC2 machines running in a VPC where this seems highly unlikely to achieve consistently.

And more pertinent to this discussion is that using gevent in high performance asynchronous environment is a very common pattern which the python wrapper around the c client does not permit.

It would be good to have a sense of what the timing is around the planned async support in C client so that we can decide if we should wait or try something else. Thanks !

[rbotzer]

That was measured on enterprise grade bare-metal servers. I'm not sure which instance type of EC2 you're on, but yes, I'd guess its performance will lie somewhere between that on the upper end, and the 1.5Ktps that I get on my laptop.

We're looking into adding async to the C client, but the next part will be solving how to integrate this functionality with the Python client. Unfortunately Python has a myriad of async approaches. Getting it to work with gevent is something I would like to achieve. As for the timeline, I don't want to make promises I can't keep, so I won't. It's progressing.

[andersonbd1]

Ronan, I thought the libevent client essentially was the asynchronous C client. Are you saying that you're looking to add async to the C client apart from the libevent client?

[rbotzer]

The libevent client is asynchronous, but only does basic key-value operations. Yes, we are looking into adding async to the C client, alongside the existing synchronous functionality (similar to the Java and C# clients).

[RonRothman]

Thanks Ronen, we appreciate your responsiveness and candor.

Quick question: We noticed that the Python client does not release the GIL before calling out to Aerospike. Is this by design?

Also, just to clarify: Your benchmark of 4.5k TPS is impressive, but TBH you've missed the point.

4.5k per second averages to 220 usecs per call, but what is the latency distribution? Does every call take exactly 220 usecs? (No.) There is a tail, and at the end are some requests that lag. And when those requests lag, your synchronous Python client gums up all the thousands of requests that are in the system at that moment. Every request being serviced suffers because of a few slow Aerospike requests. This is not an acceptable risk. Hope this helps explain the urgency on our end.

[rbotzer]

I understand that the network roundtrip time that can be reclaimed, but you are still using Python and your code is going to execute significantly slower than the extension code. There are many good reasons to use Python, but. :grin:

Good point on releasing the GIL. That modification can be tested well before the C client work is in place. I'm going to look into it.

[rbotzer]

@RonRothman @ramrengaswamy In 1.0.56 we're releasing the GIL around the calls to the server, other than ones that involve callbacks (aerospike.Query.foreach and aerospike.Scan.foreach). Let me know how that works for you with gevent.

[RonRothman]

Thanks, that's great. Our current (multiprocess) version has been quite stable but I'd certainly prefer to keep the entire client in-process. When 1.0.56 is released and I can find a few hours, I'll switch to a multithreaded implementation and let you know how it goes. Thanks again.

[rbotzer]

@RonRothman I'd love to hear your feedback when you get the chance.

[RonRothman]

Thanks Ronen. I spent this week porting our nonblocking client wrapper from gipc+multiprocess to gevent+threads+1.0.56. We've launched it and it's doing well so far. If it handles server hiccups gracefully and if I find the time, we'll be happy to share it with the group.

On Wed, Dec 2, 2015 at 10:24 PM, Ronen Botzer notifications@github.com wrote:

@RonRothman https://github.com/RonRothman I'd love to hear your feedback when you get the chance.

— Reply to this email directly or view it on GitHub https://github.com/aerospike/aerospike-client-python/issues/52#issuecomment-161504917 .

[rbotzer]

@chicagobuss it seems that @RonRothman has the >= 1.0.56 client working with gevent.

[chicagobuss]

Awesome, I'll definitely try it out soon.

Thanks! On Dec 9, 2015 6:09 PM, "Ronen Botzer" notifications@github.com wrote:

@chicagobuss https://github.com/chicagobuss it seems that @RonRothman https://github.com/RonRothman has the >= 1.0.56 client working with gevent.

— Reply to this email directly or view it on GitHub https://github.com/aerospike/aerospike-client-python/issues/52#issuecomment-163442816 .

[chicagobuss]

Getting a segfault in the latest (1.0.57) and 1.0.56 python clients when I get to the client.put line in my simple test.. using server 3.7.0.1-1 on ubuntu on EC2.. kinda weird, don't remember problems like this before

import aerospike
config = { "hosts": [ ('stats01', 3000), ('stats01', 3000), ('stats02', 3000) ], "policies": { "timeout": 1000 } }
NAMESPACE = 'stats'
SET = 'testing'
client = aerospike.client(config).connect()
client.connect()
key = (NAMESPACE, SET, 'foo')
client.put( key, {'counter': 0} )

[RonRothman]

You're calling connect() twice there; is that intentional?

On Sun, Dec 13, 2015 at 11:56 PM, Joshua Buss notifications@github.com wrote:

Getting a segfault in the latest (1.0.57) and 1.0.56 python clients when I get to the client.put line in my simple test.. using server 3.7.0.1-1 on ubuntu on EC2.. kinda weird, don't remember problems like this before

import aerospike config = { "hosts": [ ('stats01', 3000), ('stats01', 3000), ('stats02', 3000) ], "policies": { "timeout": 1000 } } NAMESPACE = 'stats' SET = 'testing' client = aerospike.client(config).connect() client.connect() key = (NAMESPACE, SET, 'foo') client.put( key, {'counter': 0} )

— Reply to this email directly or view it on GitHub https://github.com/aerospike/aerospike-client-python/issues/52#issuecomment-164342740 .

[chicagobuss]

d'oh, not sure how that snuck in there, sorry for the noise. Thanks for the extra set of eyes.

On Sun, Dec 13, 2015 at 11:02 PM, Ron Rothman notifications@github.com wrote:

You're calling connect() twice there; is that intentional?

On Sun, Dec 13, 2015 at 11:56 PM, Joshua Buss notifications@github.com wrote:

Getting a segfault in the latest (1.0.57) and 1.0.56 python clients when I get to the client.put line in my simple test.. using server 3.7.0.1-1 on ubuntu on EC2.. kinda weird, don't remember problems like this before

import aerospike config = { "hosts": [ ('stats01', 3000), ('stats01', 3000), ('stats02', 3000) ], "policies": { "timeout": 1000 } } NAMESPACE = 'stats' SET = 'testing' client = aerospike.client(config).connect() client.connect() key = (NAMESPACE, SET, 'foo') client.put( key, {'counter': 0} )

— Reply to this email directly or view it on GitHub < https://github.com/aerospike/aerospike-client-python/issues/52#issuecomment-164342740

.

— Reply to this email directly or view it on GitHub https://github.com/aerospike/aerospike-client-python/issues/52#issuecomment-164343189 .

[rbotzer]

@chicagobuss well, it definitely shouldn't segfault. :grimacing:

[ddorian]

any update on this? @RonRothman can you release your code that works with gevent?

[alekseyl1992]

any progress on this? I would like to use aerospike from Tornado based asynchronous appilcation

[2mf]

Any updates on gevent? We would like to evaluate it with our gevent based environment.

[jboone100]

Can someone provide details on what is failing when using the 2.0.12 Python client with gevent?

[aviramha]

I started a new project called aioaerospike, pure Python async Aerospike library. https://github.com/aviramha/aioaerospike It's probably no-brainer to use gevent with it.