scylladb / python-driver

ScyllaDB Python Driver, originally DataStax Python Driver for Apache Cassandra
https://python-driver.docs.scylladb.com
Apache License 2.0
70 stars 42 forks source link

Stop fallback to asyncio on python3.12 #312

Open fruch opened 5 months ago

fruch commented 5 months ago

during adding the support to python3.12 we took a call to fallback to asyncio once asyncore isn't available.

seems like doing so, we are running into issues that we didn't had coverage for, like using asyncio event loop, from actual async code. since we don't have such in the integration suite of this driver, nor in dtest.

it was hit only by scylla core test.py, that is build as async code

so we should do one of the two, or even both:

roydahan commented 5 months ago

Who should do this one?

fruch commented 5 months ago

Who should do this one?

Someone, if we wouldn't core developers are gonna hit by that once moving to python3.12

aisven commented 1 month ago

@fruch In the readme there is a statement that implies that this driver should not (yet) be used with python 3.12

https://github.com/scylladb/python-driver/blob/master/README.rst?plain=1#L13

Is this still true, or did you already implement support for python 3.12?

I am asking this, because I have a project based on python 3.12, asyncio, uvloop (but I could remove uvloop if needed), OSS Scylla 5, and I am considering to use your driver.

fruch commented 1 month ago

@fruch In the readme there is a statement that implies that this driver should not (yet) be used with python 3.12

https://github.com/scylladb/python-driver/blob/master/README.rst?plain=1#L13

Is this still true, or did you already implement support for python 3.12?

we do support python 3.12

I am asking this, because I have a project based on python 3.12, asyncio, uvloop (but I could remove uvloop if needed), OSS Scylla 5, and I am considering to use your driver.

if wouldn't recommend using the asyncio eventloop, it's still experimental, and some parts of it isn't working, for example SSL.

you can still write you code using asyncio, all of the other eventloops would work regardless.

aisven commented 3 weeks ago

@fruch Thank you for the reply. I like asyncio as my service is completely based on it, so I do not switch for now.

The service I am developing is streaming a lot of time series data from Scylla as a data source to some other data sink (think Kafka or TimescaleDB or whatever). Also, in one service instance I can configure to have not one, but multiple such streams running concurrently, i.e. one such stream per keyspace (there is one relevant table in each keyspace). Thus it is an interesting scenario with significant load.

The contents in each keyspace (relevant table) are large, but still in a ballpark that allows me to use just one stream per keyspace, i.e. I do not need to shard the streaming of the data regarding any particular keyspace.

I am making use of the normal good old paging facility of the driver API, so my design is effectively a type of micro-batched streaming.

Regrading the driver API I use:

  1. session.execute_async to execute a simple SELECT * query which does a paged full table scan
  2. future.result to eventually get the current rows of the current page
  3. result.fetch_next_page to forcibly and explicitly fetch the next page

I also employ facilities from asyncio to run synchronously waiting stuff on an executor and to properly await futures, respectively: