Run concurrent transactions

memgraph / gqlalchemy

GQLAlchemy is a library developed with the purpose of assisting in writing and running queries on Memgraph. GQLAlchemy supports high-level connection to Memgraph as well as modular query builder.

https://pypi.org/project/gqlalchemy/

Apache License 2.0

226 stars 32 forks source link

Run concurrent transactions #250

Open bilalsp opened 1 year ago

bilalsp commented 1 year ago

[New Feature Request]

Neo4j driver supports concurrent workflows to interact with the database in an asynchronous way as explained in documentation.

Documentation: Neo4j concurrent transactions

Apparently, gqlalchemy does not support concurrent workflows neither for Neo4j nor Memgraph. I would appreciate if we implement this feature in gqlalchemy compatible with Python’s asyncio.

katarinasupe commented 1 year ago

Hi @bilalsp, thank you for opening the feature request :) We considered implementing this in the future, so I will leave this issue open to better prioritize later.

Is this a blocker for you? What are you working on?

bilalsp commented 1 year ago

Hi @katarinasupe, we are doing planing for our new async microservice which we will be implementing in FastAPI framework. Graph database is really appealing for our usecase. So, we are considering all the aspects before making our decision to use graph database.

By the way, does Memgraph's python driver pymgclient support concurrent transactions similar to Neo4j driver? I didn't find any resource related to this.

alex-linx commented 1 year ago

@bilalsp Hi, I ran into this issue as well. Here is what I found out so far:

The only way to parallellize queries with pymgclient is by multiprocessing - using ProcessPoolExecutor, or a helper library such as joblib. It works well, but you are limited by the number of cpu cores.
Multithreading doesn't really work with pymgclient because apparently it doesn't release the GIL for I/O activities.
The best solution I found is to use the neo4j driver v5 which has asyncio support. I tested it with 1000 concurrent connections and it was extremely fast. Unfortunately, neo4j v5 is incompatible with gqlalchemy that requires v4.4, so they can't exist in the same python env.