search

memgraph / mage

MAGE - Memgraph Advanced Graph Extensions :crystal_ball:
Apache License 2.0
240 stars 23 forks source link

[Feature request] Add periodic commit module #438

Open double-di opened 6 months ago

double-di commented 6 months ago

Hi! We are in the middle of switching from neo4j to memgraph. We would love to see https://neo4j.com/labs/apoc/4.1/overview/apoc.periodic/apoc.periodic.commit/ in this module, since a lot of our queries are using it.

Thanks a lot!

karmenrabar commented 6 months ago

Hello @double-di ! Thank you for your interest in the periodic commit module. To better understand the situation, could you provide more details on the specific use case or scenarios where this module is being considered? Do you have a workaround for the moment? Your input will help us evaluate the impact and prioritization of the periodic commit module.

gitbuda commented 5 months ago

Related issue https://github.com/memgraph/memgraph/issues/1182

double-di commented 3 months ago

@karmenrabar Hi! Let me try to explain our usecase. Our database consist of 5B nodes & 5B relations. 100 M nodes are marked as "Starred". For any node we are trying to answer the question "Is it connected to a starred node with 16 hops or not? If it is connected - show any shortest path". Since the amount of nodes & relations is big it is not possible to answer this question in realtime by executing BFS directly. So for every "Starred" node we are precalculating (connecting with "CLOSE_TO_STARRED" relations to all the connected nodes) so we can answer this question fast. This process is very long and is implemented incrementally using flags "should_calculated_stared" on the nodes that should be calculated and apoc.periodic.commit to calculate a batch of nodes -> commit -> proceed

antejavor commented 3 months ago

Hi @double-di, thanks a lot for providing an in-depth explanation of this feature request 🚀 . Based on your explanation, I gave this feature request a bit higher priority.

How is your Memgraph experience overall?

Just for info, we are hosting Office hours calls. If you need any help from the Memgraph team in your journey or provide feedback, it could be worth while to jump on a short engineering-driven call.

double-di commented 3 months ago

Thanks a lot for your support @antejavor . We are currently on early adoption stage