Counting out_neighbours crashes

[meido]

Using the Pokec data, https://snap.stanford.edu/data/soc-pokec.html, which has about 1.6m nodes and 30m edges, we test loading time, how long it takes to count the degrees, out_neighbours, etc.

This query crashes:

MATCH (u:User)-[:Follows]->(n:User) RETURN u.id, COLLECT(n.id) AS out_neighbours Obtaining the data per node however works:

for user_id in range(1, max_user_id + 1): query = f"MATCH (u:User {{id: {user_id}}})-[:Follows]->(n:User) RETURN u.id, COLLECT(n.id) AS out_neighbours" res = self.conn.execute(query) df = res.get_as_df()

[mayil-ai[bot]]

Summary: Query to count out_neighbours crashes when executed on a large dataset with 1.6m nodes and 30m edges.

Possible Solution

Based on the provided information, the issue seems to be related to the execution of a query that collects out_neighbours for all User nodes, which crashes possibly due to resource constraints like memory when handling large result sets. The working code snippet suggests that executing the query for each User node individually does not cause a crash.

To address the issue:

• Implement pagination in the query to process a subset of User nodes at a time, reducing the memory footprint.
• Modify the query to include a SKIP and LIMIT clause to handle pagination.
• Adjust the LIMIT value based on the available memory and the size of the dataset to find a balance between performance and resource usage.

Here is an example of how you might modify the query to include pagination:

page_size = 1000  # Adjust the page size based on memory constraints and performance
skip = 0
while True:
    paginated_query = f"""
    MATCH (u:User)-[:Follows]->(n:User)
    RETURN u.id, COLLECT(n.id) AS out_neighbours
    SKIP {skip} LIMIT {page_size}
    """
    res = self.conn.execute(paginated_query)
    df = res.get_as_df()
    if df.empty:
        break
    # Process the DataFrame `df` as needed
    skip += page_size

• Ensure that the execute method in the DatabaseConnection class can handle the SKIP and LIMIT clauses without issues.
• Monitor the memory usage during the execution of the paginated queries to ensure that it stays within acceptable limits.
• If the database supports it, consider using an index on the User node's id attribute to improve the performance of the paginated queries.

Code snippets to check

• tools/python_api/test/test_df.py

  The snippet contains a query that retrieves data in a similar manner to the one causing the crash, which could help in understanding how the API handles large result sets.

• tools/python_api/test/test_networkx.py

  This snippet involves a query that returns a large graph structure, which may be relevant to understanding how the system handles complex queries and large data sets.

• tools/python_api/test/test_datatype.py

  The snippet includes a query with a variable-length path, which might be relevant for understanding how the system processes complex queries.