Bug Report

I got the following error when I attempt to populate the Neo4j database using a huge amount of sessions.

Here is the full log:

 ❯ cogralys-bench-util populate-neo4j -h "bolt://host.docker.internal:7687"
DANGER: TLS certificate validation is disabled for all hostnames
populating db: 28.63%                                                    243.4s 10287/35934 
<--- Last few GCs --->

[1864:0x558f7c8f6560]   247936 ms: Mark-Compact (reduce) 1398.1 (1426.7) -> 1397.4 (1423.5) MB, 14.67 / 0.00 ms  (+ 326.0 ms in 2399 steps since start of marking, biggest step 1.8 ms, walltime since start of marking 355 ms) (average mu = 0.246, current mu

<--- JS stacktrace --->

#
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
#

zsh: trace trap  deno run --config /workspaces/bench-source/deno.jsonc --allow-all  --unstable

Note that cogralys-bench-util is a custom typescript (Deno) command, used on my benchmark repository (used in my PhD). It is not currently open source, but I can share the 35_934 json files used with this command.

The code is run inside a worker, here is the code:

import neo4j, { Driver, Session } from "npm:neo4j-driver@5.10.0";

function errorToString(e) {
    if (Object.keys(e).every(v => ["message", "name", "code"].includes(v))) {
        return `${e.message}\n${e.stack}\n  name: ${e.name}\n  code:${e.code}`
    }
    return e;
}

let driver: Driver;

const work = (self as any as Worker);

work.onmessage = async ({ data }: { data: { taskId: string, data: { path: string, url: string, username: string, password: string, skip?: boolean, query?: string } } }) => {
    if (data.data.skip) {
        work.postMessage({ taskId: data.taskId, result: "", status: "success", path: data.data.path });
        return;
    }
    if (!driver) {
        driver = neo4j.driver(data.data.url, neo4j.auth.basic(data.data.username, data.data.password));
    }

    const session: Session = driver.session();

    try {
        const queries = JSON.parse(data.data.query || Deno.readTextFileSync(data.data.path)).statements;

        try {
            for (const query of queries) {
                const result = await session.run(query.statement.replace(/CREATE(\s*)\(/g, "MERGE$1("), query.parameters);
                work.postMessage({ taskId: data.taskId, result: result, status: "success", path: data.data.path });
            }
        } catch (error) {
            work.postMessage({ taskId: data.taskId, result: errorToString(error), status: "failure", path: data.data.path });
        } finally {
            session.close();
        }
    } catch (error) {
        work.postMessage({ taskId: data.taskId, result: errorToString(error), status: "failure", path: data.data.path });
    }

};

The command cogralys-bench-util populate-neo4j create as many workers as there are cpus (here, 8). And sends job to workers with a path to a JSON file, that contains things like:

{
    "statements": [
        {
            "statement": "UNWIND $rows AS row\nMATCH (start:`UNIQUE IMPORT LABEL`{node_id: row.start._id})\nMATCH (end:`UNIQUE IMPORT LABEL`{node_id: row.end._id})\nCREATE (start)-[r:CORRESPONDING_FIRST_SUBTYPE]->(end) SET r += row.properties;",
            "parameters": {
                "rows": [
                    {
                        "start": {
                            "_id": "/workspaces/bench-source/src/aaa/config/aaa_config.ads:17:4:AN_ORDINARY_TYPE_DECLARATION"
                        },
                        "end": {
                            "_id": "/workspaces/bench-source/src/aaa/config/aaa_config.ads:17:4:AN_ORDINARY_TYPE_DECLARATION"
                        },
                        "properties": {}
                    },
                    {
                        "start": {
                            "_id": "/usr/gnat/libexec/asis-gnsa/lib/gcc/x86_64-pc-linux-gnu/8.3.1/adainclude/interfac.ads:46:4:AN_ORDINARY_TYPE_DECLARATION"
                        },
                        "end": {
                            "_id": "/usr/gnat/libexec/asis-gnsa/lib/gcc/x86_64-pc-linux-gnu/8.3.1/adainclude/interfac.ads:46:4:AN_ORDINARY_TYPE_DECLARATION"
                        },
                        "properties": {}
                    },
                    {
                        "start": {
                            "_id": "/workspaces/bench-source/src/aaa/src/aaa-traits-types.ads:3:4:A_FORMAL_TYPE_DECLARATION"
                        },
                        "end": {
                            "_id": "/workspaces/bench-source/src/aaa/src/aaa-traits-types.ads:3:4:A_FORMAL_TYPE_DECLARATION"
                        },
                        "properties": {}
                    }
                ]
            }
        }
    ]
}

Note that the data in the previous JSON is a sample; in fact, there is a quite more elements in parameters.rows array.

Each of 8 workers create a driver and keep it alive along jobs. Every times it receive a job (path to JSON file) it parse it and sends every statements in a separate session. The script run inside a Docker container, and communicate with the Neo4j desktop on the host.

My Environment

Javascript Runtime Version: I got this error with: deno 1.34.3 (release, x86_64-unknown-linux-gnu) v8 11.5.150.2 typescript 5.0.4

But I also had this error in the past with Node (v18.16.1)

Driver Version: 5.10.0 Neo4j Version and Edition: Neo4j version 5.3.0 entreprise (Neo4j Desktop, Version 1.5.8 (1.5.8.105)) Operating System: macOS 13.4 (22F66)

neo4j / neo4j-javascript-driver

Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit #1105

Bug Report

My Environment