crashes on large syncing

[niktverd]

Dear, SvenWesterlaken!

I have faced a problem syncing 6k docs.

Everything works well on tens docs. But when I started syncing a little bigger amount of docs mongo4j stops my express server at all. I have disconected mongo4j, added all the docs into mongo, reconnected mongo4j and faced timeout limitation.

(on video you can see how elastic search synced all the docs and mongo4j crashes after 60 seconds) https://youtu.be/kefmJE2S6dQ

I have changed config, as shown below, but there was no effect

const mongo4j = require("mongo4j");
console.log("mongo4j is executiong....");
// mongo4j.init('neo4j://localhost', {user: 'neo4j', pass: '123'});
mongo4j.init(
    "bolt://localhost",
    { user: "neo4j", pass: "123" },
    {
        encrypted: "ENCRYPTION_OFF",
        maxConnectionLifetime: 3 * 60 * 60 * 1000, // 3 hours
        maxConnectionPoolSize: 150,
    }
);

const mongo4jDriver = mongo4j.getDriver();

console.log("mongo4jDriver", mongo4jDriver);

// create session
const s = mongo4jDriver.session();

// transaction
(async () =>{
    r = await s.writeTransaction(tx => f(tx), {
        timeout: 99999999 // define timeout
    });

})()

module.exports = mongo4j;

Do you have any idea how to overcome this obstacle?

PS. I have changed "neo4j://...." on "bolt://...." in driver config, because of this message Neo4jError: Could not perform discovery. No routing servers available. Known routing table: RoutingTable[database=default database, expirationTime=0, currentTime=1605783019325, routers=[], readers=[], writers=[]] as said in this article

[SvenWesterlaken]

This looks more like an issue on neo4j side, have you looked at any issues on the neo4j driver repo? Anyway, it could also be that it's bad on my part and that the configuration is not properly coming through as the timeout in your video shows 60000 which doesn't correspond to your number in the code above.

I actually had problems with the neo4j:// as well, but bolt should be fine to use. The newer neo4j protocol is just the same as the bolt one, but with routing added in or something, I've read. Are you running your neo4j locally or in something like Docker? because I fixed that problem for the GitHub action that releases this package which uses the Docker container of neo4j.

As with the other issues, the first moment I can look at this would be this evening.

[SvenWesterlaken]

Also, from the top of my head, are you saving each document separately into MongoDB? Because if you use <your schema>.insertMany(documents) this will create a single query to save all the documents for both MongoDB and neo4j, which should improve performance in any case.

[niktverd]

I think you are right.

I used desktop for linux neo4j.

I reinstalled neo4j from deb.repo. Launched it from shell. Increased all the parameters in neo4j.conf related to heap size to 10Gb.

And everything became ok

Thank you for you attention.

Can you close the issue?

[SvenWesterlaken]

Sure, glad you can find use in this repo btw!