Error importing big database in orientdb

[rleon10]

Hello, I have done an export of 15 Databases in orientdb and im getting an error when I try to import one of them. The import of the rest 14 databases worked fine, all export files (xxx_export.json.gz) weight less than 1GB. But the DB which is giving me errors after importing all records and migrating indexes is 1,8GB.

• This is the error im getting

• I have tried to create the destiny DB as a remote and plocal (in both cases I get the error).

• I have increased the Xmx,Xms,Dstorage.diskCache.bufferSize and Dmemory.chunk.size

• I have used multiple EC2 machines with different CPUs and Memory (max CPUs and memory tested 18Cpus/ 60GB)

• I have tried to add the -rebuildIndexes = false flag when doing the import.

I dont know if there is a limit size for files, limited time to do an import, if im doing anything wrong or the Databases Im trying to import has some "corrupted" recods/indexes

Thank you so much in advance

[tglman]

Hi,

This seems related to the "dictionary" index that was a special index that was used as key value index in the past, can you try to just delete this index in the source database and try to export and re-import again ?

Regards

[rleon10]

Hi, I have removed the index "dictionary" that is the one which appears in the error message. After re-importing the database (doing a previous export of the database without the "dictionary" index) , the index appears again

I have also tried to rebuild the Index but it says "Cannot rebuild index 'dictionary' because it's manual and there aren't indications of what to index"

Also I have checked the others DB that I managed to import and all of them have the Index "dictionary" as "undefined", but their imports had no issues.

I have also tried to export with the flag -includeManualIndexes=FALSE, the error im getting is when Importing "manual" indexes

[tglman]

Hi,

By the way, what version are you using for export and what for import ?

Regards

[rleon10]

Hello, here is the configuration when importing for a r5.4xlarge EC2 machine (128G memory):

• Environment='ORIENTDB_OPTS_MEMORY=-Xms96g -Xmx96g -Dstorage.diskCache.bufferSize=32768 -Dmemory.chunk.size=4096'
• ansible>2.9
• image : Ubuntu-noble-24.04-amd64-server*
• OrientDB version: orientdb-server-2.2.31.jar
• openjdk-8-jre-headless

• update-java-alternatives -s java-1.8.0-openjdk-amd64

  Here is the configuration when exporting:

  • Environment='ORIENTDB_OPTS_MEMORY=-Xmx2000m -Xms2000m -Dstorage.diskCache.bufferSize=4000 -Dmemory.chunk.size=500 -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -XX:MaxDirectMemoryS
• ansible>=2.7.6,<2.8
• image : Ubuntu-18*
• OrientDB version: orientdb-server-2.2.31.jar
• openjdk-8-jre-headless
• update-java-alternatives -s java-1.8.0-openjdk-amd64

[tglman]

Hi,

I see, I do recall some issues with the importer in 2.2.x but unfortunately we do not do release updates anymore for the 2.2.x (has been out of support for a few years actually) is there any reason you do not update to a more recent version ?

Regards

[rleon10]

We are migrating to MongoDB but meanwhile we have to maintain orientdb 2.2.31

Regards

[tglman]

Hi,

I see, anyway is quite hard to help you on that old version, one attempt you can try is to unzip the export, which is just a big compressed json file, and try to remove the json block of manual indexes.

It could be interesting for me to know the reason why you are moving out of OrientDB, so feel free to share and if you prefer to do it in private feel free to reach me on matrix: https://matrix.to/#/@tglman:matrix.org or by mail at tglman_at_tglman.com