Support old binary formats.

To support old binary formats we can use following approach:

Read storage configuration.
If storage configuration version is different from latest, look at list of migration scripts which will be used to migrate db data.
Perform migration automatically, with ability to track migration progress using listeners.

Subtasks:

Integration of component version concept into storage components
OMigration interface
Migrator implementation
API and console command for upgrade task
API and console command that show version of binary data structure
Automatic tests for testing backward compatibility
Add compatibility for
- 1.5.0
- 1.5.1
- 1.5.2
- 1.6.0
- 1.6.1
- 1.6.2
- 1.6.3
Implement upgrade tasks for:
- 1.5.0
- 1.5.1
- 1.5.2
- 1.6.0
- 1.6.1
- 1.6.2
- 1.6.3

It is needed more than 24 hours to update public repository could you try http://helios.orientechnologies.com/job/orient-maven/lastStableBuild/ ?

On Thu, Mar 6, 2014 at 4:11 PM, valenpo notifications@github.com wrote:

@laa https://github.com/laa https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-community/1.7-rc2-SNAPSHOT/

— Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-36890748 .

Best regards, Andrey Lomakin.

Orient Technologies the Company behind OrientDB

@laa I tested with provided repo with odb-community.

Get such results. (maybe I'm doing something wrong and significant changes from 1.5.1 to 1.7.x) but I use same name libraries and same code. With 1.5.1 i have no any error.

мар 07, 2014 12:57:18 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'database.ocf' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.0.oda' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.odh' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.1.oda' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.0.oda' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.odh' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'internal.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'internal.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'manindex.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'manindex.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orole.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orole.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ouser.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ouser.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ofunction.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ofunction.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'oschedule.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'oschedule.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'e.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'e.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'user.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'user.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'folder.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'folder.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'item.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'item.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'thread.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'thread.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'txlog.otx' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: Current implementation of storage does not support sbtree collections мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Wait till indexes restore after crash was finished. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: Current implementation of storage does not support sbtree collections мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@EMAIL is not automatic index and will be added as is. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@EMAIL was added in DB index list. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Start creation of index ORole.name мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index ORole.name was successfully created and rebuild is going to be started. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Rebuild of ORole.name index was successfully finished. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_NAME is not automatic index and will be added as is. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_NAME was added in DB index list. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@ID is not automatic index and will be added as is. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@ID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index OID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index OID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index EMAIL is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index EMAIL was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index mgn is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index mgn was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_ROOT_ID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_ROOT_ID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index dictionary is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index dictionary was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@___USER_NAME is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@_USER_NAME was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index ID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index ID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@_OID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index _@recordmap@OID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@mgn is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@mgn was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@_USER_ROOT_ID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@_USER_ROOT_ID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Start creation of index OUser.name мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index OUser.name was successfully created and rebuild is going to be started. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Rebuild of OUser.name index was successfully finished. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: 15 indexes were restored successfully, 0 errors мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Indexes restore after crash was finished. Exception in thread "main" com.orientechnologies.orient.core.exception.ODatabaseException: Database 'local:/Library/Application Support/MailArchiva/ROOT/database/archiva.db' is closed at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.checkOpeness(ODatabaseRecordAbstract.java:1170) at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:713) at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:64) at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:203) at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:39) at com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:520) at com.tinkerpop.blueprints.impls.orient.OrientIndex.buildKeyValueIndex(OrientIndex.java:208) at com.tinkerpop.blueprints.impls.orient.OrientIndex.load(OrientIndex.java:190) at com.tinkerpop.blueprints.impls.orient.OrientIndex.(OrientIndex.java:57) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.loadIndex(OrientBaseGraph.java:916) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.loadManualIndexes(OrientBaseGraph.java:902) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.readDatabaseConfiguration(OrientBaseGraph.java:242) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:99) at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:58) at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:54) at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:34) at com.stimulus.archiva.database.blueprints.TestCaseDB.main(TestCaseDB.java:21)

Hi I see

WARNING: segment file 'oschedule.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.0.ocl' was not closed correctly last time

Which means that database was not closed correctly.

How did you access to database ? Do you stop application, close server make copy of database and open it using distribution downloaded from http://helios.orientechnologies.com/job/orient-maven/lastStableBuild/ ?

As I wrote libraries place on sonatype repository not up to date/ Do you have some issues which does not allow you to download distribution from official OrientDB build server ?

On Fri, Mar 7, 2014 at 11:00 AM, valenpo notifications@github.com wrote:

@laa https://github.com/laa I tested with provided repo with odb-community.

Get such results. (maybe I'm doing something wrong and significant changes from 1.5.1 to 1.7.x) but I use same name libraries and same code. With 1.5.1 i have no any error.

мар 07, 2014 12:57:18 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'database.ocf' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.0.oda' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.odh' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.1.oda' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.0.oda' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.odh' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'internal.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'internal.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'manindex.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'manindex.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orole.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orole.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ouser.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ouser.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ofunction.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ofunction.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'oschedule.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'oschedule.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'e.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'e.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'user.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'user.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'folder.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'folder.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'item.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'item.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'thread.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'thread.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'txlog.otx' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: Current implementation of storage does not support sbtree collections мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Wait till indexes restore after crash was finished. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: Current implementation of storage does not support sbtree collections мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index _@recordmap@_EMAIL is not automatic index and will be added as is. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@__EMAIL was added in DB index list. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log

INFO: Start creation of index ORole.name мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index ORole.name was successfully created and rebuild is going to be started. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Rebuild of ORole.name index was successfully finished. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log

WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_NAME is not automatic index and will be added as is. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index USERNAME was added in DB index list. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@_ID is not automatic index and will be added as is. мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@__ID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

INFO: Index OID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

INFO: Index OID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index EMAIL is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index EMAIL was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index mgn is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index mgn was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

INFO: Index USER_ROOT_ID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

INFO: Index USER_ROOT_ID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

INFO: Index dictionary is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

INFO: Index dictionary was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index _@recordmap@USERNAME is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@USER_NAME was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index ID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index ID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index _@recordmap@OID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index _@recordmap@_OID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@mgn is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index _@recordmap@mgn was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index _@recordmap@__USER_ROOTID is not automatic index and will be added as is. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index @recordmap@USER_ROOT_ID was added in DB index list. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Start creation of index OUser.name мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Index OUser.name was successfully created and rebuild is going to be started. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: Rebuild of OUser.name index was successfully finished. мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log INFO: 15 indexes were restored successfully, 0 errors мар 07, 2014 12:57:20 PM com.orientechnologies.common.log.OLogManager log

INFO: Indexes restore after crash was finished. Exception in thread "main" com.orientechnologies.orient.core.exception.ODatabaseException: Database 'local:/Library/Application Support/MailArchiva/ROOT/database/archiva.db' is closed at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.checkOpeness(ODatabaseRecordAbstract.java:1170) at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:713) at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:64) at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:203) at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:39) at com.orientechnologies.orient.core.record.impl.ODocument.field(ODocument.java:520) at com.tinkerpop.blueprints.impls.orient.OrientIndex.buildKeyValueIndex(OrientIndex.java:208) at com.tinkerpop.blueprints.impls.orient.OrientIndex.load(OrientIndex.java:190) at com.tinkerpop.blueprints.impls.orient.OrientIndex.(OrientIndex.java:57) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.loadIndex(OrientBaseGraph.java:916) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.loadManualIndexes(OrientBaseGraph.java:902) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.readDatabaseConfiguration(OrientBaseGraph.java:242) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:99) at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:58) at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:54) at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:34) at com.stimulus.archiva.database.blueprints.TestCaseDB.main(TestCaseDB.java:21)

— Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-36978623 .

Best regards, Andrey Lomakin.

Orient Technologies the Company behind OrientDB

Andrey

When our server is shutdown, the blueprints method:

IndexableGraph.shutdown();

is always called. Our assumption is that this method will close the Orient DB database. Is this correct?

Jamie

On 2014/03/07, 12:00 PM, Andrey Lomakin wrote:

Hi I see

WARNING: segment file 'oschedule.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.0.ocl' was not closed correctly last time

Which means that database was not closed correctly.

I did all things as a post's above.

zip DB on running server without DB activity
unzip it locally
test it with TestCase 1.5.1 libs (above) all is fine
remove DB
unzip DB again and
update libs to 1.7.x from http://helios.orientechnologies.com/job/orient-maven/lastStableBuild/ libs
test it with TestCase (above) all is not fine

HI, No, it closes connection not storage. If you use embedded storage you should stop JVM, otherwise server should be stopped.

On Fri, Mar 7, 2014 at 12:06 PM, jamieb22 notifications@github.com wrote:

Andrey

When our server is shutdown, the blueprints method:

IndexableGraph.shutdown();

is always called. Our assumption is that this method will close the Orient DB database. Is this correct?

Jamie

On 2014/03/07, 12:00 PM, Andrey Lomakin wrote:

Hi I see

WARNING: segment file 'oschedule.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.0.ocl' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.och' was not closed correctly last time мар 07, 2014 12:57:19 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.0.ocl' was not closed correctly last time

Which means that database was not closed correctly.

—

Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37010854 .

Best regards, Andrey Lomakin.

Orient Technologies the Company behind OrientDB

Do you stop server before you update libs ? right ?

On Fri, Mar 7, 2014 at 12:08 PM, valenpo notifications@github.com wrote:

I did all things as a post's above.

zip DB on running server without DB activity

unzip it locally

test it with TestCase 1.5.1 libs (above) all is fine

remove DB

unzip DB again and

update libs to 1.7.x from http://helios.orientechnologies.com/job/orient-maven/lastStableBuild/ libs

test it with TestCase (above) all is not fine

— Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37011007 .

Best regards, Andrey Lomakin.

Orient Technologies the Company behind OrientDB

As far as I know, we do. We shutdown the server, which in turn shutdown the JVM.

On 2014/03/07, 12:12 PM, Andrey Lomakin wrote:

Do you stop server before you update libs ? right ?

On Fri, Mar 7, 2014 at 12:08 PM, valenpo notifications@github.com wrote:

I did all things as a post's above.

zip DB on running server without DB activity

unzip it locally

test it with TestCase 1.5.1 libs (above) all is fine

remove DB

unzip DB again and

update libs to 1.7.x from

http://helios.orientechnologies.com/job/orient-maven/lastStableBuild/ libs

test it with TestCase (above) all is not fine

— Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37011007 .

This is TestCase so server is doesn't matter. Server only once did DB 4 month ago, and it is not perform any action now.

We are working with same DB shanshot in any cases, difference is only libs (1.5.1 1.7.x).

Please read some posts above.

package com.stimulus.archiva.database.blueprints;

import com.orientechnologies.orient.core.config.OGlobalConfiguration; import com.tinkerpop.blueprints.Index; import com.tinkerpop.blueprints.IndexableGraph; import com.tinkerpop.blueprints.Vertex; import com.tinkerpop.blueprints.impls.orient.OrientGraph; import com.tinkerpop.gremlin.java.GremlinPipeline;

public class TestCaseDB {

public static void main(String[] args) {

    String USER_ROOT_ID = "USER_ROOT_ID";

    OGlobalConfiguration.CACHE_LEVEL1_ENABLED.setValue(false);
    OGlobalConfiguration.DB_USE_DISTRIBUTED_VERSION.setValue(false);
    OGlobalConfiguration.NON_TX_RECORD_UPDATE_SYNCH.setValue(true); //Executes a synch against the file-system at every record operation. This slows down records updates but guarantee reliability on unreliable drives
    OGlobalConfiguration.TX_LOG_SYNCH.setValue(true); //Executes a synch against the file-system for each log entry. This slows down transactions but guarantee transaction reliability on non-reliable drives
    IndexableGraph graph = new OrientGraph("local:/Library/Application Support/MailArchiva/ROOT/database/archiva.db");

    try {
        Index<Vertex> index = graph.getIndex(USER_ROOT_ID, Vertex.class);

        Iterable<Vertex> foundVertices = index.get(USER_ROOT_ID, "USER_ROOT");
        Vertex root = foundVertices.iterator().next();

        GremlinPipeline<Vertex, ?> pipe = new GremlinPipeline();
        pipe.start(root).out("user");

        while (pipe.hasNext()) {
            Vertex vertex = (Vertex) pipe.next();   
            System.out.println(vertex.getProperty("ID"));
        }
    } finally {
        graph.shutdown();
    }
}

}

Warning WARNING: segment file 'oschedule.och' was not closed correctly last time

means that processed database for sure was not closed properly.

Could you do following.

Stop server, not zip db from server without activity (absence of activity does not mean that all changes which are in memory present on the disk too) but stop server and only then zip db.
unzip DB
update libs to 1.7.x from http://helios.orientechnologies.com/job/orient-maven/lastStableBuild/ libs
test it with TestCase (which we ran on our side without problems on 1.5.1 version of database.)

On Fri, Mar 7, 2014 at 12:15 PM, valenpo notifications@github.com wrote:

This is TestCase so server is doesn't matter. Server only once did DB 4 month ago, and it is not perform any action now.

We are working with same DB shanshot in any cases, difference is only libs (1.5.1 1.7.x).

Please read some posts above.

package com.stimulus.archiva.database.blueprints;

import com.orientechnologies.orient.core.config.OGlobalConfiguration; import com.tinkerpop.blueprints.Index; import com.tinkerpop.blueprints.IndexableGraph; import com.tinkerpop.blueprints.Vertex; import com.tinkerpop.blueprints.impls.orient.OrientGraph; import com.tinkerpop.gremlin.java.GremlinPipeline;

public class TestCaseDB {

public static void main(String[] args) {
String USER_ROOT_ID = "USER_ROOT_ID";

OGlobalConfiguration.CACHE_LEVEL1_ENABLED.setValue(false);
OGlobalConfiguration.DB_USE_DISTRIBUTED_VERSION.setValue(false);
OGlobalConfiguration.NON_TX_RECORD_UPDATE_SYNCH.setValue(true); //Executes a synch against the file-system at every record operation. This slows down records updates but guarantee reliability on unreliable drives
OGlobalConfiguration.TX_LOG_SYNCH.setValue(true); //Executes a synch against the file-system for each log entry. This slows down transactions but guarantee transaction reliability on non-reliable drives
IndexableGraph graph = new OrientGraph("local:/Library/Application Support/MailArchiva/ROOT/database/archiva.db");

try {
    Index<Vertex> index = graph.getIndex(USER_ROOT_ID, Vertex.class);

    Iterable<Vertex> foundVertices = index.get(USER_ROOT_ID, "USER_ROOT");
    Vertex root = foundVertices.iterator().next();

    GremlinPipeline<Vertex, ?> pipe = new GremlinPipeline();
    pipe.start(root).out("user");

    while (pipe.hasNext()) {
        Vertex vertex = (Vertex) pipe.next();
        System.out.println(vertex.getProperty("ID"));
    }
} finally {
    graph.shutdown();
}
}

}

—

Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37011490 .

Best regards, Andrey Lomakin.

Orient Technologies the Company behind OrientDB

@laa could you please tell me what exactly libraries are you using and what code are you using to check that DB is opened correctly. Because if 1.5.1 opened TestDB fine (in any mode, with wrong close) but 1.7.x doesn't so this mean 1.7.x is not backward compatible with 1.5.1.

You open database 1.5.1 for local storage which is closed incorrectly , binary compatibility works since 1.7-rc2 and for plocal storage and as special case for 1.5.1

For local storage impossible to restore data after crash and that is what you reported yourself and that is why you have plans to update to plocal storage. Right now you open local storage which is closed incorrectly and 1.7 can not restore it, but 1.5.1 even does not try to restore it, does not make any attempt to restore that is why you do not see any errors. How is supposed to support binary compatibility for possibly broken storage which can not be restored by design ?

Did you try to close storage correctly (as I described) and run test ?

On Fri, Mar 7, 2014 at 12:52 PM, valenpo notifications@github.com wrote:

@laa https://github.com/laa could you please tell me what exactly libraries are you using and what code are you using to check that DB is opened correctly. Because if 1.5.1 opened TestDB fine (in any mode, with wrong close) but 1.7.x doesn't so this mean 1.7.x is not backward compatible with 1.5.1.

— Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37013889 .

Best regards, Andrey Lomakin.

Orient Technologies the Company behind OrientDB

@laa ok, I opened DB with latest shapshot. (properly closed DB). I know nothing about core and how is it stored, so I believe that it must be ;) Thanks!

@laa Andrey, hi. After integrating to main product, I have some problems with

empty DB startup with plocal format. Startup for me take about 20-30 minutes with CPU load 100%. And I not wait for the end of start up process.
Export db to file is fine, but import importing nothing.

Here is code, that used mainly for DB startup, convert (export, import).

I suppose it could be library missing or something. But could you tell what libraries must be used and what for, maybe other decencies.

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.nio.file.attribute.FileAttribute;
import java.nio.file.attribute.PosixFilePermission;
import java.nio.file.attribute.PosixFilePermissions;
import java.util.Date;
import java.util.List;
import java.util.Locale;
import java.util.Set;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.locks.ReentrantLock;

import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.orientechnologies.common.concur.ONeedRetryException;
import com.orientechnologies.orient.core.command.OCommandOutputListener;
import com.orientechnologies.orient.core.config.OGlobalConfiguration;
import com.orientechnologies.orient.core.db.ODatabaseRecordThreadLocal;
import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;
import com.orientechnologies.orient.core.db.record.ODatabaseRecord;
import com.orientechnologies.orient.core.db.tool.ODatabaseExport;
import com.orientechnologies.orient.core.db.tool.ODatabaseImport;
import com.stimulus.archiva.blob.BlobID;
import com.stimulus.archiva.domain.Application;
import com.stimulus.archiva.domain.Database;
import com.stimulus.archiva.domain.DatabaseException;
import com.stimulus.archiva.domain.ItemNotFoundException;
import com.stimulus.archiva.domain.database.Item;
import com.stimulus.archiva.domain.database.RootItem;
import com.stimulus.archiva.domain.database.TreeItem;
import com.stimulus.archiva.domain.database.TreeLink;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.Index;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.Parameter;
import com.tinkerpop.blueprints.TransactionalGraph;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.impls.orient.OrientGraph;

public class BluePrintDatabase extends Database {

    // protected OrientGraph graph = null;
    protected static Logger logger;
    public String location;
    private Path dbLocationPath;
    private String BACKUP_DB_FOLDER = "archiva_backup.db";
    protected AtomicBoolean connected = new AtomicBoolean();
    // protected OGraphDatabase graphDatabase = null;
    // protected ODatabaseRecord oDataRecord = null;
    protected IndexableGraph graph;
    protected ODatabaseRecord record;
    protected ReentrantLock indexLock = new ReentrantLock();

    static {

    }

    public BluePrintDatabase(String location, Application application) {
        setApplication(application);
        this.location = "plocal:" + location;
        this.dbLocationPath = Paths.get(location);
        logger = LoggerFactory.getLogger(getApplication().getLoggerName()+"."+BluePrintDatabase.class);
    }

    private void initGraph(){
        OGlobalConfiguration.CACHE_LEVEL1_ENABLED.setValue(false);
        OGlobalConfiguration.DB_USE_DISTRIBUTED_VERSION.setValue(false);
        OGlobalConfiguration.NON_TX_RECORD_UPDATE_SYNCH.setValue(true); //Executes a synch against the file-system at every record operation. This slows down records updates but guarantee reliability on unreliable drives
        OGlobalConfiguration.TX_LOG_SYNCH.setValue(true); //Executes a synch against the file-system for each log entry. This slows down transactions but guarantee transaction reliability on non-reliable drives
        graph = new OrientGraph(location);
    }

    protected void dropAutoIndex(String index) {
        graph.dropIndex(index);
    }

    /**
     * Returns an Index from IndexableGraph. If no such index exist, it create it and return.
     *
     * @param  indexedProperty  an  key that used for search index or create index with key name
     * @return           created or exist index from IndexableGraph
     * @see              IndexableGraph 
     */
    protected Index<Vertex> getIndex(IndexedProperty indexedProperty){
        return getIndex(parseIndex(indexedProperty));
    }

    /**
     * Returns an Index from IndexableGraph. If no such index exist, it create it and return.
     *
     * @param  indexedProperty  an  key that used for search index or create index with key name
     * @return           created or exist index from IndexableGraph
     * @see              IndexableGraph 
     */
    private Index<Vertex> getIndex(String indexedProperty){
        indexLock.lock();
        try {

            Index<Vertex> index = graph.getIndex(indexedProperty, Vertex.class);
            if (index == null) {
                index = graph.createIndex(indexedProperty,  Vertex.class, new Parameter<String, String>("", "")); 
            }
            return index;
        }finally { 
            indexLock.unlock();
        }
    }

    /**
     * @param indexedProperty
     */
    protected String parseIndex(IndexedProperty indexedProperty) {
        return indexedProperty.toString();
    }

    @Override
    public void connect() throws DatabaseException, ItemNotFoundException {
        if (connected.compareAndSet(false, true)) {
            if (needConvert()) {
                try {
                    convertDB();
                } catch (IOException ioe) {
                    logger.debug("failed convert database format: "+ioe.getMessage());
                }
            } else {
                initGraph();
            }
            record = ODatabaseRecordThreadLocal.INSTANCE.get();
            insertUserRoot();
        }
    }

    private boolean needConvert() {
        return false;
    }

    private void convertDB() throws FileNotFoundException, IOException {
            Path dbBackupLocationPath = Paths.get(dbLocationPath.getParent().toString()+File.separator+BACKUP_DB_FOLDER);
            if (!Files.exists(dbBackupLocationPath, LinkOption.NOFOLLOW_LINKS)) {
                Set<PosixFilePermission> perms = PosixFilePermissions.fromString("rwxr-x---");
                FileAttribute<Set<PosixFilePermission>> attr = PosixFilePermissions.asFileAttribute(perms);
                dbBackupLocationPath = Files.createDirectory(dbBackupLocationPath, attr);
                Files.move(dbLocationPath, dbBackupLocationPath, StandardCopyOption.REPLACE_EXISTING);
                Files.createDirectory(dbLocationPath, attr);
            }

            File file = new File(dbBackupLocationPath.toString()+File.separator+"export.gz");

            exportDB(file, dbBackupLocationPath.toString());
            importDB(file);
            FileUtils.deleteDirectory(dbBackupLocationPath.toFile());
    }

    private void exportDB(File exportFile, String dbLocation) throws FileNotFoundException, IOException {

        ODatabaseDocumentTx iDatabase = new ODatabaseDocumentTx("local:"+dbLocation);
        iDatabase.open("admin", "admin");

        ODatabaseExport export = new ODatabaseExport(iDatabase, exportFile.getAbsolutePath(), new OCommandOutputListener() {
            @Override
            public void onMessage(String arg0) {
                logger.debug("export db: "+arg0);
            }
        });
        export.exportDatabase();
        export.close();

        iDatabase.close();
    }

    private void importDB(File importFile) throws FileNotFoundException, IOException {
            ODatabaseDocumentTx database = new ODatabaseDocumentTx(location);
            database.create();
            ODatabaseImport dbImport;
            try (FileInputStream fis = new FileInputStream(importFile)) {
                dbImport = new ODatabaseImport(database, fis, new OCommandOutputListener() {
                    @Override
                    public void onMessage(String arg0) {
                        logger.debug("importing db: "+arg0);
                    }
                });
            };

            dbImport.setPreserveRids(true);
            dbImport.setDeleteRIDMapping(false);
            dbImport.importDatabase();
            dbImport.close();

            database.close();

            initGraph();

            Iterable<Vertex> vertexes = graph.getVertices();
            for (Vertex vertex : vertexes) {
                for (String key : vertex.getPropertyKeys()) {
                    if (isIndexed(key)) {
                        addToIndex(IndexedProperty.valueOf(key), vertex.getProperty(key), vertex);
                    }
                }
            }
    }

    @Override
    public void commit() {
    }

    protected void insertUserRoot() throws DatabaseException, ItemNotFoundException {
        BluePrintRootItem item = (BluePrintRootItem) insertItem(Database.ItemType.ROOT, Database.IndexedProperty.USER_ROOT_ID , Database.USER_ROOT_ID );
        item.setName("Users");

        commitTransaction();
    }

    public Graph getConnection() {
        return graph;
    }

    @Override
    public void close() throws DatabaseException {
        if (connected.compareAndSet(true, false)) {
            graph.shutdown();
            record = null;
        }
    }

    protected ItemType getItemTypeFromInt(int ItemTypei) {
        ItemType ItemType = Database.ItemType.BLOB;
        switch (ItemTypei) {
        case 0:
            ItemType = Database.ItemType.ROOT;
            break;
        case 1:
            ItemType = Database.ItemType.USER;
            break;
        case 2:
            ItemType = Database.ItemType.FOLDER;
            break;
        case 3:
            ItemType = Database.ItemType.BLOB;
            break;
        case 4:
            ItemType = Database.ItemType.TAG;
            break;
        }

        return ItemType;
    }

    private Item getItem(ItemType type) throws DatabaseException {
        return getItem(null, type);
    }

    private TreeItem getItem(Vertex vertex, Database.ItemType type) throws DatabaseException {
//      if (vertex == null) {
//          throw new DatabaseException("assertion failure: vertex is null.");
//      }
        switch (type) {
        case ROOT:
            return vertex == null ? new BluePrintRootItem() : new BluePrintRootItem(this, vertex);
        case BLOB:
            return vertex == null ? new BluePrintArchiveItem() : new BluePrintArchiveItem(this, vertex);
        case FOLDER:
            return vertex == null ? new BluePrintFolderItem() : new BluePrintFolderItem(this, vertex);
        case USER:
            return vertex == null ? new BluePrintUserItem() : new BluePrintUserItem(this, vertex);
        case TAG:   
            return vertex == null ? new BluePrintTagItem() : new BluePrintTagItem(this, vertex);
        case PROPERTY:
            return vertex == null ? new BluePrintPropertyItem() : new BluePrintPropertyItem(this, vertex);
        }

        throw new DatabaseException( "unknown item ItemType" );
    }

    TreeItem getItem(Vertex vertex) throws DatabaseException {
        if (vertex == null) {
            throw new DatabaseException("assertion failure: vertex is null.");
        }
        Integer itemTypei = (Integer) vertex.getProperty(Database.ITEMTYPE);

        if (itemTypei == null) {
            throw new DatabaseException("assertion failure: vertex contains an invalid type.");
        }
        ItemType type = getItemTypeFromInt(itemTypei);
        TreeItem item = getItem(vertex, type); 

        return item;
    }

    private TreeItem newItem(Vertex vertex, Database.ItemType type, String oid) throws DatabaseException {
        if (vertex == null) {
            throw new DatabaseException("assertion failure: vertex is null.");
        }

        TreeItem item = getItem(vertex, type);
        item.setProperty(Database.ITEMTYPE, type.ordinal());
        return item;
    }

    public Graph getGraph() {
        return graph;
    }

    public Database getDatabase() {
        return this;
    }

    private boolean isAssociatedItem(Vertex parentVertex, Vertex childVertex, Relationship relationship) throws DatabaseException {
        if (parentVertex == null) {
            throw new DatabaseException("assertion failure: parent vertex is null.");
        }
        Iterable<Edge> edges = childVertex.getEdges(Direction.IN, relationship.toString().toLowerCase(Locale.ENGLISH));

        for (Edge edge : edges) {
            if (edge.getVertex(Direction.OUT).getId().equals(parentVertex.getId())) {
                return true;
            }
        }
        return false;
    }

    @Override
    public TreeItem insertItem(ItemType type, IndexedProperty key, String id) throws DatabaseException {

        if (!connected.get()) {
            throw new DatabaseException("cannot insert tree item. not connected to database");
        }

        try {
            Vertex vertex = getVertex(type, id);
            return getItem(vertex);
        } catch (DatabaseException de) {
            // vertex doesn't exist, so create!
        } catch (ItemNotFoundException e) {
            // vertex doesn't exist, so create!
        }

        Vertex vertex = graph.addVertex(null);
        TreeItem item = newItem(vertex, type, id);

        if ( key != null ) {
            item.setProperty(key.toString(), id);
        }

        item.setType(type);

        return item;
    }

    @Override
    public TreeItem updateItem(ItemType type, String id) throws DatabaseException {

        if (!connected.get()) {
            throw new DatabaseException("cannot insert tree item. not connected to database");
        }
        Vertex vertex = null;
        try {
            vertex = getVertex(type, id);
            return getItem(vertex);
        } catch (DatabaseException de) {
            // vertex doesn't exist, so create!
        } catch (ItemNotFoundException e) {
            // vertex doesn't exist, so create!
        }
        vertex = graph.addVertex(null);
        TreeItem item = newItem(vertex, type, id);
        return item;
    }

    @Override
    public TreeItem updateItem(ItemType type, IndexedProperty key, String id)   throws DatabaseException {

        if (!connected.get()) {
            throw new DatabaseException("cannot insert tree item. not connected to database");
        }
        Vertex vertex = null;
        try {
            vertex = getVertex(key, id);
            return getItem(vertex);
        } catch (DatabaseException de) {
            // vertex doesn't exist, so create!
        } catch (ItemNotFoundException e) {
            // vertex doesn't exist, so create!
        }
        vertex = graph.addVertex(null);
        TreeItem item = newItem(vertex, type, id);
        return item;
    }

    @Override
    public void deleteItem(ItemType type, String id) throws DatabaseException,  ItemNotFoundException {

        if (!connected.get()) {
            throw new DatabaseException("cannot delete tree item. not connected to database");
        }

        Vertex vertex = getVertex(type, id);

        removeVertex(vertex);

    }

    @Override
    public void deleteItem(IndexedProperty key, String id)  throws DatabaseException, ItemNotFoundException {

        if (!connected.get()) {
            throw new DatabaseException("cannot delete tree item. not connected to database");
        }
        Vertex vertex = getVertex(key, id);
        removeVertex(vertex);
    }

    /**
     * @param vertex
     */
    private void removeVertex(Vertex vertex) {

        for (Edge edge : vertex.getEdges(Direction.BOTH, null)) {
            edge.remove();
        }

        graph.removeVertex(vertex);

        for (String key : vertex.getPropertyKeys()) {
            if (isIndexed(key)) {
                Object value = vertex.getProperty(key);
                removeFromIndex(key, value, vertex);
            }
        }

        commitTransaction();
    }

    @Override
    public void deleteItem(Item item) throws DatabaseException, ItemNotFoundException {
        Vertex vertex = (Vertex) item.getRawElement();
        removeVertex(vertex);
    }

    @Override
    public void deleteItems(ItemType type, List<BlobID> ids) throws DatabaseException, ItemNotFoundException {
        for (BlobID blobID : ids) {
            deleteItem(type, blobID.getID());
        }
    }

    private Vertex getVertex(ItemType type, String value) throws DatabaseException, ItemNotFoundException {

        Item item = getItem(type);

        Set<String> indexedKeys = item.getIndexedProperties();

        for (String key : indexedKeys) {

            Index<Vertex> index = getIndex(key);
            Iterable<Vertex> foundVertices = index.get(key, value);

            if (foundVertices == null) {
                continue;
            }

            for (Vertex v : foundVertices) {
                return v;
            }
        }

        throw new ItemNotFoundException("vertex not found {property='" + StringUtils.join(indexedKeys.toArray()) + "',value='" + value + "'}");

    }

    private Vertex getVertex(IndexedProperty key, String value) throws DatabaseException, ItemNotFoundException {

        Index<Vertex> index = getIndex(key);
        Iterable<Vertex> foundVertices = index.get(parseIndex(key), value);

        for (Vertex v : foundVertices) {
            return v;
        }

        throw new ItemNotFoundException("vertex not found {property='" + key + "',value='" + value + "'}");

    }

    @SuppressWarnings({ "rawtypes" })
    protected Iterable getVertices(String property, String value) throws ItemNotFoundException {

        Index<Vertex> index = getIndex(property);
        Iterable<Vertex> foundVertices = index.get(property, value);

        if (foundVertices == null) {
            throw new ItemNotFoundException("vertex not found {property='" + property + "',value='" + value + "'}");
        }

        return foundVertices;

    }

    @Override
    public TreeLink associateItem(Item sourceItem, Item destItem, Relationship relationship) throws DatabaseException,  ItemNotFoundException {
        if (!connected.get()) {
            throw new DatabaseException("cannot associate tree item. not connected to database");
        }
        Vertex parentVertex = getVertex(sourceItem);
        Vertex childVertex = getVertex(destItem);

        Edge edge = associateItem(parentVertex, childVertex, relationship, false);

        return new BluePrintTreeLink(edge, getDatabase());
    }

    /**
     * trying get vertex from Item. 
     * @param sourceItem
     * @throws DatabaseException
     * @throws ItemNotFoundException
     */
    private Vertex getVertex(Item item) throws DatabaseException, ItemNotFoundException {
        Vertex vertex = null;
        for (String indexedKey : item.getIndexedProperties()) {
            Object indexedProperty = item.getProperty(indexedKey);
            if (indexedProperty instanceof Iterable<?>) {
                for (Object property : (Iterable<?>) indexedProperty) {
                    vertex = getVertex(item.getType(), (String) property);
                    if (vertex!=null) {
                        return vertex;
                    }
                }   
            } else {
                vertex = getVertex(item.getType(), (String) indexedProperty);
                if (vertex!=null) {
                    return vertex;
                }
            }

        }

        throw new ItemNotFoundException( "cant find Item on DB" );
    }

    @Override
    public TreeLink associateItem(ItemType sourceType, String sourceId, ItemType destType, String destId, Relationship relationship) throws DatabaseException,  ItemNotFoundException {

        if (!connected.get()) {
            throw new DatabaseException("cannot associate tree item. not connected to database");
        }

        Vertex parentVertex = getVertex(sourceType, sourceId);

        Vertex childVertex = getVertex(destType, destId);

        Edge edge = associateItem(parentVertex, childVertex, relationship, false);

        return new BluePrintTreeLink(edge, getDatabase());
    }

    @Override
    public TreeLink associateItem(IndexedProperty sourceKey, String sourceId,
            IndexedProperty destKey, String destId, Relationship relationship)
            throws DatabaseException, ItemNotFoundException {

        if (!connected.get()) {
            throw new DatabaseException("cannot associate tree item. not connected to database");
        }

        Vertex parentVertex = getVertex(sourceKey, sourceId);
        Vertex childVertex = getVertex(destKey, destId);
        Edge edge = associateItem(parentVertex, childVertex, relationship, false);

        return new BluePrintTreeLink(edge, getDatabase());
    }

    @Override
    public TreeLink associateItem(IndexedProperty sourceKey, String sourceId,
            IndexedProperty destKey, String destId, Relationship relationship, boolean additionalEdge)
            throws DatabaseException, ItemNotFoundException {

        if (!connected.get()) {
            throw new DatabaseException("cannot associate tree item. not connected to database");
        }

        Vertex parentVertex = getVertex(sourceKey, sourceId);
        Vertex childVertex = getVertex(destKey, destId);

        Edge edge = associateItem(parentVertex, childVertex, relationship, additionalEdge);

        return new BluePrintTreeLink(edge, getDatabase());

    }

    private Edge associateItem(Vertex parentVertex, Vertex childVertex, Relationship relationship, boolean additionalEdge) throws DatabaseException, ItemNotFoundException {

        if (!connected.get()) {
            throw new DatabaseException("cannot associate tree item. not connected to database");
        }

        Edge edge = null;

        if (additionalEdge) {
            edge = graph.addEdge(null, parentVertex, childVertex, relationship.toString().toLowerCase(Locale.ENGLISH));
            edge.setProperty(Database.CREATION_DATE.toString().toLowerCase(Locale.ENGLISH), new Date());
        } else if (!isAssociatedItem(parentVertex, childVertex, relationship)) {
            edge = graph.addEdge(null, parentVertex, childVertex, relationship.toString().toLowerCase(Locale.ENGLISH));
            edge.setProperty(Database.CREATION_DATE.toString().toLowerCase(Locale.ENGLISH), new Date());
        }

        commitTransaction();
        return edge;
    }

    public int countVertex(Iterable<Vertex> vertices) {
        int i = 0;
        for (@SuppressWarnings("unused") Vertex vertex : vertices) {
            i++;
        }
        return i;
    }

    public int countEdge(Iterable<Edge> edges) {
        int i = 0;
        for (@SuppressWarnings("unused") Edge edge : edges) {
            i++;
        }
        return i;
    }

    private boolean isItemType(ItemType type, TreeItem item) {
        switch (type) {
        case ROOT:
            return (item instanceof BluePrintRootItem);
        case USER:
            return (item instanceof BluePrintUserItem && (!(item instanceof BluePrintRootItem)));
        case FOLDER:
            return (item instanceof BluePrintFolderItem);
        case BLOB:
            return (item instanceof BluePrintArchiveItem);
        case TAG:
            return (item instanceof BluePrintTagItem);
        default:
            break;
        }
        return false;
    }

    @Override
    public TreeItem getItem(ItemType type, String id) throws DatabaseException, ItemNotFoundException {

        if (!connected.get()) {
            throw new DatabaseException("cannot retrieve tree item. not connected to database");
        }

        try {  // orient can throw null ptr exception
            Vertex vertex = getVertex(type, id);

            return getItem(vertex);

        } catch (ItemNotFoundException nfe) {
            throw nfe;
        } catch (Exception e) {
            throw new DatabaseException("error retrieving item from db:"+e.getMessage(),e,logger);
        }
    }

    @Override
    public TreeItem getItem(IndexedProperty key, String id) throws DatabaseException, ItemNotFoundException {

        if (!connected.get()) {
            throw new DatabaseException("cannot retrieve tree item. not connected to database");
        }

        try {  // orient can throw null ptr exception
            Vertex vertex = getVertex(key, id);

            return getItem(vertex);

        } catch (ItemNotFoundException nfe) {
            throw nfe;
        } catch (Exception e) {
            throw new DatabaseException("error retrieving item from db:"+e.getMessage(),e,logger);
        }
    }

    @Override
    public void clear() {
        /*try {
            // TODO add clear database implementation 
            getGraph().clear();
            dropAutoIndex();
        } catch (Exception e) {
            logger.error("failed to drop database:" + e.getMessage(), e);
        }*/
        try {
            connect();
        } catch (Exception e) {
            logger.error("failed to connect to database:" + e.getMessage(), e);
        }

    }

    @Override
    public synchronized void commitTransaction() {
        int attempt = 0;
        int maxAttempts = 100;
        while(attempt!=maxAttempts) {
            try {
                ((TransactionalGraph) graph).commit();
                break;
            } catch (ONeedRetryException e) {
                attempt++;
                if (attempt==maxAttempts) {
                    logger.debug("commit DB changes fails: "+e.getMessage());
                }

            }
        }
    }

    @Override
    public void declineTransaction() {
        ((TransactionalGraph) graph).rollback();
    }

    @Override
    public Item getItem() {
        // TODO Auto-generated method stub
        return null;
    }

    @Override
    public RootItem getRoot() throws DatabaseException, ItemNotFoundException {
        return (RootItem) getDatabase().getItem(ItemType.ROOT, Database.USER_ROOT_ID);
    }

    void addToIndex(IndexedProperty indexedProperty, Object value, Vertex vertex) {
        Index<Vertex> index = getIndex(indexedProperty);
        if (value instanceof Iterable<?>) {
            for (Object keyValue : (Iterable<?>) value) {
                index.put(parseIndex(indexedProperty), keyValue, vertex);
            }
        } else {
            index.put(parseIndex(indexedProperty), value, vertex);
        }
    }

    /**
     * remove from index key, value pair.
     * @param key
     * @param value
     */
    void removeFromIndex(IndexedProperty key, String value) {
        try {
            Vertex vertex = getVertex(key, value);
            removeFromIndex(key, value, vertex);
        } catch (DatabaseException | ItemNotFoundException e) {
            logger.debug("remove from db index fails: "+e.getMessage());
        }
    }

    protected void removeFromIndex(String indexKey, Object value, Vertex vertex) {
        Index<Vertex> index = getIndex(indexKey);
        index.remove(indexKey, value, vertex);
    }

    protected void removeFromIndex(IndexedProperty key, Object value, Vertex vertex) {
        removeFromIndex(parseIndex(key), value, vertex);
    }

    public boolean isIndexed(String key) {
        try {
            return IndexedProperty.valueOf(key) != null ? true : false;
        } catch (Exception e) {
            return false;
        }
    }

    public boolean isIndexed(IndexedProperty key) {
        return isIndexed(parseIndex(key)); 
    }

}

I tried it too. Starting from a fresh database, Orient DB drags the performance of the entire machine to a halt and takes forever to start up. Its almost as if it gets stuck in a tight loop. When apply kill -9 to the process and start it again, it starts up. However, when I import some email folder structures into the database, nothing shows up in the tree view. It seems Orient DB is not returning any information from the index. No matter if items have been added to the index, they are not retrievable.

Ok. After downloading the very latest nightly, I am able to start the server, however, the indexes do not appear to be functioning correctly. Despite having inserted a vertex called "USER_ROOT", Iterable foundVertices = index.get(key, value) returns null after additional items have been added to the database. If I call index.get(key, value) immediately after the item has been added, it does returns vertices. However, after additional operations on the database, it would appear, the index gets corrupted.

@laa Andrey, hello Can I ask about format version

com.orientechnologies.orient.core.db.record.OCurrentStorageComponentsFactory#binaryFormatVersion ,and com.orientechnologies.orient.core.storage.OStorage#getComponentsFactory.

What is a right way of using those components, do I need open/init DB 1st, and if yes what is a proper way to do it?

Also is it possible to check either local or plocal is used?

Regards Valentin

Hi,

It is holiday today so I will look through all the questions tomorrow. In general it seems like you skipped step after upgrade, please note that local storage is not used for graph database in 1.7-rc2. Support of 1.5.1 means that you can open database, upgrade it to plocal and work as you did it before (of course that is only for 1.5.1 version, we already changed binary format in 1.7rc2 for records presentation but no one noticed that because of binary compatibility change), that is done (plocal only support for grpahdb) because previous graph version was not thread safe and any way new version of edges presentation is faster :-) .

Did you do steps which we suggested you:

Open database.
Export to JSON using com.orientechnologies.orient.core.db.tool.ODatabaseExport
Import from JSON using com.orientechnologies.orient.core.db.tool.ODatabaseImport
Run upgrade graph using com.orientechnologies.orient.graph.migration.ORidBagMigration

After that it will work as expected.

I think we can provide support to work with graphdb 1.5.1 version in runtime but what the point in such mode if we will have mutithereading problems ? Please understand me correctly it is possible to add such support but will it be reasonable ? Is not better to migrate to 1.7-rc2 and continue to work with this version ?

On Sun, Mar 9, 2014 at 7:08 AM, valenpo notifications@github.com wrote:

@laa https://github.com/laa Andrey, hello Can I ask about format version

com.orientechnologies.orient.core.db.record.OCurrentStorageComponentsFactory#binaryFormatVersion ,and com.orientechnologies.orient.core.storage.OStorage#getComponentsFactory.

What is a right way of using those components, do I need open/init DB 1st, and if yes what is a proper way to do it?

Also is it possible to check either local or plocal is used?

Regards Valentin

— Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37119300 .

Best regards, Andrey Lomakin.

Orient Technologies the Company behind OrientDB

Andrey

The problem is that 1.7-rc2 is not working starting from a fresh database, with plocal storage enabled. As stated, there is a problem with the indexes. When we store an item, and later try to retrieve the item from the index, it is not present. Please refer to bug report.

Regards

Jamie

On 2014/03/09, 8:46 AM, Andrey Lomakin wrote:

Hi,

It is holiday today so I will look through all the questions tomorrow. In general it seems like you skipped step after upgrade, please note that local storage is not used for graph database in 1.7-rc2. Support of 1.5.1 means that you can open database, upgrade it to plocal and work as you did it before (of course that is only for 1.5.1 version, we already changed binary format in 1.7rc2 for records presentation but no one noticed that because of binary compatibility change), that is done (plocal only support for grpahdb) because previous graph version was not thread safe and any way new version of edges presentation is faster :-) .

Did you do steps which we suggested you:

Open database.

Export to JSON using com.orientechnologies.orient.core.db.tool.ODatabaseExport

Import from JSON using com.orientechnologies.orient.core.db.tool.ODatabaseImport

Run upgrade graph using com.orientechnologies.orient.graph.migration.ORidBagMigration

After that it will work as expected.

I think we can provide support to work with graphdb 1.5.1 version in runtime but what the point in such mode if we will have mutithereading problems ? Please understand me correctly it is possible to add such support but will it be reasonable ? Is not better to migrate to 1.7-rc2 and continue to work with this version ?

On Sun, Mar 9, 2014 at 7:08 AM, valenpo notifications@github.com wrote:

@laa https://github.com/laa Andrey, hello Can I ask about format version

com.orientechnologies.orient.core.db.record.OCurrentStorageComponentsFactory#binaryFormatVersion ,and

com.orientechnologies.orient.core.storage.OStorage#getComponentsFactory.

What is a right way of using those components, do I need open/init DB 1st, and if yes what is a proper way to do it?

Also is it possible to check either local or plocal is used?

Regards Valentin

— Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37119300 .

Best regards, Andrey Lomakin.

Orient Technologies the Company behind OrientDB

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37120508.

Hi Jamie, Your issue has 2 reasons:

First you id not call com.orientechnologies.orient.graph.migration.OGraphMigration (previously called ORidBagMigration).
There was issue with import/export of index definitions for graph database (fixed).

I tried to look into code which you provided, but ... it is 800 lines of code in unknown business domain without any entry point or description, could you copy paste only relevant methods and provide description of what they do ?

Here is example how migration should like :

 final ODatabaseDocumentTx database = new ODatabaseDocumentTx("local:/home/andrey/Development/orientdb/archiva.db");
    database.open("admin", "admin");

    final ODatabaseExport databaseExport = new ODatabaseExport(database, "/home/andrey/Development/orientdb/archiva.db.json", new OCommandOutputListener() {
      @Override
      public void onMessage(String iText) {
        System.out.println(iText);
      }
    });

    databaseExport.exportDatabase();
    databaseExport.close();

    final ODatabaseDocumentTx newDatabase = new ODatabaseDocumentTx("plocal:/home/andrey/Development/orientdb/new.archiva.db");
    newDatabase.create();

    final ODatabaseImport databaseImport = new ODatabaseImport(newDatabase, "/home/andrey/Development/orientdb/archiva.db.json.gz", new OCommandOutputListener() {
      @Override
      public void onMessage(String iText) {
        System.out.println(iText);
      }
    });
    databaseImport.importDatabase();

    database.close();
    databaseImport.close();

    database.open("admin", "admin");

    OGraphMigration graphMigration = new OGraphMigration(newDatabase);
    graphMigration.execute();
    database.close();

About Valentin questions.

You do not need to create database components manually here is wiki page with description how it works https://github.com/orientechnologies/orientdb/wiki/Backward-compatibility
plocal has different file extensions https://github.com/orientechnologies/orientdb/wiki/plocal-storage-components , for example "internal" cluster always exists so you can detect presence of internal.pcl file to check that you work with local database.

@laa thank you! We will try it today, so if it will not work we write correct testCase

Regards Valentin

@valenpo good, wait 30 min I will push changes. Out of curiosity in xUSSR today is holiday, so what is your geographical location ?

@laa yup I know that today is holiday ;) but working. I'm from Moscow, and you?

I am from Dnepropetrovsk, it is Ukraine.

yup i know ;), I live 4 years on Ukraine Chernigov

@laa Andrey, where is best point to download latest dis't to be sync with you sources?

I think you should pull it from github, you can wait an hour and download distribution from CI server but it is too long, better just pull and run mvn clean package -DskipTests and you find distribution in ... distribution project :-) . Or you can install it the same way to your maven repository. But wait 10 min I need to run tests myself before push.

@laa Andrey is this a correct location where do you pushed changes https://github.com/orientechnologies/orientdb.git ? If yes, what is a branch name to pull? Regards Valentin

@laa I used develop branch

@valenpo yes you are right.

@laa Andrey, hello. Adding/getting vertexes from index is fine. But I follow you instruction of DB conversion. Have some misunderstandings:

when I export DB I used path name like "…./archiva.db.json"
when importing it is on you example "…./archiva.db.json.gz" Why such files has different names? Export auto add gz compression? If YES so can I use 1 file name for export and import like "…./archiva.db.json.gz"

Regards Valentin

@valenpo yes you can use single name with .gz at the end.

@laa Andrey, I follow exactly same code for conversion, and cant confirm that conversion is working.
Old DB size is ~1GB new one after conversion is 1.4M I trying convert this DB http://yadi.sk/d/gaOcvd0EKK4en. Archive is 189MB

@valenpo Yes, this is too small :-) I will try to convert it too. What is you console output ? Did you try to compare databases ?

I just run same conversion program as you provide.

мар 11, 2014 2:54:46 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'database.ocf' was not closed correctly last time мар 11, 2014 2:54:46 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.0.oda' was not closed correctly last time мар 11, 2014 2:54:46 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.odh' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.1.oda' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.0.oda' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.odh' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'internal.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'internal.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'index.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'manindex.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'manindex.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'default.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orole.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orole.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ouser.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ouser.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ofunction.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'ofunction.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'oschedule.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'oschedule.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'orids.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'v.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'e.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'e.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'user.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'user.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'folder.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'folder.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'item.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'item.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'thread.0.ocl' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'thread.och' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: segment file 'txlog.otx' was not closed correctly last time мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: Current implementation of storage does not support sbtree collections мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log INFO: Wait till indexes restore after crash was finished. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: Current implementation of storage does not support sbtree collections мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_NAME is not automatic index and will be added as is. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_NAME was added in DB index list. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log INFO: Index EMAIL is not automatic index and will be added as is. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log INFO: Index EMAIL was added in DB index list. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_ROOT_ID is not automatic index and will be added as is. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_ROOT_ID was added in DB index list. мар 11, 2014 2:54:47 PM com.orientechnologies.common.log.OLogManager log INFO: Start creation of index OUser.name мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index OUser.name was successfully created and rebuild is going to be started. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Rebuild of OUser.name index was successfully finished. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index dictionary is not automatic index and will be added as is. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index dictionary was added in DB index list. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index ID is not automatic index and will be added as is. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index ID was added in DB index list. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index OID is not automatic index and will be added as is. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index OID was added in DB index list. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Start creation of index ORole.name мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index ORole.name was successfully created and rebuild is going to be started. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Rebuild of ORole.name index was successfully finished. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index mgn is not automatic index and will be added as is. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Index mgn was added in DB index list. мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: 9 indexes were restored successfully, 0 errors мар 11, 2014 2:54:48 PM com.orientechnologies.common.log.OLogManager log INFO: Indexes restore after crash was finished.

Started export of database 'archiva.db' to /Library/Application Support/MailArchiva/ROOT/database/archiva.db.json.gz...

Exporting database info... OK

Exporting clusters... OK (15 clusters)

Exporting schema... OK (14 classes)

Exporting records...

Cluster 'internal' (id=0)... OK (records=3/3)
Cluster 'index' (id=1)... OK (records=4/4)
Cluster 'manindex' (id=2)... . . . . . . . . . . OK (records=2489/2489)
Cluster 'default' (id=3)... OK (records=0/0)
Cluster 'orole' (id=4)... OK (records=3/3)
Cluster 'ouser' (id=5)... OK (records=3/3)
Cluster 'ofunction' (id=6)... OK (records=0/0)
Cluster 'oschedule' (id=7)... OK (records=0/0)
Cluster 'orids' (id=8)... OK (records=0/0)
Cluster 'v' (id=9)... . . . . . . . . . . OK (records=256952/256952)
Cluster 'e' (id=10)... OK (records=0/0)
Cluster 'user' (id=11)... . . . . . . . . . . OK (records=137/137)
Cluster 'folder' (id=12)... . . . . . . . . . . OK (records=3566/3566)
Cluster 'item' (id=13)... . . . . . . . . . . OK (records=546636/546636)
Cluster 'thread' (id=14)... . . . . . . . . . . OK (records=84041/84041)

Done. Exported 893834 of total 893834 records

Exporting index info... мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import.

Index dictionary... OK
Index USER_NAME... OK
Index OUser.name... OK
Index EMAIL... OK
Index ID... OK
Index mgn... OK
Index ORole.name... OK
Index USER_ROOT_ID... OK
Index OID... OK

OK (9 indexes)

Exporting manual indexes content... мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 11, 2014 3:00:48 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import.

Exporting index dictionary ... OK (entries=0)
Exporting index USER_NAME ... . . . . . . . . . . OK (entries=137)
Exporting index EMAIL ... OK (entries=0)
Exporting index ID ... . . . . . . . . . . OK (entries=136)
Exporting index mgn ... . . . . . . . . . . OK (entries=253085)
Exporting index USER_ROOT_ID ... OK (entries=1)
Exporting index OID ... . . . . . . . . . . OK (entries=256811)

OK (7 manual indexes)

Database export completed in 404346ms

Started import of database 'plocal:/Library/Application Support/MailArchiva/ROOT/database/new.archiva.db' from /Library/Application Support/MailArchiva/ROOT/database/archiva.db.json.gz...

Non merge mode (-merge=false): removing all default non security classes

Class ORIDs was removed.
Class OFunction was removed.
Class ORestricted was removed.
Class OTriggered was removed.
Class OSchedule was removed.

Removed 5 classes.

Importing database info... OK

Importing clusters...

Creating cluster 'internal'... OK, assigned id=0
Creating cluster 'default'... OK, assigned id=3
Creating cluster 'orole'... OK, assigned id=4
Creating cluster 'ouser'... OK, assigned id=5
Creating cluster 'ofunction'... OK, assigned id=6
Creating cluster 'oschedule'... OK, assigned id=7
Creating cluster 'orids'... OK, assigned id=8
Creating cluster 'v'... OK, assigned id=9
Creating cluster 'e'... OK, assigned id=10
Creating cluster 'user'... OK, assigned id=11
Creating cluster 'folder'... OK, assigned id=12
Creating cluster 'item'... OK, assigned id=13
Creating cluster 'thread'... OK, assigned id=14

Rebuilding indexes of truncated clusters ...

Cluster content was truncated and index ORole.name will be rebuilt

Index ORole.name was successfully rebuilt.

Cluster content was truncated and index OUser.name will be rebuilt

Index OUser.name was successfully rebuilt.

Done 2 indexes were rebuilt.

Done. Imported 13 clusters

Importing database schema... OK (14 classes)

Importing records...

Importing records into cluster 'internal' (id=0): = 2 records
Importing records into cluster 'orole' (id=4): = 3 records
Importing records into cluster 'ouser' (id=5): = 3 records
Importing records into cluster 'v' (id=9): ......................... = 256952 records
Importing records into cluster 'user' (id=11): = 137 records
Importing records into cluster 'folder' (id=12): = 3566 records
Importing records into cluster 'item' (id=13): ...................................................... = 546636 records
Importing records into cluster 'thread' (id=14): ........ = 84040 records
Importing records into cluster 'thread' (id=14): Started migration of links (-migrateLinks=true). Links are going to be updated according to new RIDs:
Cluster default... Processed: 0
Cluster orole... Processed: 3
Cluster ouser... Processed: 3
Cluster ofunction... Processed: 0
Cluster oschedule... Processed: 0
Cluster orids... Processed: 0
Cluster v...

10000 documents were processed...

20000 documents were processed...

30000 documents were processed...

40000 documents were processed...

50000 documents were processed...

60000 documents were processed...

70000 documents were processed...

80000 documents were processed...

90000 documents were processed...

100000 documents were processed...

110000 documents were processed...

120000 documents were processed...

130000 documents were processed...

140000 documents were processed...

150000 documents were processed...

160000 documents were processed...

170000 documents were processed...

180000 documents were processed...

190000 documents were processed...

200000 documents were processed...

210000 documents were processed...

220000 documents were processed...

230000 documents were processed...

240000 documents were processed...

250000 documents were processed... Processed: 256952

Cluster e... Processed: 0
Cluster user... Processed: 137
Cluster folder... Processed: 3566
Cluster item...

10000 documents were processed...

20000 documents were processed...

30000 documents were processed...

40000 documents were processed...

50000 documents were processed...

60000 documents were processed...

70000 documents were processed...

80000 documents were processed...

90000 documents were processed...

100000 documents were processed...

110000 documents were processed...

120000 documents were processed...

130000 documents were processed...

140000 documents were processed...

150000 documents were processed...

160000 documents were processed...

170000 documents were processed...

180000 documents were processed...

190000 documents were processed...

200000 documents were processed...

210000 documents were processed...

220000 documents were processed...

230000 documents were processed...

240000 documents were processed...

250000 documents were processed...

260000 documents were processed...

270000 documents were processed...

280000 documents were processed...

290000 documents were processed...

300000 documents were processed...

310000 documents were processed...

320000 documents were processed...

330000 documents were processed...

340000 documents were processed...

350000 documents were processed...

360000 documents were processed...

370000 documents were processed...

380000 documents were processed...

390000 documents were processed...

400000 documents were processed...

410000 documents were processed...

420000 documents were processed...

430000 documents were processed...

440000 documents were processed...

450000 documents were processed...

460000 documents were processed...

470000 documents were processed...

480000 documents were processed...

490000 documents were processed...

500000 documents were processed...

510000 documents were processed...

520000 documents were processed...

530000 documents were processed...

540000 documents were processed... Processed: 546636

Cluster thread...

10000 documents were processed...

20000 documents were processed...

30000 documents were processed...

40000 documents were processed...

50000 documents were processed...

60000 documents were processed...

70000 documents were processed...

80000 documents were processed... Processed: 84041

Total links updated: 891338

Done. Imported 891339 records

Importing indexes ...

Index 'dictionary'... OK
Index 'USER_NAME'... OK
Index 'OUser.name'... OK
Index 'EMAIL'... OK
Index 'ID'... OK
Index 'mgn'... OK
Index 'ORole.name'... OK
Index 'USER_ROOT_ID'... OK
Index 'OID'... OK

Done. Created 9 indexes.

Importing manual index entries...

Index 'dictionary'... OK (0 entries)
Index 'USER_NAME'... OK (137 entries)
Index 'EMAIL'... OK (0 entries)
Index 'ID'... OK (136 entries)
Index 'mgn'... Error on database import happened just before line 0, column 658852613 java.lang.NullPointerException at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importManualIndexes(ODatabaseImport.java:360) at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:199) at com.stimulus.archiva.database.blueprints.TestCaseDB.main(TestCaseDB.java:51) Exception in thread "main" com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'new.archiva.db' from file: /Library/Application Support/MailArchiva/ROOT/database/archiva.db.json.gz at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:217) at com.stimulus.archiva.database.blueprints.TestCaseDB.main(TestCaseDB.java:51) Caused by: java.lang.NullPointerException at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importManualIndexes(ODatabaseImport.java:360) at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:199) ... 1 more

:-) you did not close again )) but it is not the issue now, I am looking what is wrong with manual indexes import.

nope, this DB (1.5.1) was closed correctly 100% via the graph.shutdown().

Also FYI i see new converted DB is greater 2.4 times when old one. But I suppose it must be lesser, as snappy used.

Regards Valentin

@valenpo the correct way is to shutdown JVM before database copy (does not matter embedded or server), call of shutdwon() method leads to connection close not storage close. You have invalid link in your manual index that is why you have NPE. But I fixed this NPE any way, I will run tests and commit.

You have bigger database because it contains not only data but operations log too.

Also I have changed code a bit to provide better logging support. So new code is:

final ODatabaseDocumentTx database = new ODatabaseDocumentTx("local:/home/andrey/Development/orientdb/archiva.db");
    database.open("admin", "admin");

    final ODatabaseExport databaseExport = new ODatabaseExport(database, "/home/andrey/Development/orientdb/archiva.db.json",
        new OCommandOutputListener() {
          @Override
          public void onMessage(String iText) {
            System.out.println(iText);
          }
        });

    databaseExport.exportDatabase();
    databaseExport.close();

    final ODatabaseDocumentTx newDatabase = new ODatabaseDocumentTx("plocal:/home/andrey/Development/orientdb/new.archiva.db");
    newDatabase.create();

    final ODatabaseImport databaseImport = new ODatabaseImport(newDatabase, "/home/andrey/Development/orientdb/archiva.db.json.gz",
        new OCommandOutputListener() {
          @Override
          public void onMessage(String iText) {
            System.out.println(iText);
          }
        });
    databaseImport.importDatabase();

    database.close();
    databaseImport.close();

    database.open("admin", "admin");

    OGraphMigration graphMigration = new OGraphMigration(newDatabase, new OCommandOutputListener() {
      @Override
      public void onMessage(String iText) {
        System.out.println(iText);
      }
    });

    graphMigration.execute();
    database.close();

@laa Andrey, morning FYI, I made some huge test till night, archiving 100 huge pst files with all structure of emails and folders, I can see about 100 pst root folders on tree view (this is stored on DB). After server restart with correct close (graph.shutdown()) I can see ~32 root folders, so 2/3 are lost. Those are E connections from ROOT vertex to USER vertex.

Also same DB with local are 4 times smaller (1GB local vs 4.4GB plocal). I assume that plocal use some compression?

I can provide generated DB and access to testing platform.

Making test can take alot of time, as alot of business logic involved, and I think better to see it on real environment Regards Valentin

@valenpo I mentioned several times before here https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37011101 and here https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37305543, that graph.shutdown() is not correct way to close storage. Could you confirm that you close storage by closing JVM, not only by calling graph.shutdown() ?

@laa yes, JVM is closed via standard Tomcat scripts.

@valenpo when you open db do you see message "WARNING: segment file '...' was not closed correctly last time"

Do you see files with .wal extension into the closed database ?

Andrey, logs are behind. Libraries are used on Tomcat instance, so tomcat start up normally as shutdown too.

I shutdown tomcat when the server perform no actions … Right now I see no one record. Should I upgrade to night build and test it again?

2014-03-12 15:34:36.788 WARN - Cannot find default script language for javascript 2014-03-12 15:34:36.879 WARN - segment file 'database.ocf' was not closed correctly last time 2014-03-12 15:34:37.043 WARN - Can not restore 1 WAL master record for storage archiva.db 2014-03-12 15:34:37.162 WARN - Storage archiva.db was not closed properly. Will try to restore from write ahead log. 2014-03-12 15:34:37.162 INFO - Looking for last checkpoint... 2014-03-12 15:34:37.162 INFO - Checkpoints are absent, the restore will start from the beginning. 2014-03-12 15:34:37.162 INFO - Date restore procedure is started. 2014-03-12 15:34:37.185 ERROR - Exception during storage data restore. com.orientechnologies.orient.core.exception.OStorageException: File with id 3 does not exist. at com.orientechnologies.orient.core.index.hashindex.local.cache.OWOWCache.openFile(OWOWCache.java:200) at com.orientechnologies.orient.core.index.hashindex.local.cache.OReadWriteDiskCache.openFile(OReadWriteDiskCache.java:125) at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.restoreFrom(OLocalPaginatedStorage.java:502) at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.restoreFromBegging(OLocalPaginatedStorage.java:468) at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.restoreFromWAL(OLocalPaginatedStorage.java:307) at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.restoreIfNeeded(OLocalPaginatedStorage.java:274) at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.open(OLocalPaginatedStorage.java:256) at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.open(ODatabaseRaw.java:101) at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:54) at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.open(ODatabaseRecordAbstract.java:131) at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:54) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.openOrCreate(OrientBaseGraph.java:878) at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:103) at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:58) at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:54) at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:34) at com.stimulus.archiva.database.blueprints.BluePrintDatabase.initGraph(BluePrintDatabase.java:95) at com.stimulus.archiva.database.blueprints.BluePrintDatabase.connect(BluePrintDatabase.java:145) at com.stimulus.archiva.database.DatabaseService.startup(DatabaseService.java:168) at com.stimulus.archiva.domain.Services.startStop(Services.java:75) at com.stimulus.archiva.domain.Services.startAll(Services.java:58) at com.stimulus.archiva.domain.Application.startup(Application.java:625) at com.stimulus.archiva.domain.Applications.startupApps(Applications.java:267) at com.stimulus.archiva.domain.Applications.startup(Applications.java:250) at com.stimulus.archiva.plugin.Startup.init(Startup.java:61) at com.stimulus.archiva.security.MailArchivaSecurityFilter.init(MailArchivaSecurityFilter.java:46) at com.stimulus.archiva.security.MailArchivaSecurityFilter.init(MailArchivaSecurityFilter.java:111) at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:281) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:262) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:107) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4656) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5309) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1114) at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1673) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-03-12 15:34:37.186 INFO - Storage data restore was completed 2014-03-12 15:34:37.239 ERROR - Database's schema is empty! Recreating the system classes and allow the opening of the database but double check the integrity of the database 2014-03-12 15:34:37.520 INFO - SMTP server :8091 starting 2014-03-12 15:34:37.522 INFO - SMTP server :8091 started 2014-03-12 15:35:14.782 INFO - Successfully connected to vm:broker:(tcp://localhost:61617)?persistent=false&useJmx=false

Andrey

Our server always shutdown clearly. We have spent many hours making sure that the JVM will exit cleanly. In any case, even if the server did not shutdown cleanly, it seems like a huge amount of records to lose. What happens if there is a powercut?

Jamie

On 2014/03/12, 12:42 PM, Andrey Lomakin wrote:

@valenpo https://github.com/valenpo I mentioned this several times before, that graph.shutdown() is not correct way to close storage. Could you confirm that you close storage by closing JVM, not only by calling graph.shutdown() ?

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-37394918.

@valenpo @jamieb22 I see message "WARN - Storage archiva.db was not closed properly. Will try to restore from write ahead log."

Which means that storage was not closed correctly. And I have seen this message in all instances of db which you send to us. So it seems like your issues with records lost were not related to local or plocal storage implementations. But to the fact that even during Tomcat grace full shutdown storage was not closed correctly. That is what we need to solve, about records lost, you lost not records but manual indexes content because by default they work in rollback only mode, I will provide settings to make possible to restore manual index records too.

So first step which we need to do is put following code in ServletContextListener.html#contextDestroyed method (implementation of interface of course).

Orient.isntance().shutdown();

And check whether we see given warning again, and check data presence. Could you do this ?

@Override public void contextDestroyed(ServletContextEvent event) { Orient.instance().shutdown(); }

done, so hope right now it will shutdown correct.

@laa Andrey, thank for clue. After that hook Orient.isntance().shutdown(); it looks DB started fine after server restart. I see all records.

Thanks

Regards Valentin

Ok. The only issue that needs to be fixed to have full durability support is this one https://github.com/orientechnologies/orientdb/issues/1761 could you arrange it's schedule with @lvca . I have written above that I will provide you settings to restore indexes but this issue is only partially fixed. And of course at the beginning of the change we have no idea that you use manual indexes it is so rare case. BTW can you replace them by automatic indexes ? they are faster than manual indexes.

orientechnologies / orientdb

Support old binary formats. #1899