Support old binary formats.

[andrii0lomakin]

To support old binary formats we can use following approach:

1. Read storage configuration.
2. If storage configuration version is different from latest, look at list of migration scripts which will be used to migrate db data.
3. Perform migration automatically, with ability to track migration progress using listeners.

Subtasks:

• Integration of component version concept into storage components
• OMigration interface
• Migrator implementation
• API and console command for upgrade task
• API and console command that show version of binary data structure
• Automatic tests for testing backward compatibility
• Add compatibility for
  • 1.5.0
  • 1.5.1
  • 1.5.2
  • 1.6.0
  • 1.6.1
  • 1.6.2
  • 1.6.3
• Implement upgrade tasks for:
  • 1.5.0
  • 1.5.1
  • 1.5.2
  • 1.6.0
  • 1.6.1
  • 1.6.2
  • 1.6.3

[andrii0lomakin]

May be it is not the best way. Too long time for migration. It is possible to use sometimes but in very rare cases. But we can preserve old versions of clusters and indexes. We can do it as in before mentioned approach using storage configuration number.

So for for each binary format change we change configuration number and use old clusters and index versions if needed. Also with warning that used data structures out of dated.

Also we can provide migration tool that can be run pragmatically to migrate db versions without opening them.

[jamieb22]

Test scripts should be implemented to test that a new version of Orient has the ability to read/write and upgrade old database formats. These test scripts should be incorporated into the Orient build process to ensure that successive versions are backwards compatible.

[andrii0lomakin]

@jamieb22 +1 ).

[lvca]

This will begin on January 24th.

[jamieb22]

Luca

Please may I have status update on this project.

So far, Orient DB is causing instability to our platform. The database simply gets corrupted.

I am hoping an upgrade will resolve the matter.

Regards

Jamie

On 2014/01/21, 2:41 PM, Luca Garulli wrote:

This will begin on January 24th.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-32870519.

[lvca]

Hi Jamie, we already started this after a analysis phase to understand all the impacts of it (not 100% of the internal structured had a version number). Andrey and Artem could update better on this.

[andrii0lomakin]

Hi Jamie, We would be appreciate if you send us of sample of your database so we will be sure that we provided all means to make possible migration of your data.

[jamieb22]

Please download the file orientdb.zip from sftp site enterprise:fl0w3r@mailarchiva.com

On 2014/02/10, 7:22 PM, Andrey Lomakin wrote:

Hi Jamie, We would be appreciate if you send us of sample of your database so we will be sure that we provided all means to make possible migration of your data.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-34657201.

[enisher]

Thanks @jamieb22!

We expect to finish compatibility with 1.5.1 in the middle of the next week.

[jamieb22]

Great... I am hoping this effort will offer a permanent upgrade path for subsequent releases too!

On 2014/02/14, 10:34 AM, Artem Orobets wrote:

Thanks @jamieb22 https://github.com/jamieb22!

We expect to finish compatibility with 1.5.1 in the middle of the next week.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-35065182.

[jamieb22]

Middle of next week has arrived. Any update?

On 2014/02/14, 10:34 AM, Artem Orobets wrote:

Thanks @jamieb22 https://github.com/jamieb22!

We expect to finish compatibility with 1.5.1 in the middle of the next week.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-35065182.

[lvca]

We've completed this issue at 90%, we need today to finish tests, maximum tomorrow.

[andrii0lomakin]

Hi, We have tried your database with new changes and we are able to open it. But we need time to make stable. Sorry for delay we are completing several tasks at the same time.

[jamieb22]

Thank you for the update and for your efforts. Please rather take your time and make it very stable. We cannot afford to lose data.

On 2014/02/20, 11:36 AM, Andrey Lomakin wrote:

Hi, We have tried your database with new changes and we are able to open it. But we need time to make stable. Sorry for delay we are completing several tasks at the same time.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-35602668.

[jamieb22]

Regarding the comment:

We expect to finish compatibility with 1.5.1 in the middle of the next week.

Isn't the latest version of Orient DB 1.7? Will this upgrade script take us up to the latest version?

[lvca]

This issue will be deployed on v. 1.7-rc2 we will release as soon as this issue is fixed. We'd like to ask you to make some tests against multiple "old" databases with 1.7-rc2-SNAPSHOT as soon as this issue is closed, to be sure to release a bug free version with 1.7-rc2.

WDYT?

[andrii0lomakin]

@jamieb22 Could you send me your database once again.

[jamieb22]

The sftp site is still active. You can download.

Andrey Lomakin notifications@github.com wrote:

@jamieb22 Could you send me your database once again.

---

Reply to this email directly or view it on GitHub: https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-35800558

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[andrii0lomakin]

I was asked for the password and can not login.

[jamieb22]

The credentials are as follows:

sftp mailarchiva.com username: enteprise password: fl0w3r

On 2014/02/24, 11:38 AM, Andrey Lomakin wrote:

I was asked for the password and can not login.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-35870096.

[andrii0lomakin]

I have very good news. I have opened your databases using 1.5.1 and exported it in JSON. Then:

1. Opened database using latest develop branch.
2. Created new plocal database and did JSON import.
3. Compared imported database and old one.

Got result "databases match". Here is full console output https://gist.github.com/laa/2a387cbcc53326872fc7 . I am creating automatic binary compatibility test, will commit and close issue.

Could you as @lvca proposed build database from latest sources and provide the same comparison procedure too ? Also after db compare not before could you run upgrade graph command to use new version of graph database ?

Please note that rc2 is not released yet (we have several critical issues to fix) so we do not recommend to use it (yet) in production.

[jamieb22]

Andrey, that's great news. Thank you. May I ask whether your upgrade procedure will automatically detect whether a database requires an upgrade, and thereafter perform the upgrade automatically? We are not using the OrientDB console. We are using straight API integration. Our expectation is that when the embedded database is started, it will be automatically upgraded. Is this correct? Thus, our test would be to get the latest sources and start our server. In essence, all the records should be visible from our GUI. I would appreciate your feedback.

On 2014/02/24, 1:39 PM, Andrey Lomakin wrote:

I have very good news. I have opened your databases using 1.5.1 and exported it in JSON. Then:

1. Opened database using latest develop branch.
2. Created new plocal database and did JSON import.
3. Compared imported database and old one.

Got result "databases match". Here is full console output https://gist.github.com/laa/2a387cbcc53326872fc7 . I am creating automatic binary compatibility test, will commit and close issue.

Could you as @lvca https://github.com/lvca proposed build database from latest sources and provide the same comparison procedure too ? Also after db compare not before could you run upgrade graph command to use new version of graph database ?

Please note that rc2 is not released yet (we have several critical issues to fix) so we do not recommend to use it (yet).

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-35878263.

[jamieb22]

Andrey... can this function also be used as a backup procedure. It would be nice to be able to use a JSON export function to backup the DB on occasion? Is the JSON export thread safe? Are you creating public API functions for upgrading, exporting, etc?

[jamieb22]

Don't forget to build in upgrade test in the Orient build scripts. If we do not do this, it is guaranteed that the upgrade will be broken at some point in time.

[andrii0lomakin]

Hi,

1. Database is not upgraded automatically it will take hours if we will go in this way, it is able to work with previous version of binary format till you will not upgrade it yourself, please note that it may take hours for upgrade if database is huge. We simple can not freeze everything if database is opened.
2. Console is wrapper for database API, so any thing which you able to do in console you able to do in Java.
3. You can do JSON backup but it means that database will be frozen for writes, JSON export is slow (that is why you need to do export to JSON and import from it manually) so you can do it, but what the point to do it in run time ? database freeze/release is much faster.

About binary compatibility test we were planing to include them from beginning, otherwise it will be unprofessional :-) .

Actually here is how this test will run:

1. get list of tags and form list of all which have 2 latest middle versions. so if we have 1.5, 1.5.1, 1.6.1, 1.6.2, 1.7, 1.7.1 we will have to test all 1, 6 and 1.7 tags.
2. Checkout latest version. run test-plocal task on this version. but do not delete test database and exported database (it will contain all existing data structures in very different combinations).
3. checkout next tag and do db import and compare content of imported and original database.
4. do the same with snapshot.

will do this on CI every night and because it is kind of tricky will be written on Gradle.

[jamieb22]

Is there a way to determine the current version of Orient in code? We would obviously need to determine this, in order to assess whether an upgrade is necessary?

Will latest version of Orient be able to read older formats without upgrade?

On 2014/02/25, 11:51 AM, Andrey Lomakin wrote:

Hi,

1. Database is not upgraded automatically it will take hours if we will go in this way, it is able to work with previous version of binary format till you will not upgrade it yourself, please note that it may take hours for upgrade if database is huge. We simple can not freeze everything if database is opened.
2. Console is wrapper for database API, so any thing which you able to do in console you able to do in Java.
3. You can do JSON backup but it means that database will be frozen for writes, JSON export is slow (that is why you need to do export to JSON and import from it manually) so you can do it, but what the point to do it in run time ? database freeze/release is much faster.

About binary compatibility test we were planing to include them from beginning, otherwise it will be unprofessional :-) .

Actually here is how this test will run:

1. get list of tags and form list of all which have 2 latest middle versions. so if we have 1.5, 1.5.1, 1.6.1, 1.6.2, 1.7, 1.7.1 we will have to test all 1, 6 and 1.7 tags.
2. Checkout latest version. run test-plocal task on this version. but do not delete test database and exported database (it will contain all existing data structures in very different combinations).
3. checkout next tag and do db import and compare content of imported and original database.
4. do the same with snapshot.

will do this on CI every night and because it is kind of tricky will be written on Gradle.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-35991218.

[andrii0lomakin]

Hi,

Will latest version of Orient be able to read older formats without upgrade?

That is how it works.

During development database binary format can be changed several times, reading of version is not usable but you can read binary format version it can be taken from com.orientechnologies.orient.core.db.record.OCurrentStorageComponentsFactory#binaryFormatVersion and com.orientechnologies.orient.core.storage.OStorage#getComponentsFactory . Latest binary format version is here com.orientechnologies.orient.core.config.OStorageConfiguration#CURRENT_BINARY_FORMAT_VERSION. Note that binary compatibility on design level is started from 1.7-rc2, but as exception for your needs we also support 1.5.1 version of local storage.

[andrii0lomakin]

Small update database backup is based on db freeze/release to do export/import could you look at com.orientechnologies.orient.test.database.auto.DbImportExportTest

[lvca]

Take a look at: https://github.com/orientechnologies/orientdb/wiki/Console-Command-Export

[jamieb22]

Will you let us know when you would like us to run our own internal tests? For this, we need the latest snapshot. There is no 1.7 snapshot currently available at https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb/ yet.

[lvca]

Last snapshot is 1.7-rc2 you can find here: https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-community/1.7-rc2-SNAPSHOT/. Please let us know if there are problems.

[andrii0lomakin]

Nightly binary compatibility test was added http://helios.orientechnologies.com/job/binary-compatibility-test/

[jamieb22]

I can't get it to work. When I started up our server, it spewed out the following error on an old database:

2014-03-02 08:09:28 t.c.s.a.d.DatabaseService [ERROR] failed to connect to database:Cannot open local storage '/Library/Application Support/MailArchiva/ROOT/database/archiva.db' with mode=rw com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/Library/Application Support/MailArchiva/ROOT/database/archiva.db' with mode=rw at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.open(OStorageLocal.java:225) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.open(ODatabaseRaw.java:101) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:54) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.open(ODatabaseRecordAbstract.java:131) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:54) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.openOrCreate(OrientBaseGraph.java:783) ~[blueprints-orient-graph-2.5.0-SNAPSHOT.jar:na] at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:100) ~[blueprints-orient-graph-2.5.0-SNAPSHOT.jar:na] at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:36) ~[blueprints-orient-graph-2.5.0-SNAPSHOT.jar:na] at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:27) ~[blueprints-orient-graph-2.5.0-SNAPSHOT.jar:na] at com.stimulus.archiva.database.blueprints.BluePrintDatabase.initGraph(BluePrintDatabase.java:74) ~[BluePrintDatabase.class:na] at com.stimulus.archiva.database.blueprints.BluePrintDatabase.connect(BluePrintDatabase.java:131) ~[BluePrintDatabase.class:na] at com.stimulus.archiva.database.DatabaseService.startup(DatabaseService.java:168) ~[DatabaseService.class:na] at com.stimulus.archiva.domain.Services.startStop(Services.java:75) [Services.class:na] at com.stimulus.archiva.domain.Services.startAll(Services.java:58) [Services.class:na] at com.stimulus.archiva.domain.Application.startup(Application.java:625) [Application.class:na] at com.stimulus.archiva.domain.Applications.startupApps(Applications.java:267) [Applications.class:na] at com.stimulus.archiva.domain.Applications.startup(Applications.java:250) [Applications.class:na] at com.stimulus.archiva.plugin.Startup.init(Startup.java:61) [Startup.class:na] at com.stimulus.archiva.security.MailArchivaSecurityFilter.init(MailArchivaSecurityFilter.java:46) [MailArchivaSecurityFilter.class:na] at com.stimulus.archiva.security.MailArchivaSecurityFilter.init(MailArchivaSecurityFilter.java:111) [MailArchivaSecurityFilter.class:na] at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:281) [catalina.jar:7.0.47] at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:262) [catalina.jar:7.0.47] at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:107) [catalina.jar:7.0.47] at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4775) [catalina.jar:7.0.47] at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5452) [catalina.jar:7.0.47] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:7.0.47] at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) [catalina.jar:7.0.47] at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) [catalina.jar:7.0.47] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] Caused by: java.lang.IllegalStateException: java.lang.NumberFormatException: For input string: "db" at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.extractOrder(OWriteAheadLog.java:665) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.(OWriteAheadLog.java:591) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.(OWriteAheadLog.java:557) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog.(OWriteAheadLog.java:111) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog.(OWriteAheadLog.java:83) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.open(OStorageLocal.java:218) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] ... 31 common frames omitted Caused by: java.lang.NumberFormatException: For input string: "db" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.7.0_45] at java.lang.Long.parseLong(Long.java:441) ~[na:1.7.0_45] at java.lang.Long.parseLong(Long.java:483) ~[na:1.7.0_45] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.extractOrder(OWriteAheadLog.java:662) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] ... 36 common frames omitted

On 2014/02/28, 7:42 PM, Andrey Lomakin wrote:

Closed #1899 https://github.com/orientechnologies/orientdb/issues/1899.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899.

[jamieb22]

I added orientdb2.zip to the ftp site. It is a much smaller database.

On 2014/03/02, 8:12 AM, Jamie wrote:

I can't get it to work. When I started up our server, it spewed out the following error on an old database:

2014-03-02 08:09:28 t.c.s.a.d.DatabaseService [ERROR] failed to connect to database:Cannot open local storage '/Library/Application Support/MailArchiva/ROOT/database/archiva.db' with mode=rw com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/Library/Application Support/MailArchiva/ROOT/database/archiva.db' with mode=rw at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.open(OStorageLocal.java:225) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.open(ODatabaseRaw.java:101) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:54) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.open(ODatabaseRecordAbstract.java:131) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:54) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.openOrCreate(OrientBaseGraph.java:783) ~[blueprints-orient-graph-2.5.0-SNAPSHOT.jar:na] at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:100) ~[blueprints-orient-graph-2.5.0-SNAPSHOT.jar:na] at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:36) ~[blueprints-orient-graph-2.5.0-SNAPSHOT.jar:na] at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:27) ~[blueprints-orient-graph-2.5.0-SNAPSHOT.jar:na] at com.stimulus.archiva.database.blueprints.BluePrintDatabase.initGraph(BluePrintDatabase.java:74) ~[BluePrintDatabase.class:na] at com.stimulus.archiva.database.blueprints.BluePrintDatabase.connect(BluePrintDatabase.java:131) ~[BluePrintDatabase.class:na] at com.stimulus.archiva.database.DatabaseService.startup(DatabaseService.java:168) ~[DatabaseService.class:na] at com.stimulus.archiva.domain.Services.startStop(Services.java:75) [Services.class:na] at com.stimulus.archiva.domain.Services.startAll(Services.java:58) [Services.class:na] at com.stimulus.archiva.domain.Application.startup(Application.java:625) [Application.class:na] at com.stimulus.archiva.domain.Applications.startupApps(Applications.java:267) [Applications.class:na] at com.stimulus.archiva.domain.Applications.startup(Applications.java:250) [Applications.class:na] at com.stimulus.archiva.plugin.Startup.init(Startup.java:61) [Startup.class:na] at com.stimulus.archiva.security.MailArchivaSecurityFilter.init(MailArchivaSecurityFilter.java:46) [MailArchivaSecurityFilter.class:na] at com.stimulus.archiva.security.MailArchivaSecurityFilter.init(MailArchivaSecurityFilter.java:111) [MailArchivaSecurityFilter.class:na] at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:281) [catalina.jar:7.0.47] at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:262) [catalina.jar:7.0.47] at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:107) [catalina.jar:7.0.47] at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4775) [catalina.jar:7.0.47] at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5452) [catalina.jar:7.0.47] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:7.0.47] at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) [catalina.jar:7.0.47] at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) [catalina.jar:7.0.47] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] Caused by: java.lang.IllegalStateException: java.lang.NumberFormatException: For input string: "db" at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.extractOrder(OWriteAheadLog.java:665) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.(OWriteAheadLog.java:591) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.(OWriteAheadLog.java:557) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog.(OWriteAheadLog.java:111) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog.(OWriteAheadLog.java:83) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.open(OStorageLocal.java:218) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] ... 31 common frames omitted Caused by: java.lang.NumberFormatException: For input string: "db" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.7.0_45] at java.lang.Long.parseLong(Long.java:441) ~[na:1.7.0_45] at java.lang.Long.parseLong(Long.java:483) ~[na:1.7.0_45] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.extractOrder(OWriteAheadLog.java:662) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] ... 36 common frames omitted

On 2014/02/28, 7:42 PM, Andrey Lomakin wrote:

Closed #1899 https://github.com/orientechnologies/orientdb/issues/1899.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899.

[valenpo]

Hi everybody, I've tried open DB with created with 1.5.1 with current RC, db wasn't opened. DB can be downloaded here http://yadi.sk/d/OKomt898JqYFS 180MB

[enisher]

@jamieb22 it seems that you are trying to open the db by the other engine. The old version of db work on local but you are trying to open it with plocal engine.

Could you try to change engine type to local?

I've also created an issue to implement more appropriate exceptions for such cases (#2084).

[jamieb22]

Artem

...and how do you force the use of local?

We are current prefixing the location with "local:".. as described below

this.location = "local:" + location; graph = new OrientGraph(location);

I was under the impression, the "local:" indicates to orient to use the local database? Is there another way we are supposed to be doing it?

Jamie

On 2014/03/03, 1:04 PM, Artem Orobets wrote:

@jamieb22 https://github.com/jamieb22 it seems that you are trying to open the db by the other engine. The old version of db work on local but you are trying to open it with plocal engine.

Could you try to change engine type to local?

I've also created an issue to implement more appropriate exceptions for such cases.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899#issuecomment-36500095.

[jamieb22]

Artem

Can we please have a reply to our question? We are anxious to get this working.

Thanks

Jamie

[jamieb22]

Andrey

Please can you reply to our earlier comment on GitHub. We have not received a response from the Orient Technologies Organization yet.

Thank you!

Jamie

On 2014/02/28, 7:42 PM, Andrey Lomakin wrote:

Closed #1899 https://github.com/orientechnologies/orientdb/issues/1899.

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/1899.

[enisher]

@jamieb22, sorry for delay. Could you mention my github name in messages directed to me? In this case I'll receive a notification.

It is strange, I'm not able to reproduce the issue. Is it possible to provide me your application for test? If it is appropriate you can use my corporate mail to share it privately.

[valenpo]

@enisher Artem hello, in previous message "Hi everybody, I've tried open DB with created with 1.5.1 with current RC, db wasn't opened. DB can be downloaded here http://yadi.sk/d/OKomt898JqYFS 180MB"

This is DB created with local engine and it cant be open. It is same error as Jamie has.

Can you test this DB. (created with 1.5.1 local engine)

Regards Valentin

[enisher]

@valenpo how can I reproduce the problem? Is it enough to open the db with a console and run some query?

[valenpo]

@enisher Artem, thank's for quick response. Let me think about TestCase for it.

[valenpo]

@enisher @jamieb22 Artem, with provided code you can test the database provided above.

package com.stimulus.archiva.database.blueprints;

import com.orientechnologies.orient.core.config.OGlobalConfiguration;
import com.tinkerpop.blueprints.Index;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.impls.orient.OrientGraph;
import com.tinkerpop.gremlin.java.GremlinPipeline;

public class TestCaseDB {

    public static void main(String[] args) {

        String USER_ROOT_ID = "USER_ROOT_ID";

        OGlobalConfiguration.CACHE_LEVEL1_ENABLED.setValue(false);
        OGlobalConfiguration.DB_USE_DISTRIBUTED_VERSION.setValue(false);
        OGlobalConfiguration.NON_TX_RECORD_UPDATE_SYNCH.setValue(true); //Executes a synch against the file-system at every record operation. This slows down records updates but guarantee reliability on unreliable drives
        OGlobalConfiguration.TX_LOG_SYNCH.setValue(true); //Executes a synch against the file-system for each log entry. This slows down transactions but guarantee transaction reliability on non-reliable drives
        IndexableGraph graph = new OrientGraph("local:/Library/Application Support/MailArchiva/ROOT/database/archiva.db");

        try {
            Index<Vertex> index = graph.getIndex(USER_ROOT_ID, Vertex.class);

            Iterable<Vertex> foundVertices = index.get(USER_ROOT_ID, "USER_ROOT");
            Vertex root = foundVertices.iterator().next();

            GremlinPipeline<Vertex, ?> pipe = new GremlinPipeline();
            pipe.start(root).out("user");

            while (pipe.hasNext()) {
                Vertex vertex = (Vertex) pipe.next();   
                System.out.println(vertex.getProperty("ID"));
            }
        } finally {
            graph.shutdown();
        }
    }

}

In output you should get

k..allen@enron.com
john.arnold@enron.com
harry.arora@enron.com
robert.badeer@enron.com
susan.bailey@enron.com
eric.bass@enron.com
don.baughman@enron.com
sally.beck@enron.com
robert.benson@enron.com
lynn.blair@enron.com
f..brawner@enron.com
rick.buy@enron.com
f..campbell@enron.com
mike.carson@enron.com
michelle.cash@enron.com
monika.causholli@enron.com
shelley.corman@enron.com
sean.crandall@enron.com
martin.cuilla@enron.com
jeff.dasovich@enron.com
dana.davis@enron.com
craig.dean@enron.com
w..delainey@enron.com
james.derrick@enron.com

lindy.donoho@enron.com
tom.donohoe@enron.com
chris.dorland@enron.com
frank.ermis@enron.com
j..farmer@enron.com
mary.fischer@enron.com
m..forney@enron.com
lisa.gang@enron.com
l..gay@enron.com
tracy.geaccone@enron.com
chris.germany@enron.com
doug.gilbert-smith@enron.com
c..giron@enron.com
john.griffith@enron.com
mike.grigsby@enron.com
e..haedicke@enron.com
rod.hayslett@enron.com
marie.heard@enron.com
scott.hendrickson@enron.com
juan.hernandez@enron.com
t..hodge@enron.com
keith.holst@enron.com
stanley.horton@enron.com
kevin.hyatt@enron.com
tana.jones@enron.com
j.kaminski@enron.com
j..kean@enron.com
f..keavey@enron.com
kam.keiser@enron.com
jeff.king@enron.com
louise.kitchen@enron.com
tori.kuykendall@enron.com
lavorato@enron.com
kenneth.lay@enron.com
matthew.lenhart@enron.com
h..lewis@enron.com
michelle.lokay@enron.com
teb.lokey@enron.com
m..love@enron.com
t..lucci@enron.com
mike.maggi@enron.com
kay.mann@enron.com
a..martin@enron.com
larry.may@enron.com
danny.mccarty@enron.com
mark.mcconnell@enron.com
brad.mckay@enron.com
jonathan.mckay@enron.com
errol.mclaughlin@enron.com
albert.meyers@enron.com
l..mims@enron.com
matt.motley@enron.com
scott.neal@enron.com
gerald.nemec@enron.com
stephanie.panus@enron.com
joe.parks@enron.com
w..pereira@enron.com
debra.perlingiere@enron.com
vladi.pimenov@enron.com
phillip.platter@enron.com
m..presto@enron.com
joe.quenet@enron.com
dutch.quigley@enron.com
bill.rapp@enron.com
jay.reitmeyer@enron.com
cooper.richey@enron.com
andrea.ring@enron.com
richard.ring@enron.com
benjamin.rogers@enron.com
kevin.ruscitti@enron.com
elizabeth.sager@enron.com
eric.saibi@enron.com
holden.salisbury@enron.com
monique.sanchez@enron.com
b..sanders@enron.com
diana.scholtes@enron.com
darrell.schoolcraft@enron.com
jim.schwieger@enron.com
m..scott@enron.com
cara.semperger@enron.com
sara.shackleton@enron.com
a..shankman@enron.com
richard.shapiro@enron.com
s..shively@enron.com
jeff.skilling@enron.com
ryan.slinger@enron.com
matt.smith@enron.com
geir.solberg@enron.com
theresa.staab@enron.com
d..steffes@enron.com
joe.stepenovitch@enron.com
geoff.storey@enron.com
j..sturm@enron.com
mike.swerzbin@enron.com
.taylor@enron.com
m..tholt@enron.com
d..thomas@enron.com
judy.townsend@enron.com
barry.tycholiz@enron.com
.ward@enron.com
kimberly.watson@enron.com
charles.weldon@enron.com
greg.whalley@enron.com
w..white@enron.com
mark.whitt@enron.com
jason.williams@enron.com
bill.williams@enron.com
jason.wolfe@enron.com
paul.y'barbo@enron.com
andy.zipper@enron.com
john.zufferli@enron.com
null

[enisher]

@valenpo ok, I'm on it.

[enisher]

@valenpo, @jamieb22,

The bug is fixed. I've received expected output on your test case. Could you retry on the last snapshot?

I also noticed that the database has not been closed correctly. Please, make sure that your database is properly closed before migration to newer version. The db recovery procedure should be executed on the same version of the library.

[valenpo]

@enisher Artem, thanks for fix. It is open now old DB (1.5.1) fine. But problem with dot in DB name as I see doesn't fixed.

2014-03-06 15:40:23 t.c.s.a.d.DatabaseService [ERROR] failed to connect to database:Cannot open local storage '/Library/Application Support/MailArchiva/ROOT/database/archiva.db' with mode=rw com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/Library/Application Support/MailArchiva/ROOT/database/archiva.db' with mode=rw at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.open(OStorageLocal.java:225) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.open(ODatabaseRaw.java:101) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:54) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.open(ODatabaseRecordAbstract.java:131) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:54) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.openOrCreate(OrientBaseGraph.java:882) ~[orientdb-graphdb-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:98) ~[orientdb-graphdb-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:58) ~[orientdb-graphdb-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:54) ~[orientdb-graphdb-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:34) ~[orientdb-graphdb-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.stimulus.archiva.database.blueprints.BluePrintDatabase.initGraph(BluePrintDatabase.java:74) ~[BluePrintDatabase.class:na] at com.stimulus.archiva.database.blueprints.BluePrintDatabase.connect(BluePrintDatabase.java:131) ~[BluePrintDatabase.class:na] at com.stimulus.archiva.database.DatabaseService.startup(DatabaseService.java:168) ~[DatabaseService.class:na] at com.stimulus.archiva.domain.Services.startStop(Services.java:75) [Services.class:na] at com.stimulus.archiva.domain.Services.startAll(Services.java:58) [Services.class:na] at com.stimulus.archiva.domain.Application.startup(Application.java:625) [Application.class:na] at com.stimulus.archiva.domain.Applications.startupApps(Applications.java:267) [Applications.class:na] at com.stimulus.archiva.domain.Applications.startup(Applications.java:250) [Applications.class:na] at com.stimulus.archiva.plugin.Startup.init(Startup.java:61) [Startup.class:na] at com.stimulus.archiva.security.MailArchivaSecurityFilter.init(MailArchivaSecurityFilter.java:46) [MailArchivaSecurityFilter.class:na] at com.stimulus.archiva.security.MailArchivaSecurityFilter.init(MailArchivaSecurityFilter.java:111) [MailArchivaSecurityFilter.class:na] at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:281) [catalina.jar:7.0.35] at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:262) [catalina.jar:7.0.35] at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:107) [catalina.jar:7.0.35] at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4656) [catalina.jar:7.0.35] at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5312) [catalina.jar:7.0.35] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:7.0.35] at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) [catalina.jar:7.0.35] at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) [catalina.jar:7.0.35] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: java.lang.IllegalStateException: java.lang.NumberFormatException: For input string: "db" at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.extractOrder(OWriteAheadLog.java:665) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.(OWriteAheadLog.java:591) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.(OWriteAheadLog.java:557) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog.(OWriteAheadLog.java:111) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog.(OWriteAheadLog.java:83) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.open(OStorageLocal.java:218) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] ... 32 common frames omitted Caused by: java.lang.NumberFormatException: For input string: "db" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.7.0_51] at java.lang.Long.parseLong(Long.java:441) ~[na:1.7.0_51] at java.lang.Long.parseLong(Long.java:483) ~[na:1.7.0_51] at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWriteAheadLog$LogSegment.extractOrder(OWriteAheadLog.java:662) ~[orientdb-core-1.7-rc2-SNAPSHOT.jar:1.7-rc2-SNAPSHOT] ... 37 common frames omitted

[valenpo]

@enisher Artem, sorry. I cant confirm bug fixed. I used latest snapshot from site and get error when opening DB

SEVERE: Error during addition of index {type:UNIQUE,name:OUser.name,indexDefinition:{className:OUser,field:name,keyType:STRING},indexDefinitionClass:com.orientechnologies.orient.core.index.OPropertyIndexDefinition,clusters:[1],mapRid:#1:0} java.lang.NullPointerException at com.orientechnologies.orient.core.index.OIndexManagerShared$1.run(OIndexManagerShared.java:413) at java.lang.Thread.run(Thread.java:744)

мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: Start creation of index ORole.name мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log SEVERE: Error during addition of index {type:UNIQUE,name:ORole.name,indexDefinition:{className:ORole,field:name,keyType:STRING},indexDefinitionClass:com.orientechnologies.orient.core.index.OPropertyIndexDefinition,clusters:[1],mapRid:#1:2} java.lang.NullPointerException at com.orientechnologies.orient.core.index.OIndexManagerShared$1.run(OIndexManagerShared.java:413) at java.lang.Thread.run(Thread.java:744)

мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: Index dictionary is not automatic index and will be added as is. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: Index dictionary was added in DB index list. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_ROOT_ID is not automatic index and will be added as is. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: Index USER_ROOT_ID was added in DB index list. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log WARNING: Index was created using MVRBTREESET as values container. This container is deprecated and is not supported any more. To avoid this message please drop and recreate indexes or perform DB export/import. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: Index OID is not automatic index and will be added as is. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: Index OID was added in DB index list. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: 7 indexes were restored successfully, 2 errors мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log INFO: Indexes restore after crash was finished. мар 06, 2014 4:17:23 PM com.orientechnologies.common.log.OLogManager log WARNING: Sun Unsafe direct memory implementation is going to be used, this implementation is not stable so please use JNA version instead. Exception in thread "main" java.lang.NullPointerException at com.stimulus.archiva.database.blueprints.TestCaseDB.main(TestCaseDB.java:26)

[andrii0lomakin]

@valenpo which site do you mean CI server http://helios.orientechnologies.com/job/orient-maven/lastStableBuild/com.orientechnologies$orientdb-community/ or sonatype repository https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-community/1.7-rc2-SNAPSHOT/ ?

[valenpo]

@laa https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-community/1.7-rc2-SNAPSHOT/