search

orientechnologies / orientdb

OrientDB is the most versatile DBMS supporting Graph, Document, Reactive, Full-Text and Geospatial models in one Multi-Model product. OrientDB can run distributed (Multi-Master), supports SQL, ACID Transactions, Full-Text indexing and Reactive Queries.
https://orientdb.dev
Apache License 2.0
4.75k stars 871 forks source link

Getting NullPointerException on database startup (at com.orientechnologies.orient.core.storage.cache.chm.AsyncReadCache.releaseFromRead) #9367

Open earshinov opened 4 years ago

earshinov commented 4 years ago

OrientDB Version: 3.0.31

Java Version:

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

OS:

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian

Expected behavior

No exceptions on startup.

Actual behavior

         ,,.     `,                                              VELOCE  
       ``        `.                                                          
                 ``                                       www.orientdb.com
                 `                                    
2020-08-07 14:59:26:321 INFO  Detected limit of amount of simultaneously open files is 1048576,  limit of open files for disk cache will be set to 523776 [ONative]
2020-08-07 14:59:26:332 INFO  Loading configuration from: /orientdb/config/orientdb-server-config.xml... [OServerConfigurationLoaderXml]
2020-08-07 14:59:26:486 INFO  OrientDB Server v3.0.31 - Veloce (build 2a8412458048d53cd1e1a3544d19b87c190c2d73, branch UNKNOWN) is starting up... [OServer]
2020-08-07 14:59:26:531 INFO  67557953536 B/64428 MB/62 GB of physical memory were detected on machine [ONative]
2020-08-07 14:59:26:532 INFO  Soft memory limit for this process is set to -1 B/-1 MB/-1 GB [ONative]
2020-08-07 14:59:26:532 INFO  Hard memory limit for this process is set to -1 B/-1 MB/-1 GB [ONative]
2020-08-07 14:59:26:533 INFO  Path to 'memory' cgroup is '/kubepods/burstable/pod381302ef-d709-11ea-9320-525400a3535a/d8d499430f21742087c284f2ee53229e346b4baac671b4b44099b5ad521cb32d' [ONative]
2020-08-07 14:59:26:535 INFO  Mounting path for memory cgroup controller is '/sys/fs/cgroup/memory' [ONative]
2020-08-07 14:59:26:535 INFO  Can not find '/sys/fs/cgroup/memory/kubepods/burstable/pod381302ef-d709-11ea-9320-525400a3535a/d8d499430f21742087c284f2ee53229e346b4baac671b4b44099b5ad521cb32d' path for memory cgroup, it is supposed that process is running in container, will try to read root '/sys/fs/cgroup/memory' memory cgroup data [ONative]
2020-08-07 14:59:26:535 INFO  cgroup soft memory limit is 9223372036854771712 B/8796093022207 MB/8589934591 GB [ONative]
2020-08-07 14:59:26:536 INFO  cgroup hard memory limit is 524288000 B/500 MB/0 GB [ONative]
2020-08-07 14:59:26:536 INFO  Detected memory limit for current process is 524288000 B/500 MB/0 GB [ONative]
2020-08-07 14:59:26:537 INFO  JVM can use maximum 371MB of heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2020-08-07 14:59:26:537 INFO  Because OrientDB is running inside a container 12% of memory will be left unallocated according to the setting 'memory.leftToContainer' not taking into account heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2020-08-07 14:59:26:539 INFO  OrientDB auto-config DISKCACHE=68MB (heap=371MB os=500MB) [orientechnologies]
2020-08-07 14:59:26:543 INFO  System is started under an effective user : `root` [OEngineLocalPaginated]
2020-08-07 14:59:26:544 INFO  Allocation of 1033 pages. [OEngineLocalPaginated]
2020-08-07 14:59:26:643 INFO  WAL maximum segment size is set to 190,617 MB [OrientDBDistributed]
2020-08-07 14:59:26:644 INFO  Databases directory: /orientdb/databases [OServer]
2020-08-07 14:59:26:710 INFO  Page size for WAL located in /orientdb/databases/OSystem is set to 4096 bytes. [OCASDiskWriteAheadLog]
2020-08-07 14:59:27:020 INFO  Storage 'plocal:/orientdb/databases/OSystem' is opened under OrientDB distribution : 3.0.31 - Veloce (build 2a8412458048d53cd1e1a3544d19b87c190c2d73, branch UNKNOWN) [OLocalPaginatedStorage]Exception `558633DB` in storage `plocal:/orientdb/databases/OSystem`: 3.0.31 - Veloce (build 2a8412458048d53cd1e1a3544d19b87c190c2d73, branch UNKNOWN)
java.lang.NullPointerException
    at com.orientechnologies.orient.core.storage.cache.chm.AsyncReadCache.releaseFromRead(AsyncReadCache.java:226)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.releasePageFromRead(ODurableComponent.java:167)
    at com.orientechnologies.orient.core.storage.cluster.v1.OClusterPositionMapV1.getFirstPosition(OClusterPositionMapV1.java:511)
    at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.getFirstPosition(OPaginatedClusterV1.java:1231)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.getClusterDataRange(OAbstractPaginatedStorage.java:1349)
    at com.orientechnologies.orient.core.iterator.ORecordIteratorCluster.<init>(ORecordIteratorCluster.java:62)
    at com.orientechnologies.orient.core.iterator.ORecordIteratorCluster.<init>(ORecordIteratorCluster.java:44)
    at com.orientechnologies.orient.core.sql.executor.FetchFromClusterExecutionStep.syncPull(FetchFromClusterExecutionStep.java:53)
    at com.orientechnologies.orient.core.sql.executor.FetchFromClassExecutionStep$1.hasNext(FetchFromClassExecutionStep.java:128)
    at com.orientechnologies.orient.core.sql.executor.OrderByStep.init(OrderByStep.java:101)
    at com.orientechnologies.orient.core.sql.executor.OrderByStep.lambda$syncPull$0(OrderByStep.java:40)
    at java.util.Optional.ifPresent(Optional.java:159)
    at com.orientechnologies.orient.core.sql.executor.OrderByStep.syncPull(OrderByStep.java:40)
    at com.orientechnologies.orient.core.sql.executor.OSelectExecutionPlan.fetchNext(OSelectExecutionPlan.java:37)
    at com.orientechnologies.orient.core.sql.parser.OLocalResultSet.fetchNext(OLocalResultSet.java:39)
    at com.orientechnologies.orient.core.sql.parser.OLocalResultSet.<init>(OLocalResultSet.java:30)
    at com.orientechnologies.orient.core.sql.parser.OSelectStatement.execute(OSelectStatement.java:276)
    at com.orientechnologies.orient.core.sql.parser.OStatement.execute(OStatement.java:79)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.query(ODatabaseDocumentEmbedded.java:538)
    at com.orientechnologies.orient.core.metadata.function.OFunctionLibraryImpl.load(OFunctionLibraryImpl.java:83)
    at com.orientechnologies.orient.core.db.OSharedContextEmbedded.load(OSharedContextEmbedded.java:69)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.loadMetadata(ODatabaseDocumentEmbedded.java:291)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.init(ODatabaseDocumentEmbedded.java:161)
    at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:262)
    at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:56)
    at com.orientechnologies.orient.server.OSystemDatabase.openSystemDatabase(OSystemDatabase.java:97)
    at com.orientechnologies.orient.server.OServer.initSystemDatabase(OServer.java:1212)
    at com.orientechnologies.orient.server.OServer.activate(OServer.java:421)
    at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:48)
2020-08-07 14:59:27:222 INFO  - shutdown storage: OSystem... [OrientDBDistributed]Error during server execution
com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database 'OSystem'
    at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:268)
    at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:56)
    at com.orientechnologies.orient.server.OSystemDatabase.openSystemDatabase(OSystemDatabase.java:97)
    at com.orientechnologies.orient.server.OSystemDatabase.checkServerId(OSystemDatabase.java:174)
    at com.orientechnologies.orient.server.OSystemDatabase.init(OSystemDatabase.java:162)
    at com.orientechnologies.orient.server.OSystemDatabase.<init>(OSystemDatabase.java:49)
    at com.orientechnologies.orient.server.OServer.initSystemDatabase(OServer.java:1212)
    at com.orientechnologies.orient.server.OServer.activate(OServer.java:421)
    at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:48)
Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database url=plocal:/orientdb/databases/OSystem
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.init(ODatabaseDocumentEmbedded.java:175)
    at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthorization(OrientDBEmbedded.java:262)
    ... 8 more
Caused by: java.lang.NullPointerException
    at com.orientechnologies.orient.core.storage.cache.chm.AsyncReadCache.releaseFromRead(AsyncReadCache.java:226)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.releasePageFromRead(ODurableComponent.java:167)
    at com.orientechnologies.orient.core.storage.cluster.v1.OClusterPositionMapV1.getFirstPosition(OClusterPositionMapV1.java:511)
    at com.orientechnologies.orient.core.storage.cluster.v1.OPaginatedClusterV1.getFirstPosition(OPaginatedClusterV1.java:1231)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.getClusterDataRange(OAbstractPaginatedStorage.java:1349)
    at com.orientechnologies.orient.core.iterator.ORecordIteratorCluster.<init>(ORecordIteratorCluster.java:62)
    at com.orientechnologies.orient.core.iterator.ORecordIteratorCluster.<init>(ORecordIteratorCluster.java:44)
    at com.orientechnologies.orient.core.sql.executor.FetchFromClusterExecutionStep.syncPull(FetchFromClusterExecutionStep.java:53)
    at com.orientechnologies.orient.core.sql.executor.FetchFromClassExecutionStep$1.hasNext(FetchFromClassExecutionStep.java:128)
    at com.orientechnologies.orient.core.sql.executor.OrderByStep.init(OrderByStep.java:101)
    at com.orientechnologies.orient.core.sql.executor.OrderByStep.lambda$syncPull$0(OrderByStep.java:40)
    at java.util.Optional.ifPresent(Optional.java:159)
    at com.orientechnologies.orient.core.sql.executor.OrderByStep.syncPull(OrderByStep.java:40)
    at com.orientechnologies.orient.core.sql.executor.OSelectExecutionPlan.fetchNext(OSelectExecutionPlan.java:37)
    at com.orientechnologies.orient.core.sql.parser.OLocalResultSet.fetchNext(OLocalResultSet.java:39)
    at com.orientechnologies.orient.core.sql.parser.OLocalResultSet.<init>(OLocalResultSet.java:30)
    at com.orientechnologies.orient.core.sql.parser.OSelectStatement.execute(OSelectStatement.java:276)
    at com.orientechnologies.orient.core.sql.parser.OStatement.execute(OStatement.java:79)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.query(ODatabaseDocumentEmbedded.java:538)
    at com.orientechnologies.orient.core.metadata.function.OFunctionLibraryImpl.load(OFunctionLibraryImpl.java:83)
    at com.orientechnologies.orient.core.db.OSharedContextEmbedded.load(OSharedContextEmbedded.java:69)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.loadMetadata(ODatabaseDocumentEmbedded.java:291)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.init(ODatabaseDocumentEmbedded.java:161)
    ... 9 more

Steps to reproduce

None, but maybe you have come clues?

earshinov commented 4 years ago

Is it something that could possibly be fixed by upgrading to 3.0.32?

andrii0lomakin commented 4 years ago

Hi @earshinov . OSystem database is not critical, you can delete it. But in general, the problem is caused by the fact that you use V1 version of cluster. You should use V2 version of cluster, which is durable. If your database is not big, you can export to JSNO and then import it and you will perform the migration. In future, we are planning to implement automatic migration between versions.

andrii0lomakin commented 4 years ago

Hi @earshinov , did removal of system database resolve your issue?

earshinov commented 4 years ago

@laa , Yes, rm -rf databases/OSystem/ helped, thanks! I will try migrating to cluster V2 a bit later.

earshinov commented 4 years ago

@laa , Can you please point me to the documentation on cluster versions (v1 vs v2)? I must be missing something: as far as I understand, in the test setup I am tinkering with at the moment I shouldn't be using OrientDB in distributed mode at all because I only have a single server.

andrii0lomakin commented 4 years ago

@earshinov by cluster I mean storage cluster, disk-based cluster, like a table, not distributed cluster.

earshinov commented 4 years ago

@laa , Could you give me a hint about how to switch between cluster v1 and v2?

earshinov commented 4 years ago

One of the reasons why I'm asking is that I recreated the database recently. Given that OrientDB used cluster v1 when recreating the database (apparently), why would it switch to cluster v2 when I export and import the database?

I'm still trying to glean some information on cluster v1 and v2. Is cluster v2 something that appeared between 3.0.30 and 3.0.33? I didn't use older versions of OrientDB. I can't find any information about cluster v2 on the OrientDB 3.0 Release Notes page.

andrii0lomakin commented 4 years ago

Hi @earshinov if I remember correctly I have already answered your question. Have not I?

earshinov commented 4 years ago

@laa, We didn't experience this issue any more, and also after migrating to OrientDB 3.1.3 we have storage.cluster.version = 2 in global configuration (studio/index.html#/dashboard/general). I guess, everything should be fine and this ticket can be closed, at least for now.