Closed rikaze closed 4 years ago
@rikaze did you try upgrade back and execute the same query? I explain why I am asking. Till 2.2.25 version if I am not mistaken under some rare circumstances data can be not written to the disk. It is fixed in later versions though. Could you try on 2.2.21 version and provide feedback about results?
@laa yes, I reverted back to 2.2.21, and it works fine.
@rikaze is it possible to get the database to reproduce the issue?
@laa sorry, the db is production with confidential data. any other workarond?
@rikaze yes sure I will add some logging data in special distribution. So you will execute a query on it and send me logs back. It will take a couple of iterations but I suppose will work.
@laa that would be great.
@laa any update on this? any additional information needed from my side? may I have the queries to debug?
@rikaze let me finish my current task and I will send you some :-)
Hi @rikaze , could you open your DB using this distribution and send results back to me?
Which distribution do you refer? share me the download URL or build method?
com.orientechnologies.orient.core.exception.OStorageException: File with name ___exportImportRIDMap.him does not exist in storage XXX DB name="XXX" at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.loadFile(OWOWCache.java:417) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.openFile(ODurableComponent.java:183) at com.orientechnologies.orient.core.index.hashindex.local.OLocalHashTable.load(OLocalHashTable.java:671) at com.orientechnologies.orient.core.index.engine.OHashTableIndexEngine.load(OHashTableIndexEngine.java:119) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.loadExternalIndexEngine(OAbstractPaginatedStorage.java:1809) at com.orientechnologies.orient.core.index.OIndexAbstract.loadFromConfiguration(OIndexAbstract.java:329) at com.orientechnologies.orient.core.index.OIndexManagerShared.fromStream(OIndexManagerShared.java:401) at com.orientechnologies.orient.core.type.ODocumentWrapperNoClass.reload(ODocumentWrapperNoClass.java:74) at com.orientechnologies.orient.core.index.OIndexManagerAbstract.load(OIndexManagerAbstract.java:85) at com.orientechnologies.orient.core.index.OIndexManagerAbstract.load(OIndexManagerAbstract.java:57) at com.orientechnologies.orient.core.metadata.OMetadataDefault$2.call(OMetadataDefault.java:191) at com.orientechnologies.orient.core.metadata.OMetadataDefault$2.call(OMetadataDefault.java:181) at com.orientechnologies.common.concur.resource.OSharedContainerImpl.getResource(OSharedContainerImpl.java:62) at com.orientechnologies.orient.core.storage.OStorageAbstract.getResource(OStorageAbstract.java:143) at com.orientechnologies.orient.core.metadata.OMetadataDefault.init(OMetadataDefault.java:181) at com.orientechnologies.orient.core.metadata.OMetadataDefault.load(OMetadataDefault.java:89) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.initAtFirstOpen(ODatabaseDocumentTx.java:3179) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:270) at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:962) at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:932) at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:919) at com.orientechnologies.orient.server.network.protocol.http.command.OServerCommandAuthenticatedDbAbstract.authenticate(OServerCommandAuthenticatedDbAbstract.java:164) at com.orientechnologies.orient.server.network.protocol.http.command.OServerCommandAuthenticatedDbAbstract.beforeExecute(OServerCommandAuthenticatedDbAbstract.java:124) at com.orientechnologies.orient.server.network.protocol.http.command.get.OServerCommandGetConnect.beforeExecute(OServerCommandGetConnect.java:50) at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:166) at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:621) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)
I copied "___exportImportRIDMap.hi*" files from a backup into database folder, then the above exception disappeared. But the original NULL pointer exception still exists.
@rikaze without any additional messages just NPE? Could you send it to me ?
Caused by: java.lang.NullPointerException
at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.load(OSBTreeBonsaiLocal.java:500)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerShared.loadTree(OSBTreeCollectionManagerShared.java:107)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.loadSBTree(OSBTreeCollectionManagerAbstract.java:174)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.loadTree(OSBTreeRidBag.java:976)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.access$800(OSBTreeRidBag.java:59)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.init(OSBTreeRidBag.java:501)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.
java.lang.NullPointerException
at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.load(OSBTreeBonsaiLocal.java:500)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerShared.loadTree(OSBTreeCollectionManagerShared.java:107)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.loadSBTree(OSBTreeCollectionManagerAbstract.java:174)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.loadTree(OSBTreeRidBag.java:976)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.access$800(OSBTreeRidBag.java:59)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.init(OSBTreeRidBag.java:501)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.
@rikaze could you look for the message like "Root entry is null, root pointer is"?
i cleaned up the log folder, and started the server with brand new log files. no message like this found.
btw, with your package, i couldn't start the server with sh files in bin folder, so replaced the bin folder with my working files. anything need to be modified in th shell scripts to enable the log?
@laa Sorry, I found the message you referred.
SEVER {db=XXX} Root entry is null, root pointer is OBonsaiBucketPointer{pageIndex=10, pageOffset=61440} file is filled up to 8 [OSBTreeBonsaiLocal]$ANSI{green {db=XXX}} Exception 460B6A99
in storage XXX
Hi @rikaze , unfortunately, I have to ask you to debug a bit. Could you setup breakpoint on NPE.
Then could you go here https://i.imgur.com/BeWYbps.png in a debugger and then evaluate following expression owner.getIdentity()
and send it to me. I will provide you further instructions. About.sh files. I have built on windows so probably that is the reason. You can use your own .sh files does not matter.
But how to setup breakpoint. I assume it wouldn't be easy to setup a local dev environment with IDE. Any easier way to setup the breakpoint for remote debug?
@rikaze I assume that you connect to your server using ssh connection. In such case, you can create a tunnel by
addition ofthe following flag while you establish ssh connection -L 5005:localhost:5005
. In .sh file add following parameter inside of java call -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
and you can establish remote debugggin session.
which codeline should I setup? I am using eclipse, can you share me the 1-click way to setup?
Hello, any document could I refer to setup the build env in eclipse?
@rikaze I will send you new distribution soon which should print all information which I need to the log.
okay, wait for it.
Hi @rikaze could you try this build ?https://drive.google.com/file/d/10wRN4TySKw3G5QPsDqgq86tWrSOnEvZO/view?usp=sharing
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.updateCache(O2QCache.java:1098)
at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:346)
at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:291)
at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:148)
at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:142)
at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.load(OSBTreeBonsaiLocal.java:499)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerShared.loadTree(OSBTreeCollectionManagerShared.java:108)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.loadSBTree(OSBTreeCollectionManagerAbstract.java:175)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.loadTree(OSBTreeRidBag.java:976)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.access$800(OSBTreeRidBag.java:59)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.init(OSBTreeRidBag.java:501)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.
@rikaze is it the same query on the same content ?
yes, all the same.
any update on this?
@rikaze could you try this one https://drive.google.com/file/d/10wRN4TySKw3G5QPsDqgq86tWrSOnEvZO/view?usp=sharing
Root entry is null, root pointer is OBonsaiBucketPointer{pageIndex=10, pageOffset=61440} file is filled up to 8 ridbag owner is #133:0 [OSBTreeBonsaiLocal]$ANSI{green {db=XXXX}} Exception 1C608B9B
in storage XXXX
Hello, @laa any clue with the error message?
Hi @rikaze . Sorry for the delay, I have been busy with the implementation of a big feature. Could you try this distribution https://drive.google.com/file/d/10wRN4TySKw3G5QPsDqgq86tWrSOnEvZO/view?usp=sharing ? Hope next step will be sooner.
Hi @rikaze . Could you try distribution.
Root entry is null, root pointer is OBonsaiBucketPointer{pageIndex=10, pageOffset=61440} file is filled up to 8 ridbag owner is #133:0, field name
out_IsProductOfEquipment [OSBTreeBonsaiLocal]$ANSI{green {db=XXXX}} Exception 0B38CB51
in storage XXXX
Hi @rikaze could you try this distribution https://drive.google.com/file/d/1O-USJ_u5fiHrB8i2haKmLQkeLIPIJoMV/view?usp=sharing and check the output? There should not be an exception but a log message.
Yes, no exception now. See log: SEVER {db=XXXX} Root pointer is OBonsaiBucketPointer{pageIndex=10, pageOffset=61440} file is filled up to 22 ridbag owner is #133:0, field name out_IsProductOfEquipment [OSBTreeBonsaiLocal]
Hah, I see sorry but we need a couple of more iterations to find a root cause. I will provide the distribution tomorrow.
Hello, @laa will you include the fix in the next release?
@laa is the fix okay now?
Hi @rikaze we are now busy with commercial customers, once I complete my task I will be back to this issue. Sorry for the delay.
@laa did you get change to look into this issue?
Hi, if that is still actual we can work on it. Otherwise the issue will be closed.
OrientDB Version: 2.2.22
Java Version: 1.8.0
OS: CentOS 7
Expected behavior
Upgrade from 2.2.21, copy databases folder into new installation, should work as before.
Actual behavior
Some SQL failed to run some query (for example:
select out('IsProductOfEquipment') from Product
) in Studio, NullPointerException reported.Tried 2.2.22 to 2.2.35, all same error, seems the bug was introduced since 2.2.22.
However, following queries can be executed without error:
IsProductOfEquipment
is from vertexProduct
to vertexEquipment
.Steps to reproduce
com.orientechnologies.orient.core.exception.OCommandExecutionException: Error on execution of command: sql.select out('IsProductOfEquipment') from Product at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3221) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3138) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.execute(OServerCommandPostCommand.java:106) at com.orientechnologies.orient.graph.server.command.OServerCommandPostCommandGraph.execute(OServerCommandPostCommandGraph.java:37) at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:169) at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:621) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) Caused by: java.lang.NullPointerException at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.load(OSBTreeBonsaiLocal.java:495) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerShared.loadTree(OSBTreeCollectionManagerShared.java:107) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.loadSBTree(OSBTreeCollectionManagerAbstract.java:174) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.loadTree(OSBTreeRidBag.java:976) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.access$800(OSBTreeRidBag.java:59) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.init(OSBTreeRidBag.java:501) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.(OSBTreeRidBag.java:437)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.rawIterator(OSBTreeRidBag.java:560)
at com.orientechnologies.orient.core.db.record.ridbag.ORidBag.rawIterator(ORidBag.java:161)
at com.tinkerpop.blueprints.impls.orient.OrientVertex.getVertices(OrientVertex.java:435)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove.v2v(OSQLFunctionMove.java:103)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionOut.move(OSQLFunctionOut.java:54)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove$1$2.call(OSQLFunctionMove.java:87)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove$1$2.call(OSQLFunctionMove.java:84)
at com.orientechnologies.orient.core.sql.OSQLEngine.foreachRecord(OSQLEngine.java:294)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove$1.call(OSQLFunctionMove.java:84)
at com.orientechnologies.orient.graph.sql.OGraphCommandExecutorSQLFactory.runWithAnyGraph(OGraphCommandExecutorSQLFactory.java:210)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove.execute(OSQLFunctionMove.java:69)
at com.orientechnologies.orient.core.sql.functions.OSQLFunctionRuntime.execute(OSQLFunctionRuntime.java:128)
at com.orientechnologies.orient.core.sql.ORuntimeResult.applyRecord(ORuntimeResult.java:141)
at com.orientechnologies.orient.core.sql.ORuntimeResult.getProjectionResult(ORuntimeResult.java:255)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.addResult(OCommandExecutorSQLSelect.java:707)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.handleResult(OCommandExecutorSQLSelect.java:673)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:630)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1619)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1566)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:525)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:488)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3200)
... 7 more
Internal server error:
java.lang.NullPointerException at com.orientechnologies.orient.core.index.sbtreebonsai.local.OSBTreeBonsaiLocal.load(OSBTreeBonsaiLocal.java:495) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerShared.loadTree(OSBTreeCollectionManagerShared.java:107) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeCollectionManagerAbstract.loadSBTree(OSBTreeCollectionManagerAbstract.java:174) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.loadTree(OSBTreeRidBag.java:976) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.access$800(OSBTreeRidBag.java:59) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.init(OSBTreeRidBag.java:501) at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag$SBTreeMapEntryIterator.(OSBTreeRidBag.java:437)
at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.rawIterator(OSBTreeRidBag.java:560)
at com.orientechnologies.orient.core.db.record.ridbag.ORidBag.rawIterator(ORidBag.java:161)
at com.tinkerpop.blueprints.impls.orient.OrientVertex.getVertices(OrientVertex.java:435)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove.v2v(OSQLFunctionMove.java:103)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionOut.move(OSQLFunctionOut.java:54)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove$1$2.call(OSQLFunctionMove.java:87)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove$1$2.call(OSQLFunctionMove.java:84)
at com.orientechnologies.orient.core.sql.OSQLEngine.foreachRecord(OSQLEngine.java:294)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove$1.call(OSQLFunctionMove.java:84)
at com.orientechnologies.orient.graph.sql.OGraphCommandExecutorSQLFactory.runWithAnyGraph(OGraphCommandExecutorSQLFactory.java:210)
at com.orientechnologies.orient.graph.sql.functions.OSQLFunctionMove.execute(OSQLFunctionMove.java:69)
at com.orientechnologies.orient.core.sql.functions.OSQLFunctionRuntime.execute(OSQLFunctionRuntime.java:128)
at com.orientechnologies.orient.core.sql.ORuntimeResult.applyRecord(ORuntimeResult.java:141)
at com.orientechnologies.orient.core.sql.ORuntimeResult.getProjectionResult(ORuntimeResult.java:255)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.addResult(OCommandExecutorSQLSelect.java:707)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.handleResult(OCommandExecutorSQLSelect.java:673)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:630)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1619)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1566)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:525)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:488)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3200)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3138)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.execute(OServerCommandPostCommand.java:106)
at com.orientechnologies.orient.graph.server.command.OServerCommandPostCommandGraph.execute(OServerCommandPostCommandGraph.java:37)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:169)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:621)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)