packet too large , NPE - Githubissues

mervynzhang commented 7 months ago

mongodb 文档最大是16mb https://www.mongodb.com/docs/manual/core/document/#document-size-limit lealone 目前只支持8mb得，大文档就报错了 https://github.com/lealone/Lealone/blob/cdecc1fc31abce64cf15588e4615520c06ced0fe/lealone-net/src/main/java/com/lealone/net/nio/NioEventLoop.java#L72

我本地修改了下， mongorestore 可以跑完了，但读数据还有npe，客户端卡死


ERROR 11:41:41.876 Failed to execute command: aggregate
org.bson.BsonInvalidOperationException: Value expected to be of type DOCUMENT is of unexpected type INT32
        at org.bson.BsonValue.throwIfInvalidType(BsonValue.java:419) ~[lealone-6.0.0-SNAPSHOT.jar:?]
        at org.bson.BsonValue.asDocument(BsonValue.java:47) ~[lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.plugins.mongo.bson.command.BCAggregate.aggregate(BCAggregate.java:129) ~[lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.plugins.mongo.bson.command.BCAggregate.execute(BCAggregate.java:35) ~[lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.plugins.mongo.bson.command.BsonCommand.execute(BsonCommand.java:271) ~[lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.plugins.mongo.server.MongoTask.run(MongoTask.java:45) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.server.scheduler.SessionInfo.runTask(SessionInfo.java:104) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.server.scheduler.SessionInfo.submitTask(SessionInfo.java:68) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.plugins.mongo.server.MongoServerConnection.handleCommand(MongoServerConnection.java:237) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.plugins.mongo.server.MongoServerConnection.handleMessage(MongoServerConnection.java:219) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.plugins.mongo.server.MongoServerConnection.handle(MongoServerConnection.java:173) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.net.nio.NioEventLoop.read(NioEventLoop.java:275) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.net.nio.NioEventLoop.handleSelectedKeys(NioEventLoop.java:492) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.net.NetScheduler.runEventLoop(NetScheduler.java:62) [lealone-6.0.0-SNAPSHOT.jar:?]
        at com.lealone.server.scheduler.GlobalScheduler.run(GlobalScheduler.java:70) [lealone-6.0.0-SNAPSHOT.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Exception in thread "ScheduleService-3" java.lang.NullPointerException
        at com.lealone.server.scheduler.SessionInfo.sendError(SessionInfo.java:143)
        at com.lealone.server.scheduler.GlobalScheduler.executeNextStatement(GlobalScheduler.java:261)
        at com.lealone.server.scheduler.GlobalScheduler.run(GlobalScheduler.java:68)
        at java.base/java.lang.Thread.run(Thread.java:829)
Exception in thread "ScheduleService-4" java.lang.NullPointerException
        at com.lealone.server.scheduler.SessionInfo.sendError(SessionInfo.java:143)
        at com.lealone.server.scheduler.GlobalScheduler.executeNextStatement(GlobalScheduler.java:261)
        at com.lealone.server.scheduler.GlobalScheduler.run(GlobalScheduler.java:68)
        at java.base/java.lang.Thread.run(Thread.java:829)

codefollower commented 7 months ago

maxPacketSize 的问题只需要在 lealone.yaml 文件中加上以下配置即可:

scheduler:
    parameters: {
        max_packet_size: 16777216 # 16M
    }

codefollower commented 7 months ago

Value expected to be of type DOCUMENT is of unexpected type INT32

aggregate 命令有些语法还没有完全支持

mervynzhang commented 7 months ago

我mongorestore了3，4g数据，读不出来，过会有OOM 出来


Exception in thread "ScheduleService-0" java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.ConcurrentHashMap$ValuesView.iterator(ConcurrentHashMap.java:4730)
        at java.base/java.util.concurrent.ConcurrentHashMap$CollectionView.toArray(ConcurrentHashMap.java:4454)
        at java.base/java.util.ArrayList.addAll(ArrayList.java:702)
        at com.lealone.transaction.aote.tm.ConcurrentTransactionManager.currentTransactions(ConcurrentTransactionManager.java:41)
        at com.lealone.transaction.aote.AOTransactionEngine.currentTransactions(AOTransactionEngine.java:107)
        at com.lealone.storage.aose.btree.page.PageReference.canGc(PageReference.java:330)
        at com.lealone.storage.aose.btree.BTreeGC.collect(BTreeGC.java:180)
        at com.lealone.storage.aose.btree.BTreeGC.lambda$collect$1(BTreeGC.java:177)
        at com.lealone.storage.aose.btree.BTreeGC$$Lambda$148/0x0000000840244440.accept(Unknown Source)
        at com.lealone.storage.aose.btree.BTreeGC.forEachPage(BTreeGC.java:246)
        at com.lealone.storage.aose.btree.BTreeGC.collect(BTreeGC.java:176)
        at com.lealone.storage.aose.btree.BTreeGC.lambda$collect$1(BTreeGC.java:177)
        at com.lealone.storage.aose.btree.BTreeGC$$Lambda$148/0x0000000840244440.accept(Unknown Source)
        at com.lealone.storage.aose.btree.BTreeGC.forEachPage(BTreeGC.java:246)
        at com.lealone.storage.aose.btree.BTreeGC.collect(BTreeGC.java:176)
        at com.lealone.storage.aose.btree.BTreeGC.lambda$collect$1(BTreeGC.java:177)
        at com.lealone.storage.aose.btree.BTreeGC$$Lambda$148/0x0000000840244440.accept(Unknown Source)
        at com.lealone.storage.aose.btree.BTreeGC.forEachPage(BTreeGC.java:246)
        at com.lealone.storage.aose.btree.BTreeGC.collect(BTreeGC.java:176)
        at com.lealone.storage.aose.btree.BTreeGC.lru(BTreeGC.java:161)
        at com.lealone.storage.aose.btree.BTreeGC.gc(BTreeGC.java:99)
        at com.lealone.storage.aose.btree.BTreeGC.gc(BTreeGC.java:80)
        at com.lealone.storage.aose.btree.BTreeGC.fullGc(BTreeGC.java:73)
        at com.lealone.storage.aose.btree.BTreeMap.fullGc(BTreeMap.java:430)
        at com.lealone.transaction.aote.CheckpointService.fullGc(CheckpointService.java:161)
        at com.lealone.transaction.aote.AOTransactionEngine.fullGc(AOTransactionEngine.java:72)
        at com.lealone.server.scheduler.GlobalScheduler.gc(GlobalScheduler.java:210)
        at com.lealone.server.scheduler.GlobalScheduler.executeNextStatement(GlobalScheduler.java:223)
        at com.lealone.server.scheduler.GlobalScheduler.run(GlobalScheduler.java:68)
        at java.base/java.lang.Thread.run(Thread.java:829)
Exception in thread "ScheduleService-2" java.lang.NullPointerException
        at com.lealone.server.scheduler.SessionInfo.sendError(SessionInfo.java:143)
        at com.lealone.server.scheduler.GlobalScheduler.executeNextStatement(GlobalScheduler.java:261)
        at com.lealone.server.scheduler.GlobalScheduler.run(GlobalScheduler.java:68)
        at java.base/java.lang.Thread.run(Thread.java:829)

codefollower commented 7 months ago

mongodb 文档太大，还没有试过，可能 gc 时会有问题。目前一个 page 就16K，如果一个 mongodb 文档就大于16K了，一个 page 只能存一个文档。

lealone / Lealone

packet too large , NPE #219