Open mervynzhang opened 9 months ago
mongorestore lealone 还有报错
mongorestore --drop --archive=authordb.bson 2023-11-14T15:43:31.890+0800 error getting server version: buildInfo.versionArray had fewer than 3 elements
我刚装了 mongodump 试了一下是可以的。 如是出现上面的 WARN 只是 lealone 发现客户端把连接关闭了而已。 那些表是 lealone 内部的。
mongorestore lealone 还有报错
mongorestore --drop --archive=authordb.bson 2023-11-14T15:43:31.890+0800 error getting server version: buildInfo.versionArray had fewer than 3 elements
buildInfo 命令还没有实现全部,给客户端的信息它要求太严所以解析错误了。
给buildInfo 加了versionArray 能跑到下一步了, 但有新的报错, 看起来像bug
case "buildInfo": {
BsonDocument document = new BsonDocument();
BsonArray versionArray = new BsonArray();
versionArray.add(new BsonInt32(6));
versionArray.add(new BsonInt32(0));
versionArray.add(new BsonInt32(0));
append(document, "version", "6.0.0");
document.append("versionArray", versionArray);
现在报错
17:11:25.423 [ScheduleService-3] ERROR org.lealone.plugins.mongo.server.MongoTask - Failed to execute command: insert
org.lealone.common.exceptions.JdbcSQLException: Column "_ID" not found [42122-0]
at org.lealone.common.exceptions.DbException.getJdbcSQLException(DbException.java:342) ~[classes/:?]
at org.lealone.common.exceptions.DbException.get(DbException.java:173) ~[classes/:?]
at org.lealone.common.exceptions.DbException.get(DbException.java:150) ~[classes/:?]
at org.lealone.db.table.Table.getColumn(Table.java:614) ~[classes/:?]
at org.lealone.plugins.mongo.bson.BsonBase.parseColumn(BsonBase.java:181) ~[classes/:?]
at org.lealone.plugins.mongo.bson.command.BCInsert.addRows(BCInsert.java:52) ~[classes/:?]
at org.lealone.plugins.mongo.bson.command.BCInsert.execute(BCInsert.java:33) ~[classes/:?]
at org.lealone.plugins.mongo.bson.command.BsonCommand.execute(BsonCommand.java:261) ~[classes/:?]
at org.lealone.plugins.mongo.server.MongoTask.run(MongoTask.java:44) [classes/:?]
at org.lealone.server.SessionInfo.runTask(SessionInfo.java:106) [classes/:?]
at org.lealone.server.SessionInfo.submitTask(SessionInfo.java:69) [classes/:?]
at org.lealone.plugins.mongo.server.MongoServerConnection.handleCommand(MongoServerConnection.java:249) [classes/:?]
at org.lealone.plugins.mongo.server.MongoServerConnection.handleMessage(MongoServerConnection.java:231) [classes/:?]
at org.lealone.plugins.mongo.server.MongoServerConnection.handle(MongoServerConnection.java:185) [classes/:?]
at org.lealone.net.nio.NioEventLoop.read(NioEventLoop.java:276) [classes/:?]
at org.lealone.net.nio.NioEventLoop.handleSelectedKeys(NioEventLoop.java:493) [classes/:?]
at org.lealone.server.GlobalScheduler.runEventLoop(GlobalScheduler.java:395) [classes/:?]
at org.lealone.server.GlobalScheduler.run(GlobalScheduler.java:100) [classes/:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
更新一下代码试试
已经是最新代码了。 我debug了下, 问题应该是这里
pymongo insert 新表的话, 这里table == null mongorestore的话, 这里 table = StandardTable@30 "PUBLIC.AUTHORS_RESTORE", columnMap 是空的, 所以会报“_id” 为空
如果 mongorestore 触发的是 create 命令,且 viewOn 参数没有指定, lealone 实现这个 create 命令就是建一张空表, 后续 mongorestore 再往这张表 insert 数据时 lealone 目前还没有实现动态增加字段的功能,所以会报告字段找不到。
如果 mongorestore 触发的是 create 命令,且 viewOn 参数没有指定, lealone 实现这个 create 命令就是建一张空表, 后续 mongorestore 再往这张表 insert 数据时 lealone 目前还没有实现动态增加字段的功能,所以会报告字段找不到。
这样的话,虽然是mongo的api, 但还是需要用固定的schema。 原来的字段是TIMESTAMP, 后边document改成string就报错了。 这样的话,数据基本没法导入了,原有数据里大量的同一个字段不同类型的情况。
pymongo.errors.OperationFailure: Cannot parse "TIMESTAMP" constant "date ya" [22007-0], full error: {'minWireVersion': 0, 'maxWireVersion': 17, 'ok': 0, 'n': 1, 'code': 22007, 'errmsg': 'Cannot parse "TIMESTAMP" constant "date ya" [22007-0]'}
如果 mongorestore 触发的是 create 命令,且 viewOn 参数没有指定, lealone 实现这个 create 命令就是建一张空表, 后续 mongorestore 再往这张表 insert 数据时 lealone 目前还没有实现动态增加字段的功能,所以会报告字段找不到。
这样的话,虽然是mongo的api, 但还是需要用固定的schema。 原来的字段是TIMESTAMP, 后边document改成string就报错了。 这样的话,数据基本没法导入了,原有数据里大量的同一个字段不同类型的情况。
pymongo.errors.OperationFailure: Cannot parse "TIMESTAMP" constant "date ya" [22007-0], full error: {'minWireVersion': 0, 'maxWireVersion': 17, 'ok': 0, 'n': 1, 'code': 22007, 'errmsg': 'Cannot parse "TIMESTAMP" constant "date ya" [22007-0]'}
这种动态功能容易实现,分析当前执行的文档字段个数和类型跟现有表结构比较一下,动作执行 alter 语句修改表结构就好了,都是体力活,只是优先级排在后面,现在忙着别的事,工期还没有排到 mongodb 插件。
mongodb、mysql、postgresql 三个内置插件依然是开发阶段,没有到生产级别的,不支持的特性后续补上就好了。 如果你有兴趣,也很欢迎你提交 PR。
如果 mongorestore 触发的是 create 命令,且 viewOn 参数没有指定, lealone 实现这个 create 命令就是建一张空表, 后续 mongorestore 再往这张表 insert 数据时 lealone 目前还没有实现动态增加字段的功能,所以会报告字段找不到。
这样的话,虽然是mongo的api, 但还是需要用固定的schema。 原来的字段是TIMESTAMP, 后边document改成string就报错了。 这样的话,数据基本没法导入了,原有数据里大量的同一个字段不同类型的情况。
pymongo.errors.OperationFailure: Cannot parse "TIMESTAMP" constant "date ya" [22007-0], full error: {'minWireVersion': 0, 'maxWireVersion': 17, 'ok': 0, 'n': 1, 'code': 22007, 'errmsg': 'Cannot parse "TIMESTAMP" constant "date ya" [22007-0]'}
这种动态功能容易实现,分析当前执行的文档字段个数和类型跟现有表结构比较一下,动态执行 alter 语句修改表结构就好了,都是体力活,只是优先级排在后面,现在忙着别的事,工期还没有排到 mongodb 插件。
一个字段想兼容多种类型无非是全转到字符串和二进制。
mongodb、mysql、postgresql 三个内置插件依然是开发阶段,没有到生产级别的,不支持的特性后续补上就好了。 如果你有兴趣,也很欢迎你提交 PR。
我主要做devopps,开发经验太少, 能读代码, 写新功能代码就有点力不从心了。
拉最新代码测试了下, 少量数据可以mongorestore了。 大量导入就报 NullPointerException
ERROR [FsyncService] [2023-12-07T10:12:08,766] Failed to sync redo log
java.lang.NullPointerException: null
at org.lealone.transaction.aote.log.RedoLogChunk.checkpoint(RedoLogChunk.java:268) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.transaction.aote.log.RedoLogChunk.save(RedoLogChunk.java:170) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.transaction.aote.log.RedoLog.save(RedoLog.java:123) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.transaction.aote.log.LogSyncService.sync(LogSyncService.java:112) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.transaction.aote.log.LogSyncService.run(LogSyncService.java:88) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.main.Lealone.run(Lealone.java:211) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.main.Lealone.start(Lealone.java:135) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.main.Lealone.start(Lealone.java:97) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.main.Lealone.main(Lealone.java:51) [lealone-6.0.0-SNAPSHOT.jar:?]
checkpoint 还有 bug,还未修复
报告一个新错误, 还是mongorestore 出来的
ERROR 16:54:38.628 Failed to execute command: insert
org.lealone.common.exceptions.JdbcSQLException: Column count does not match [21002-0]
at org.lealone.common.exceptions.DbException.getJdbcSQLException(DbException.java:342) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.common.exceptions.DbException.get(DbException.java:173) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.common.exceptions.DbException.get(DbException.java:150) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.common.exceptions.DbException.get(DbException.java:139) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.sql.dml.MerSert.prepare(MerSert.java:116) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.plugins.mongo.bson.command.BCInsert.addRows(BCInsert.java:83) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.plugins.mongo.bson.command.BCInsert.execute(BCInsert.java:36) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.plugins.mongo.bson.command.BsonCommand.execute(BsonCommand.java:262) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.plugins.mongo.server.MongoTask.run(MongoTask.java:44) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.server.scheduler.SessionInfo.runTask(SessionInfo.java:104) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.server.scheduler.SessionInfo.submitTask(SessionInfo.java:68) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.plugins.mongo.server.MongoServerConnection.handleCommand(MongoServerConnection.java:236) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.plugins.mongo.server.MongoServerConnection.handleMessage(MongoServerConnection.java:218) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.plugins.mongo.server.MongoServerConnection.handle(MongoServerConnection.java:172) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.net.nio.NioEventLoop.read(NioEventLoop.java:275) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.net.nio.NioEventLoop.handleSelectedKeys(NioEventLoop.java:492) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.net.NetScheduler.runEventLoop(NetScheduler.java:62) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.server.scheduler.GlobalScheduler.run(GlobalScheduler.java:70) [lealone-6.0.0-SNAPSHOT.jar:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
mongorestore 还有一个错,但lealone 没有报错
Hexadecimal string with odd number of characters: "xyz" [90003-0]
有新报错
ERROR 11:18:08.343 Failed to sync redo log
java.lang.NullPointerException: null
at org.lealone.storage.fs.FileStorage.sync(FileStorage.java:436) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.transaction.aote.log.RedoLogChunk.save(RedoLogChunk.java:119) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.transaction.aote.log.RedoLog.save(RedoLog.java:123) ~[lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.transaction.aote.log.LogSyncService.sync(LogSyncService.java:112) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.transaction.aote.log.LogSyncService.run(LogSyncService.java:88) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.main.Lealone.run(Lealone.java:211) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.main.Lealone.start(Lealone.java:135) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.main.Lealone.start(Lealone.java:97) [lealone-6.0.0-SNAPSHOT.jar:?]
at org.lealone.main.Lealone.main(Lealone.java:51) [lealone-6.0.0-SNAPSHOT.jar:?]
Exception in thread "ScheduleService-5" java.lang.IllegalStateException: Chunk too large, max size: 2147475455, current size: 2147530719 [6.0.0/2]
at org.lealone.common.util.DataUtils.newIllegalStateException(DataUtils.java:616)
at org.lealone.storage.aose.btree.page.Page.updateChunkAndPage(Page.java:353)
at org.lealone.storage.aose.btree.page.LeafPage.writeRowStorage(LeafPage.java:251)
at org.lealone.storage.aose.btree.page.LeafPage.writeUnsavedRecursive(LeafPage.java:226)
at org.lealone.storage.aose.btree.page.NodePage.writeUnsavedRecursive(NodePage.java:205)
at org.lealone.storage.aose.btree.page.NodePage.writeUnsavedRecursive(NodePage.java:205)
at org.lealone.storage.aose.btree.page.NodePage.writeUnsavedRecursive(NodePage.java:205)
at org.lealone.storage.aose.btree.page.NodePage.writeUnsavedRecursive(NodePage.java:205)
at org.lealone.storage.aose.btree.BTreeStorage.executeSave(BTreeStorage.java:365)
at org.lealone.storage.aose.btree.BTreeStorage.save(BTreeStorage.java:336)
at org.lealone.storage.aose.btree.BTreeMap.fullGc(BTreeMap.java:429)
at org.lealone.transaction.aote.CheckpointService.fullGc(CheckpointService.java:161)
at org.lealone.transaction.aote.AOTransactionEngine.fullGc(AOTransactionEngine.java:72)
at org.lealone.server.scheduler.GlobalScheduler.gc(GlobalScheduler.java:210)
at org.lealone.server.scheduler.GlobalScheduler.executeNextStatement(GlobalScheduler.java:223)
at org.lealone.server.scheduler.GlobalScheduler.run(GlobalScheduler.java:68)
at java.base/java.lang.Thread.run(Thread.java:829)
mongorestore的报错
2023-12-11T11:19:41.486+0800 finished restoring test.test (282053 documents, 0 failures)
2023-12-11T11:19:41.486+0800 Failed: test.test: error restoring from archive on stdin: General error: "java.lang.IllegalStateException: This map is closed [6.0.0/4]" [50000-0]
2023-12-11T11:19:41.486+0800 282053 document(s) restored successfully. 0 document(s) failed to restore
java.lang.IllegalStateException: Chunk too large, max size: 2147475455, current size: 2147530719
这个异常是没有及时刷脏页,导致 chunk 的大小超过2G了。这个问题先放着,后续再处理。
之前用apt install的mongodump/mongorestore, 版本有点老, 下载了https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu2004-x86_64-100.9.1.deb,
mongodump 可以用了,但导出的collection 比mongodb 多很多。
mongodb
lealone