search

apache / incubator-hugegraph

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)
https://hugegraph.apache.org
Apache License 2.0
2.62k stars 518 forks source link

升级 hugegraph 版本的时候迁移图的方案 #452

Closed suesunss closed 2 years ago

suesunss commented 5 years ago

Expected behavior 期望表现

使用 RocksDB 作为存储后端，在升级到新版本的 HugeGraph 后（master，或新的 release 版本）后，使用之前同一份的数据文件.

Actual behavior 实际表现

使用 RocksDB 作为存储后端，在升级到新版本的 HugeGraph 后（master，或新的 release 版本）后，使用之前同一份的数据文件，使用和之前的版本相同的图配置文件（*.properties），启动 HugeGraphServer 服务器成功。

但是，无法查询到数据，只有查询所有图名这样的 API 能用，其他的比如『分页查询所有顶点』之类的 API 都无法使用。比如分页查询所有顶点的时候，RESTful API 返回：

{
    "exception": "class com.baidu.hugegraph.HugeException",
    "message": "Failed to serialize vertices",
    "cause": "java.lang.IllegalArgumentException: Undefined vertex label id: '4'"
}
2019-04-11 22:11:14 127261 [gremlin-server-exec-4] [WARN ] org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler [] - Invalid request - responding with 500 Internal Server Error and Undefined property key: 'name'
java.lang.IllegalArgumentException: Undefined property key: 'name'
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:146) ~[guava-19.0.jar:?]
    at com.baidu.hugegraph.util.E.checkArgument(E.java:56) ~[hugegraph-common-1.5.8.jar:1.5.8.0]
    at com.baidu.hugegraph.HugeGraph.propertyKey(HugeGraph.java:416) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at com.baidu.hugegraph.traversal.optimize.TraversalUtil.convPredicateValue(TraversalUtil.java:489) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at com.baidu.hugegraph.traversal.optimize.TraversalUtil.convHasStep(TraversalUtil.java:480) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at com.baidu.hugegraph.traversal.optimize.TraversalUtil.convAllHasSteps(TraversalUtil.java:473) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at com.baidu.hugegraph.traversal.optimize.HugeGraphStepStrategy.apply(HugeGraphStepStrategy.java:49) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversalStrategies.applyStrategies(DefaultTraversalStrategies.java:86) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.applyStrategies(DefaultTraversal.java:119) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:191) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.fill(IteratorUtils.java:62) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.list(IteratorUtils.java:85) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.asList(IteratorUtils.java:382) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:239) ~[gremlin-server-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:320) ~[gremlin-groovy-3.2.5.jar:3.2.5]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
    at com.baidu.hugegraph.auth.HugeGraphAuthProxy$ContextTask.run(HugeGraphAuthProxy.java:282) [hugegraph-api-0.9.2.jar:0.37.0.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2019-04-11 22:11:14 127261 [gremlin-server-exec-4] [WARN ] org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler [] - Invalid request - responding with 500 Internal Server Error and Undefined property key: 'name'
java.lang.IllegalArgumentException: Undefined property key: 'name'
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:146) ~[guava-19.0.jar:?]
    at com.baidu.hugegraph.util.E.checkArgument(E.java:56) ~[hugegraph-common-1.5.8.jar:1.5.8.0]
    at com.baidu.hugegraph.HugeGraph.propertyKey(HugeGraph.java:416) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at com.baidu.hugegraph.traversal.optimize.TraversalUtil.convPredicateValue(TraversalUtil.java:489) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at com.baidu.hugegraph.traversal.optimize.TraversalUtil.convHasStep(TraversalUtil.java:480) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at com.baidu.hugegraph.traversal.optimize.TraversalUtil.convAllHasSteps(TraversalUtil.java:473) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at com.baidu.hugegraph.traversal.optimize.HugeGraphStepStrategy.apply(HugeGraphStepStrategy.java:49) ~[hugegraph-core-0.9.2.jar:0.9.2.0]
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversalStrategies.applyStrategies(DefaultTraversalStrategies.java:86) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.applyStrategies(DefaultTraversal.java:119) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:191) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.fill(IteratorUtils.java:62) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.list(IteratorUtils.java:85) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.asList(IteratorUtils.java:382) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:239) ~[gremlin-server-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36) ~[gremlin-core-3.2.5.jar:3.2.5]
    at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:320) ~[gremlin-groovy-3.2.5.jar:3.2.5]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
    at com.baidu.hugegraph.auth.HugeGraphAuthProxy$ContextTask.run(HugeGraphAuthProxy.java:282) [hugegraph-api-0.9.2.jar:0.37.0.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

请问在升级版本后，数据是可以无缝继续使用的吗？（比如使用 RocksDB 作为 backend）

目前来看，应该是丢失了元数据。

这种情况下，迁移图数据需要使用 HugeGraph-tools 导出图的元数据和原始数据然后导入到新的 HugeGraph 实例中吗？

Steps to reproduce the problem 复现步骤

  1. {step 1}
  2. {step 2}
  3. {step 3}

Status of loaded data 数据状态

Vertex/Edge summary 数据量

Vertex/Edge example 数据示例

{type something here...}

Schema(VertexLabel, EdgeLabel, IndexLabel) 元数据结构

vertex: person properties: name, ...

edge: knows

Specifications of environment 环境信息

zhoney commented 5 years ago

@suesunss 

请问在升级版本后，数据是可以无缝继续使用的吗？（比如使用 RocksDB 作为 backend）

各个版本的hugegraph后端并不一定兼容，如果后端存储数据的格式出现变动的时候，会改变相应的后端的版本号。以RocksDB为例，在RocksDBStoreProvider.java中会修改版本号。

这种情况下，迁移图数据需要使用 HugeGraph-tools 导出图的元数据和原始数据然后导入到新的 HugeGraph 实例中吗？

是的，可以使用hugegraph-tools的backup和restore子命令，对旧版本的数据进行备份导出，然后再恢复到新版本的图数据库中即可。可以参考：Backup Resotre

javeme commented 4 years ago

相关问题 https://github.com/hugegraph/hugegraph-tools/issues/35

github-actions[bot] commented 2 years ago

Due to the lack of activity, the current issue is marked as stale and will be closed after 20 days, any update will remove the stale label