Currently, the architecture of the community version of HugeGraph is still in version 1.0. In the internal version 2.0 of HugeGraph, we have the following design goals:
Support trillion-scale data storage and tens of thousands of graph storages.
Support multi-active, high availability, dynamic scalability, and automated operation and maintenance.
Maximize read and write performance.
Based on these goals, we have designed a distributed architecture that supports graph data partitioning and multiple replicas, and separates storage from computation for flexible scaling.
In version 2.0, in addition to the hugegraph-server, we also introduce two additional components: hugegraph-pd and hugegraph-store
The responsibilities of these two components are as follows:
hugegraph-pd: pd stands for placement driver, which can be simply understood as a meta server responsible for service discovery, partition information storage, and node scheduling.
hugegraph-store: as a new built-in storage backend, it uses RocksDB as the distributed backend storage foundation.
We will gradually merge the internal version 2.0 of HugeGraph into the community version. Therefore, the first step is to integrate the hugegraph-pd and hugegraph-store modules into this repository.
Tasks
Introduce PD and Store on the pd-store branch.
Currently, all tasks should be carried out on the `pd-store` branch:
- Adjust the project structure of this repository to include three sub-modules: hugegraph-server, hugegraph-pd, hugegraph-store at the root level.
- [x] https://github.com/apache/incubator-hugegraph/pull/2266
- Merge the internal version of pd into hugegraph.
- [x] https://github.com/apache/incubator-hugegraph/pull/2270
- [x] https://github.com/apache/incubator-hugegraph/pull/2273
- Merge the internal version of store into hugegraph.
- [x] https://github.com/apache/incubator-hugegraph/pull/2272
- Adjust the pom structure in order to directly build from root.
- [x] https://github.com/apache/incubator-hugegraph/pull/2275
- [x] https://github.com/apache/incubator-hugegraph/pull/2289
- Configuration adjustments (including merging dependencies, unifying version numbers, Maven compilation adjustments, etc.).
- [x] https://github.com/apache/incubator-hugegraph/pull/2288
- [x] https://github.com/apache/incubator-hugegraph/pull/2305
- [x] https://github.com/apache/incubator-hugegraph/pull/2321
- Perform code adaptation tasks (hugegraph-server needs to introduce modules to connect with hugegraph-pd and hugegraph-store, and make adjustments to other modules, etc.).
- [x] https://github.com/apache/incubator-hugegraph/pull/2301
- [x] https://github.com/apache/incubator-hugegraph/pull/2319
- Synchronization from the `master` branch.
- [x] https://github.com/apache/incubator-hugegraph/pull/2345
- [x] https://github.com/apache/incubator-hugegraph/pull/2414
Project structure adjustments on the master branch.
After https://github.com/apache/incubator-hugegraph/pull/2301, it should be possible to perform simple CRUD operations on HugeGraph on the `pd-store` branch. In the next phase, the `pd-store` branch will need to be merged into the `master` branch with an appropriate granularity.
- (master) project structure adjustments and CI configurations
- [x] https://github.com/apache/incubator-hugegraph/pull/2338
- [x] https://github.com/apache/incubator-hugegraph/pull/2382
We will reorganize commit messages and merge changes ahead of the pd-store branch into the master branch:
Feature Description (功能描述)
Relative Issues
#435 #828 #1218 #1398 #1517 #1581 #1609 #1760 #1925 #1968 #1979Background
Currently, the architecture of the community version of HugeGraph is still in version 1.0. In the internal version 2.0 of HugeGraph, we have the following design goals:
Based on these goals, we have designed a distributed architecture that supports graph data partitioning and multiple replicas, and separates storage from computation for flexible scaling.
In version 2.0, in addition to the hugegraph-server, we also introduce two additional components: hugegraph-pd and hugegraph-store
The responsibilities of these two components are as follows:
We will gradually merge the internal version 2.0 of HugeGraph into the community version. Therefore, the first step is to integrate the hugegraph-pd and hugegraph-store modules into this repository.
Tasks
Introduce PD and Store on the pd-store branch.
Currently, all tasks should be carried out on the `pd-store` branch: - Adjust the project structure of this repository to include three sub-modules: hugegraph-server, hugegraph-pd, hugegraph-store at the root level. - [x] https://github.com/apache/incubator-hugegraph/pull/2266 - Merge the internal version of pd into hugegraph. - [x] https://github.com/apache/incubator-hugegraph/pull/2270 - [x] https://github.com/apache/incubator-hugegraph/pull/2273 - Merge the internal version of store into hugegraph. - [x] https://github.com/apache/incubator-hugegraph/pull/2272 - Adjust the pom structure in order to directly build from root. - [x] https://github.com/apache/incubator-hugegraph/pull/2275 - [x] https://github.com/apache/incubator-hugegraph/pull/2289 - Configuration adjustments (including merging dependencies, unifying version numbers, Maven compilation adjustments, etc.). - [x] https://github.com/apache/incubator-hugegraph/pull/2288 - [x] https://github.com/apache/incubator-hugegraph/pull/2305 - [x] https://github.com/apache/incubator-hugegraph/pull/2321 - Perform code adaptation tasks (hugegraph-server needs to introduce modules to connect with hugegraph-pd and hugegraph-store, and make adjustments to other modules, etc.). - [x] https://github.com/apache/incubator-hugegraph/pull/2301 - [x] https://github.com/apache/incubator-hugegraph/pull/2319 - Synchronization from the `master` branch. - [x] https://github.com/apache/incubator-hugegraph/pull/2345 - [x] https://github.com/apache/incubator-hugegraph/pull/2414Project structure adjustments on the master branch.
After https://github.com/apache/incubator-hugegraph/pull/2301, it should be possible to perform simple CRUD operations on HugeGraph on the `pd-store` branch. In the next phase, the `pd-store` branch will need to be merged into the `master` branch with an appropriate granularity. - (master) project structure adjustments and CI configurations - [x] https://github.com/apache/incubator-hugegraph/pull/2338 - [x] https://github.com/apache/incubator-hugegraph/pull/2382We will reorganize commit messages and merge changes ahead of the pd-store branch into the master branch:
pd-grpc
,pd-common
,pd-client
(~8k loc)pd-core
(~9k loc)pd-service
(~11k loc)[x]
pd-dist
,pd-test
(~1k loc)store-grpc
,store-common
,store-client
(~13k loc)store-rocksdb
(~4k loc)store-core
(~14k loc)store-node
(~11k loc)[x]
store-dist
,store-test
,store-cli
(~3k loc)server-hstore
(~3k loc)server-core
(~12k loc)Project Structure