alibaba / GraphScope

🔨 🍇 💻 🚀 GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | 一站式图计算系统
https://graphscope.io
Apache License 2.0
3.28k stars 442 forks source link

Reorganize the scripts for GIE launching #518

Closed yecol closed 3 years ago

yecol commented 3 years ago

It seems in the current version, the scripts for GIE are not well organized. I suggest we reorg the files in /interactive_engine/distribution and /interactive_engine/deploy

Currently, the file tree of distribution:

distribution
├── pom.xml 
└── src
    ├── assembly
    │   └── assembly.xml
    ├── bin 
    │   ├── admin_tools.sh
    │   ├── coordinator.sh
    │   ├── frontend.sh
    │   ├── ingestor.sh
    │   ├── load_tool.sh
    │   ├── max_node.sh
    │   ├── maxgraph-env
    │   ├── maxgraph.sh
    │   ├── startall.sh
    │   └── store.sh
    └── config
        ├── config.template
        ├── coordinator.config
        ├── frontend0.config
        ├── ingestor0.config
        ├── ingestor1.config
        ├── log4rs.yml
        ├── logback.xml
        ├── logback_debug.xml
        ├── sample.config
        ├── store0.config
        └── store1.config

problems:

For interactive_engine/deploy folder, currently the files are organized like this:

deploy
├── docker
│   └── dockerfile
│       ├── coordinator-entrypoint.sh
│       ├── coordinator.application.properties
│       ├── create_maxgraph_instance.sh
│       ├── data_loader.sh
│       ├── delete_maxgraph_instance.sh
│       ├── executor-entrypoint.sh
│       ├── executor.vineyard.properties
│       ├── expose_gremlin_server.sh
│       ├── frontend-entrypoint.sh
│       ├── frontend.vineyard.properties
│       ├── func.sh
│       ├── kill_process.sh
│       ├── load_data.sh
│       ├── load_modern_graph.sh
│       ├── manager-entrypoint.sh
│       ├── maven.settings.xml
│       ├── pod.yaml
│       └── set_config.sh

Similar to the files in distribution, most of the files are simple wrappers for a single-line Java command. In addition, it seems many files are for the same functions as those in distribution. (As the filename implies, starting coordinator/front-end and exposing an entrypoint.) It would be better if the scripts to start the coordinator (and for others) can be unified, whether to start it in a docker container/ or in a physical machine (or local). We may not need a set of docker-specific scripts and the giectl is sufficient.


I suggest we reorg them as:

interactive_engine
└── bin
    ├── giectl # with sub-commands, like `start-coordinator`, `start-frontend`, `stop-all` ....
    ├──data_loader.sh # data_loader can be separate if in need.
└── config
    ├── log4rs.yml 
    ├── logback.xml
    ├── frontend.vineyard.properties
    ├── executor.vineyard.properties
    ├── coordinator.application.properties
yecol commented 3 years ago

@shirly121 @zhengpingq, how do you think, could you give some comments? If the proposal is workable, I think zedong or weibin can help for the implementation.

yecol commented 3 years ago

And the scripts under interactive_engine/src/instance-manager/script/* can also unified:

├── bin
│   ├── start.sh
│   └── stop.sh
├── config
│   ├── application.properties
│   ├── coordinator.local.application.properties.tpl
│   ├── executor.local.vineyard.properties.tpl
│   └── frontend.local.vineyard.properties.tpl
├── script
│   ├── close_local_instance.sh
│   ├── common.sh
│   ├── create_local_instance.sh
│   ├── start_local_coordinator.sh
│   ├── start_local_executor.sh
│   └── start_local_frontend.sh
tianliplus commented 3 years ago

Better seperate gie and v2, and put v2 related scripts under interactive_engine/distribution.