Sub-project 1: build a command line tool to install/start/stop a single node HPCC system

[larryxiao]

Sub-project 1: build a command line tool to install/start/stop a single node HPCC system Assume: both HPCC build and HPCC configuration file has been created on the node.

• [ ] Deploy/Redeploy HPCC system on the node and log the events/results;
  • [ ] deploy:pkg_name_with_full_path
  • [ ] optional_configuration_file_with_full_path

in-house> install-cluster.sh [-h|--help] [-k|--newkey] [-n|--concurrent <number>] <Platform Package>"
new> install-manager.sh <deploy/redeploy/undeploy> <package file path> 

• [ ] Start/stop the HPCC system and log the events/results;
  • [ ] Start or Stop Single Components

in-house> hpcc-run.sh [hpcc-init | dafilesrv] [-c component] <cmd>

• [ ] Monitor/log system status.

in-house> hpcc-push.sh (public key authentication)
new> scppass.sh (password authentication)

https://github.com/larryxiao/HPCC-Platform/tree/HPCC-9736/HPCC-9736

[larryxiao]

This command line tool runs in two modes, server mode and client mode. In the client mode, the program collects user’s input in command line and forwards to the input to the program in the server mode. In the server mode, the program does the work as the client asks.

In the rest of the sub-projects, the server code will be converted into a deployment agent which runs on every box of an HPCC system. The client code will be converted into an ESP web service with GUI.

An ESP server (which is a web server) with that ESP web service runs as a deployment control center which controls the deployments/start/stop/monitoring for multiple HPCC systems. Which is out of the HPCC system. It is a tool for installing HPCC systems.

Commands

deploy, undeploy, redeploy(undeploy and deploy), start, stop, restart(stop and start)

Example:

   deploy_manager -c -host xxx.xxx.xxx.xxx –p port_number -command "deploy:pkg_name_with_full_path:optional_configuration_file_with_full_path "
   deploy_manager -c -host xxx.xxx.xxx.xxx –p port_number -command "start:optional_hpcc_component_name"

[larryxiao]

Reference

• Client Tools | HPCC Systems
  • This manual contains documentation for the set of Client Tools for use with the HPCC Systems platform. These tools include: ECLPlus, ECL Compiler, DFUPlus, and the ECL Command Line Interface.
• Using Configuration Manager | HPCC Systems
  • This document covers configuring an HPCC environment using the Configuration Manager, including Advanced View.

[larryxiao]

• Work done:
  • For the client/server communication, I used ISocket, and it currently sends only command to execute to server.
  • For the start/stop/restart, I used cluster_script.py in combination of manager_script.sh to manage system.
  • For deploy/undeploy/redeploy, I use install-cluster.sh to deploy.
• Work todo:
  • For the client/server communication, I think there are higher level encapsulations of passing parameters in HPCC, I'll explore and use it. If not, I might use RPC.
  • For the start/stop/restart, I just found out about hpcc-run.sh, I'll use it instead.
  • For deploy/undeploy/redeploy, I'll implement undeploy/redeploy based on install-cluster.sh.
  • Design how to send the response back to client, may need to retrieve log files.

[larryxiao]

• RPC
  • google/protobuf

[larryxiao]

• deploy manager responsibility
  • provide interactions during the installation: progress checking, error handling
  • [ ] check OS type and call dpkg or rpm directly from the C++ code
  • [ ] handle file transfer (build file, configuration file, etc)

[wangkx]

To handle file transfer, you may add a C++ function to call the scppass.sh you sent me today.