Designing new command line interface (hilbert)

[porst17]

Please use this issue to create and discuss a specification for the new unified command line interface.

Never reference anything below this line in the comments. It may change at any time.

---

Sub-issues to solve:

• [x] "Configuration Design (super issue)": hilbert/hilbert-cli#13
• [x] General Hilbert Architecture (see corresponding part of the Google Document)
• [x] Hilbert-CLI-Server Design (works with the general YAML config, communicates with CMS)
• [x] Hilbert-CLI-Station Design (e.g. https://github.com/hilbert/hilbert-docker-images/issues/28#issuecomment-257195798, written in BASH, sourced local config)

[malex984]

DRAFT (v.0.5):

General design ideas for hilbert CLI (what should it be able to do):

• find/read host's configuration (if corresponding configuration is available and necessary for the action)
• run action locally on the host (depending on absent/found/read host's configuration)
• find/read configuration for remote system from a cache (if corresponding configuration is available and necessary for the action)
• create new (or refresh) configuration for a remote system (in a cache) using a set of templates
• run action remotely on all/any of remote systems (e.g. via SSH or any other supported access method).
  • For example: deploy configs and hilbert tool to remote Linux systems using ssh-keys retrieved via rsync from the source specified in station config file (e.g. via SFTP URL)
• handle POSIX-like remote system (e.g. Windows) uniformly via the same CLI

Notes:

• most local commands require to read the host's configuration
• all remote or remote-like commands require access to the cache for reading the remote system's config
• supported access methods: ssh and docker-machine for virtual testing environments
• no action may block execution indefinitely. For example: no user input should be required by default
• possible return codes should be fixed by specification
• actions that change configs (locally or on remote system) have to do that in the atomic way + keep a backup of previous configs
• it may be possible to manage any topology of system connections with parents and children.
• some commands should not run simultaneously - we will require a locking mechanism for them

Some global options:

• --debug to output human readable action log
• --trace to output each performed check and command
• --quiet to suppress any aux. output
• --version get the tool's version
• --help should be available globally and for each local command
• --config_dir=<PATH> overwrite for the configuration location (also ENV. Var. HILBERT_CONFIG_DIR)
• --configs_cache=<PATH> overwrite for the location of configs. cache (also ENV. Var. HILBERT_CONFIGS_CACHE)

Note about precedence rules for all possible config-related specifications:

• location of the local host's config (CONFIG_DIR):
  1. by default in ~/.config/hilbert or
  2. alongside with hilbert or
  3. may be overwritten by $HILBERT_CONFIG_DIR or
  4. most definitely via --config_dir=<PATH>
• location of configs. cache for remote systems is:
  1. in $CONFIG_DIR/STATIONS
  2. specified inside station config (e.g. field CONFIGS_CACHE)
  3. may be overwritten by $HILBERT_CONFIGS_CACHE or
  4. most definitely via --configs_cache=<PATH>

Main top-level drivers:

• local COMMAND [args]: run command locally on the host system. This is the default/assumed driver.
• @remote REMOTE_STATION [opts] COMMAND [args]: run hilbert's command on specified remote station:
  1. read corresponding access configuration
  2. locate and run hilbert on remote system: $SSH $REMOTE_CONFIG_DIR/hilbert $opts $COMMAND $args

NOTE: remote commands may differ for different remote stations, use hilbert remote REMOTE_STATION local --cmds to get them for a running prepared remote system REMOTE_STATION. Only basic local commands are available by default.

Basic local commands:

• --cmds list available commands
• shell [args] - run a single shell command specified by $args (no default working directory)
• shutdown [args] - run sudo -n -P /sbin/shutdown $args
• poweroff [args] - run sudo -n -P /sbin/poweroff $args
• reboot [args] - run sudo -n -P /sbin/reboot $args
• ptmx [args] - run ptmx work-around (maybe even --background)

Possible local low-level* commands (e.g. on Linux systems with Docker Engine):

• docker - run docker with corresponding args (if docker is accessible).
• launch - run docker-compose with corresponding args within necessary location after reading all station configs (if docker-compose is accessible). See current templates/luncher.sh

Possible local high-level commands (e.g. on Linux systems with Docker Engine + hilbert):

Note: the following actions should not run in parallel - they will require a locking mechanism

• prepare - run preparation actions (e.g. cleanup & pre-download). See current templates/prepare.sh
• start_apps - run services according to station config. See current templates/default.sh
• switch_top - switch the current GUI app to a different one. See current templates/topswitch.sh
• finish_apps - stop/kill/remove services according to station config. See current templates/finishall.sh

Standard local management-related commands:

• cfg_create REMOTE_STATION [args] - create new child configuration inside the config. cache (--templates=PATH, or ENV: HILBERT_TEMPLATES=<PATH>). See current init.sh
• cfg_refresh REMOTE_STATION [args] - refresh configuration due to template changes (--templates=PATH, or ENV: HILBERT_TEMPLATES=<PATH>). See current refresh.sh
• list_children, list known remote systems in specified format (e.g. --output=json ). See current dockapp_dashboard/server/scripts/list_stations.sh
• sync SOURCE TARGET - sync this station configs + children config cache (--source= UPSTREAM). See current sync.sh. For example: hilbert sync <CMS_SFTP_URL> <CONFIGS_CACHE>.
• forall [opts] COMMAND [args]: run hilbert @remote $REMOTE_STATION $opts $COMMAND $args for each known REMOTE_STATION (according to config. cache). See current forall.sh

Basic management-related (do something low-level w.r.t. remote system) commands:

NOTE: at least remote system access configuration is required to be in the local config. cache

• @shell REMOTE_STATION args - run a single shell command on remote system (via specified access method). For example: $SSH $REMOTE_STATION_ID $args
• @poweron REMOTE_STATION [args] - trigger power-on (e.g. via WOL). Non-blocking. Use option: --wait to wait for complete system startup. For example: wakeonlan $REMOTE_MAC. See current start.sh
• @status REMOTE_STATION [args] - get current status of remote system: e.g. OFF, inaccessible, virgin, ready, running, stopped... See current status.sh
• @deploy REMOTE_STATION [args] - deploy corresponding configs and hilbert tool to the remote system. See current deploy.sh. (maybe hilbert sync <CONFIGS_CACHE>/$REMOTE_STATION $REMOTE_STATION:$CONFIG_DIR?)
• @lasttop REMOTE_STATION [args] - get the latest chosen GUI app. from remote station. See current lastapp.sh

Standard management-related (do something high-level w.r.t. remote system) commands:

NOTE: complete remote system configuration is required to be in the local config. cache + Docker Engine should be available on the remote system + Hilbert.

• @start REMOTE_STATION [args] - ensure that remote station is UP, its configs are up-to-date and running all the prescribed services. See current dockapp_dashboard/server/scripts/start_station.sh:
  1. hilbert @poweron $REMOTE_STATION --wait
  2. hilbert @deploy $REMOTE_STATION
  3. hilbert @remote $REMOTE_STATION local prepare
  4. hilbert @remote $REMOTE_STATION local start_apps
  5. hilbert @lasttop $REMOTE_STATION
• @top_switch REMOTE_STATION args - gracefully stop the current GUI app. and start a new one (specified via arguments). See current dockapp_dashboard/server/scripts/appchange_station.sh:
  1. hilbert @remote $REMOTE_STATION local switch_top $args
  2. hilbert @lasttop $REMOTE_STATION
• @stop REMOTE_STATION [args] - gracefully stop all the prescribed services and shut it down. See current dockapp_dashboard/server/scripts/stop_station.sh:
  1. hilbert @remote $REMOTE_STATION local finish_apps
  2. hilbert @remote $REMOTE_STATION local poweroff

[elondaits]

Some comments:

• In high level languages (e.g. Python, Ruby, node.js etc.) there are very good libraries for handling command line option parsing. The case where parsing is dependent on part of the command (i.e. depending on the COMMAND you have different valid args) is one of the most complicated they handle, and having TWO instances in which the parsing branches (local/remote and then COMMAND) will probably be too much. My recommendation is simply to design the CLI taking into consideration the library that will be used. It will probably expose the unexpected complications one is adding to the system.
• ... The local case could simply be covered through a reserved station name such as local.
• But then again I'm not too sure about having a tool that does commands locally and remotely, because they seem to be VERY different use cases and just like it happened to me when I ran make and shut down the main server you probably don't want to run most commands locally. I don't yet know what the COMMANDs are, so it's just a feeling.

[malex984]

@elondaits, i am almost ready with my draft. Please wait a bit.

[elondaits]

Let me know... but consider I get an email message when new things are posted and none when you edit your previous messages.

[malex984]

@elondaits Ok, i am mostly done.

I suggest we start discussing the use cases and usage scenarios.

[malex984]

To make implementation in bash simpler the following looks promising:

The global subcommand utility: http://people.tuebingen.mpg.de/maan/gsu/

gsu is a small library of bash functions intended to ease the task of writing and documenting large shell scripts with multiple subcommands, each providing different functionality. gsu is known to work on Linux, FreeBSD, NetBSD and MacOS.

[elondaits]

General coments

• The dashboard only uses "stop", "start" and "top_switch". Why do we need so many more things? We probably need more things, of course... specially for setup/init and perhaps diagnosing / fixing things.. but they should be really well justified. We need the smallest interface possible, because that's the easiest one to maintain.
• The high level commands don't need to be expressable in terms of low level commands... it could be solved in a function the user can't reproduce by himself.
• If the "real" configuration (in the CMS) is outside of the scope and reach of our system we should stop saying our configuration is a cache. It complicates explanations a lot. We should only consider what we have control over and provide an external mechanism for importing the remote config.
• The word "cache" shouldn't be used either to referr to the copy of the configuration that is on the host (as opossed to the station). A "cache" is never upstream and is an optional thing added for performance. If it's used as a cfg, it's a cfg.
• I don't think the same script should be used for the host, where there's a user taking high-level actions and the stations where much smaller, specific, lower level scripts are run. They're two very different things with different security concerns. We can also have any high level language in the host (which can have more strict requirements in terms of OS and dependencies), which is not as easy in the stations.
• Regarding the command "forall". If my dashboard server has the responsibility of executing several hilbert commands in parallel asynchronously I don't think it's a good idea to have another way of doing the same differently, with possibly different semantics, side effects, etc. We duplicate efforts.

Security

• The document currently says

For example: deploy configs and hilbert tool to remote Linux systems using ssh-keys retrieved via rsync from the source specified in station config file (e.g. via SFTP URL)

If I understand this right this suggests placing the ssh key to the remote station somewhere it can be fetched through ftp so the system can then issue ssh commands. If that's the idea... This is a crazy backdoor! It's possible that I misunderstood but then the text should be clarified because that's what I read now.

• Regarding the command

shell [args] - run a single shell command specified by $args (no default working directory)

I don't think it's a good idea to have a generic backdoor. Our system should, if possible, only allow remote execution of "safe" commands. ssh already does this, as well.

• Same for the "docker" and "launch" commands. The user can already run docker himself... unless totally necessary our system should only support a well defined workflow. Run "on rails".
• Regarding args in some commands as in

shutdown [args] - run sudo -n -P /sbin/shutdown $args

allowed args should be explicitly documented, validated and never ever sent directly from the hilbert invocation to the sudoed command. Otherwise one could do something like

shutdown "; rm -rf /*"

(or some smarter / trickier way of escaping things in bash) and piggyback a command. All user input is evil and should be treated as such.

• The proposed envvar $CONFIG_DIR is too general and might clash with other systems that use it for something else. We should use something like $HILBERTPATH, $HILBERTCFG or similar.

Semantics

• We already discussed many times that the dashboard will not use the start command as it is anymore, but instead first start the station, wait for MK to report it as up, and then deploy / prepare / start apps. So I'd need two high level commands: one to start the station and one to start the app.
• How does the --wait in the start command work? If it is through polling then it's a very bad idea because it'll behave very badly in terms of working set and memory cache when running in parallel in many processes. It's much more effective to do the polling on a single thread for all stations like I do now in the dashboard and only wake their individual threads when MK indicates a change. If the dashboard is already doing this effort then we shouldn't have a --wait in the CLI.

Syntax / aesthetics

• Other systems that work similarly use @ to identify target systems instead of commands. This is reasonable because @ is read as "at" in english and already used to separate user@host.
• Re the "prepare" command... what it does is not clear to me. "prepare" is a transitive verb and needs an indication of WHAT you're preparing to do. According to the description it does two different things, "cleanup" and "pre downloading" (pre-downloading is still downloading... pre fetching perhaps?) Perhaps it can be split, perhaps it can be automatically performed when it's needed, or at least renamed to something more familiar.
• Taking as an example the description of the command "list_children":

list_children, list known remote systems...

when the description of the command says something different than the name it's a good indication that the name should be changed (list_remote_systems in this case, for instance)

• switch_top, top_switch and topswitch appear in the document. We should only use one form. Also since there's a start_apps command I'd change it to switch_app (or start_top). If it's the same thing then it should always be called by the same name.

Other

• In high level languages (e.g. Python, Ruby, node.js etc.) there are very good libraries for handling command line option parsing. The case where parsing is dependent on part of the command (i.e. depending on the COMMAND you have different valid args) is one of the most complicated they handle, and having TWO instances in which the parsing branches (local/remote and then COMMAND) will probably be too much. My recommendation is simply to design the CLI taking into consideration the library that will be used. It will probably expose the unexpected complications one is adding to the system.

Not covered

• What are the semantics of each command?
  • Preconditions:
  • In what state should the system / remote station be before the command is issued for it to succeed?
  • In what way can the user check this preconditions are valid? Does the system provide a way? Is something else (e.g. MKLivestatus) needed?
  • Postconditions:
  • What things are true once if command finished succesfuly?
  • What things are true once if command failed?
  • How can you tell these cases apart?
• Other than stop/start/switch and list the other big use cases I can think of are related to adding new stations to the system or removing them. What commands are issued for that? Who does it?
• How are remote stations updated if the scripts change?

[malex984]

Minimal hilbert interface: https://github.com/malex984/dockapp/blob/devel4_docs/mng/README.md#minimal-hilbert-interfaces

The purpose of actions that replace current scripts is the same as that of the superseded scripts. Same with semantics for now...

[malex984]

@elondaits Could you please move your comments to the shared Google Document and attach them to relevant places in the draft design.

Let us continue the discussion there.

[elondaits]

I added them. Some were general comments that might be relevant to several parts or the document as a whole... so consider what they say and not only the place where I added them.

[porst17]

Has to be moved to https://github.com/hilbert/hilbert-cli/issues

[malex984]

I updates the google document:

• Configuration part
• Hilbert-CLI descibes 3 tools: hilbert-cli-config, hilbert-cli-server (both: Python) and hilbert-cli-station (bash)
• What follows "Current State (TO BE OBSOLETED)" (page 13) is the description of the current scripts - just for reference.

[malex984]

See also "Configuration Design (super issue)": hilbert/hilbert-cli#13

[malex984]

More specifics on Hilbert-CLI-Station local configuration:

1. All key/value pairs listed in Station::settings should added to some ${HOME}/.config/hilbert-cli-station/settings.cfg in some plain format, e.g. (%s=%s) % (key, value).
2. Additionally there may be other settings defined elsewhere in general YAML config for this station: e.g. default_application & services & applications etc...
3. ${HOME}/.config/hilbert-cli-station/settings.cfg is to be used by Hilbert-CLI-Station tool itself. The settings may influence the general operation: (e.g. autostart & autostart_delay & default_application / services etc.).
4. The settings from ${HOME}/.config/hilbert-cli-station/settings.cfg are to be forwarded into containers created by Hilbert-CLI-Station:
   • ${HOME}/.config/hilbert-cli-station/ may be mounted (read-only) by each started container
   • or via environment variables (e.g. via env-file specification)

Note: prior to starting some services/applications Hilbert-CLI-Station may be required to run some special shell code on the host and set some environment variables before passing them into containers.

For instance: auto-detection of variables for using some currently running native services or device configurations: PULSE_SOCKET/PULSE_COOKIE (pulseaudion), DISPLAY/XAUTHORITY (x11), DOCKER_SOCKET (Docker Engine) etc.

Note: services started by Hilbert-CLI-Station (e.g. x11 / qrhandler) are to follow pre-defined schema to enable local access (e.g. share sockets via /tmp/ or by serving on localhost network address). They will also have to create individual temporary (single session only) config-files (e.g. in /tmp/) specifying corresponding access settings (e.g. with DISPLAY=... & XAUTHORITY=...).

Of course Hilbert-CLI-Station may provide some auto-detection for some standard services e.g. X11, pulseaudio, docker, but what about general case?

IMO in general Hilbert-CLI-Station should be able to run some custom script on the host system (which may create something in /tmp/) before creating a docker container with a service or an application.

NOTE: It has to be attached to service/application definition in general YAML configuration: Suggestion: Service::auto_detections.

[porst17]

This issue is a super-issue again that needs to be split up. I already modified the issue description. @malex984 Please prepare the individual issues and add them to the list. A summary of the results should be added and updated on a regular basis, e.g. each time a sub-issue is closed (you can add commit-like comments to this issue and reference them in the description to make clear with state is reflected in the description).

[malex984]

Note: this one is in hilbert-docker-images since it was inherited by default from dockapp repository.

@porst17 Q: What about using hilbert/hilbert-cli#14 as a super issue instead of this one?

[malex984]

Please let me elaborate on the operation of Hilbert-CLI-Server:

According to https://github.com/hilbert/hilbert-docker-images/issues/28#issuecomment-257195798 ${SPECIAL_USERS_HOME}/.config/hilbert-cli-station will contain all local settings and resources (e.g. docker-compose.yml generated for this station) required for running Hilbert-CLI-Station on the host. In particular:

• the local settings will be exposed as environment variables
• additionally services may perform some auto_detection, which may also result in further environment variables
• any service/application may mount locations (on the host system), possibly specified by environment variables (e.g. $PWD, $HOME/.ssh or /tmp/.ssh, $HILBERT_SERVER_CONFIG_PATH, $HILBERT_OMD_PATH, $HILBERT_REGISTRY_DATA_PATH ) as data volumes (read-only or read-write according to corresponding definition in the docker-compose.yml).

[malex984]

Let me extend https://github.com/hilbert/hilbert-docker-images/issues/28#issuecomment-257339008 wrt Server specifics:

1. Management Backend container running Hilbert-CLI-Server on the Server only may accept a local setting like HILBERT_SERVER_SYNC_URL (which is a part of Station::settings), which would enable Hilbert-CLI-Server to reload (sync?) the configuration-related files and docker-image tarballs from some external location using the predefined syncing mechanism (e.g. rsync over sFTP or SSH).
2. The external resources should be persistently stored on the host system. For that, some predefined location will be mounted (read-write) into the Management Backend container when it is created. I imagine Server's Station::settings may explicitly specify that location, e.g. via a variable like HILBERT_SERVER_CONFIG_PATH, which may default to ${SPECIAL_USERS_HOME}/.config/hilbert-cli-server on the host system.

[porst17]

I missed that this issue is still listed under hilbert-docker-images. Feel free to move relevant stuff over to hilbert/hilbert-cli#14 and then close this issue.

[malex984]

For reference: the use of auto-detection for using anything located on a host or running there

[malex984]

For reference: command aliases

[malex984]

CLI server/client tools were initially implemented within https://github.com/hilbert/hilbert-cli/pull/34

Corresponding design is in the "Management Back-end" Document.

Current CLI: https://gist.github.com/malex984/59d2b62c5098eb1005100f9249719505

[malex984]

Further implementation will be due to https://github.com/hilbert/hilbert-cli/issues/14