riboseinc / nereon-syntax

Nereon configuration syntax (in RSD)
0 stars 1 forks source link

The examples of NOS configurations #8

Open jjr840430 opened 6 years ago

jjr840430 commented 6 years ago

@ronaldtse @drystone

I would like to discuss about NOS and NOC syntax using the example. Here is the NOS configuration example what rvc is using:

config_option "config_file" {
    type = "config"
    cmdline "switch" {
        short = "f"
    }
    cmdline "description" {
        short = "config file"
        long = "set configuration file"
    }
    env = "RVD_CONFIG_FILE"
    default = "/opt/rvc/etc/rvd.conf"
}

config_option "go_daemon" {
    type = "bool"
    cmdline "switch" {
        short = "D"
    }
    cmdline "description" {
        long = "daemonize"
    }
}

config_option "check_config" {
    type = "bool"
    cmdline "switch" {
        short = "c"
    }
    cmdline "description" {
        long = "only check config and exit"
    }
}

config_option "print_version" {
    type = "bool"
    cmdline "switch" {
        short = "v"
    }
    cmdline "description" {
        long = "print version"
    }
}

config_option "helper" {
    type = "helper"
    cmdline "switch" {
        short = "h"
    }
    cmdline "description" {
        long = "print help message"
    }
}

config_option "openvpn_bin" {
    type = "string"
    cmdline "switch" {
        short = "o"
    }
    cmdline "description" {
        short = "OpenVPN binary path"
        long = "specify OpenVPN binary path"
    }
    env = "OPENVPN_BIN_PATH"
    config = "openvpn.path"
    default = "/opt/openvpn/sbin/openvpn"
}

config_option "openvpn_root_check" {
    type = "bool"
    cmdline "switch" {
        short = "r"
    }
    cmdline "description" {
        long = "enable root check for OpenVPN binary"
    }
    config = "openvpn.enable_root_check"
    default = true
}

config_option "openvpn_updown_scripts" {
    type = "bool"
    cmdline "switch" {
        short = "s"
    }
    cmdline "description" {
        long = "enable up/down scripts for OpenVPN"
    }
    config = "openvpn.enable_updown_scripts"
    default = false
}

config_option "user_id" {
    type = "int"
    cmdline "switch" {
        short = "u"
    }
    cmdline "description" {
        short = "User ID"
        long = "specify User ID for RVC process"
    }
    env = "RVC_USER_ID"
    config = "global.user_id"
}

config_option "restrict_socket" {
    type = "bool"
    cmdline "switch" {
        short = "r"
    }
    cmdline "description" {
        long = "restrict access to communication socket"
    }
    config = "global.restrict_socket"
    default = true
}

config_option "log_directory" {
    type = "string"
    cmdline "switch" {
        short = "l"
    }
    cmdline "description" {
        short = "log directory path"
        long = "specify the log directory of RVD process"
    }
    env = "RVD_LOG_DIR"
    config = "global.log_directory"
    default = "/var/log/rvd"
}

config_option "vpn_config_paths" {
    type = "string"
    cmdline "switch" {
        short = "p"
    }
    cmdline "description" {
        short = "VPN config path"
        long = "specify the directory path of VPN configurations"
    }
    config = "global.vpn_config_paths"
    default = "/opt/rvc/etc/vpn.d"
}

Here is NOC configuration for rvc.

global {
    user_id = RVC_USER_ID
    restrict_socket = true
    log_directory = "/var/log/rvd"
    vpn_config_paths = "/opt/rvc/etc/vpn.d"
}

openvpn {
    sbin_path = "/opt/openvpn/sbin/openvpn"
    root_check = true
    enable_updown_scripts = false
}
drystone commented 6 years ago

Thanks for this, @jjr840430

NOS syntax is still up for discussion, please make comments/suggestions. As an initial translation, something like:

option config {
    short f
    long "config-file"
    usage "Specify configuration file"
    hint FILE
    env RVD_CONFIG_FILE
    default "/opt/rvc/etc/rvd.conf"
}

option help {
    short h
    long help
    usage "Print this help message"
    default_arg true
    default false
}

option version {
    short v
    long version
    usage "Print version information"
    default_arg true
    default false
}

option go_daemon {
    short D
    usage "Run in daemon mode"
    default_arg true
    default false
}

option check_config {
    short c
    usage "Check configuration and exit"
    default_arg true
    default false
}

option openvpn_bin {
    short o
    usage "Specify OpenVPN binary path"
    hint PATH
    config "openvpn.path"
    env OPENVPN_BIN_PATH
    default "/opt/openvpn/sbin/openvpn"
}

option openvpn_root_check {
    short r
    usage "Enable root check for OpenVPN binary"
    hint BOOL
    config "openvpn.enable_root_check"
    default_arg true
    default true
}

option openvpn_updown_scripts {
    short s
    usage "Enable up/down scripts for OpenVPN"
    hint BOOL
    config "openvpn.enable_updown_scripts"
    default_arg true
    default false
}

option user_id {
    short u
    usage "Specify User ID for RVC process"
    hint UID
    env RVC_USER_ID
    config "global.user_id"
    default "-1"
}

option restrict_socket {
    short r
    usages "Restrict access to communication socket"
    config "global.restrict_socket"
    hint BOOL
    default_arg true
    default true
}

option log_directory {
    short l
    usage "Specify the log directory of RVD process"
    hint DIR
    env RVD_LOG_DIR
    config "global.log_directory"
    default "/var/log/rvd"
}

option vpn_config_paths {
    short p
    usage "specify the directory path of VPN configurations"
    hint DIR
    config "global.vpn_config_paths"
    default "/opt/rvc/etc/vpn.d"
}

After processing each option generates exactly one value which is inserted into the configuration tree.

jjr840430 commented 6 years ago

@drystone Thanks for your sharing your structure. It seems your structure is better than one for libnereon.

The remaining is how to define sub-options.

I think we may take some example models for sub-options. I will post my thoughts shortly.

jjr840430 commented 6 years ago

For example, rvc has the following sub-options:

rvc edit <connection name> <auto-connect|pre-exec-cmd|profile> <value>

edit is main option, and auto-connect,pre-exec-cmd, profile is sub option for edit. Also edit should have at least one suboption of them.

Then we may think the following NOS syntax for edit option.

option edit {
    short e
        long edit
    usage "Edit RVC VPN connection"
        sub-option-required true
        sub-option auto-connect {
              long "auto-connect"
        }
        sub-option pre-exec-cmd {
              long "pre-exec-cmd"
        }
        sub-option profile {
               long "profile"
        }
}
drystone commented 6 years ago

Perhaps we could call edit a sub-command?

I think NOS could handle sub-commands but options without names may need to be handled outside of NOS. Or would this be acceptable?

rvc edit <--connection=NAME> <--auto-connect=VALUE|--pre-exec-cmd=VALUE|--profile=VALUE>

or

rvc edit <--connection=NAME> <--option=OPTION --value=VALUE>
jjr840430 commented 6 years ago

The command line format for rvc was suggested by @ronaldtse. I'm not sure whether all command line formats should be handled by NOS configuration.

I think we need to define command line syntax too which is supported by nereon.

@ronaldtse What do you think about this? I think it's not possible to support all command line formats by nereon. So we need to define command line formats too with NOS configuration syntax.

ronaldtse commented 6 years ago

I agree we should support subcommands. Let's first distinguish a "command" from an "option".

For example:

In the case of aws s3 ls <bucket-name> [options]

drystone commented 6 years ago

Commands are permitted more than one non-option argument:

ls dir1 dir2
cp file 1 file2 dir

These should be described in NOS so they can be inserted into the resulting NOC. Something like

args {
    config "args"
}

Subcommands could be described as:

# common options for <progname>
...
subcommand one {
    # common options for <progname> one
    ....
    subcommand a {
        # specific options for <progname> one a
        ....
    }
    subcommand b {
        ...
    }
}
subcommand two {
    ...
}
subcommand three {
    ...
    subcommand c {
        ....
    }
    subcommand d {
        ....
    }
}
jjr840430 commented 6 years ago

@drystone we don't need to separate command line option to command and option. it may be compromised when implementing several types of the command line by NOS.

we may think command line as the following syntax:

prog_name <option1> [args1] <suboption1> <> <options2> [args2] <suboption2> <>...

option1 and option2 is the item with statically defined names by the program. it has the following requirements.

suboptions is the item with statically defined names by the program, but it can't be used without option or other suboption. suboption has the same requirements as option.

args is the item which is defined with arbitrary data so it's not defined in NOS configuration.


From my thoughts, let's take some examples:

rvc edit vpn1 --auto-connect enable

edit is option, --auto-connect is suboption which may be used by only edit option, vpn1 and enable is args for edit and --auto-connect

cp /tmp/1 /tmp2

this example doesn't have any option and suboption. it has only args, /tmp/1 and /tmp/2

git checkout -b new_branch

checkout is option -b is suboption new_branch is args for -b

aws s3 ls <bucket-name> [options]

s3 is option ls is suboption of s3 bucket-name is args for ls.

jjr840430 commented 6 years ago

@ronaldtse @drystone Here is NOS configuration example for rvc. Since rvc doesn't have configuration file so I will post another example related to NOC configuration in the next comment.

suboption json_flag {
    switch "--json"
    type "boolean"
}

option connect {
    type "string"
    switch "connect"
    desc "connect to a VPN with given name (default:all)"
    hint "[all|connection name] [--json]"
    default "all"

    suboptions {
        option json_flag
    }
}

option disconnect {
    type "string"
    switch "disconnect"
    desc "disconnect VPN with given name (default:all)"
    hint "[all|connection name] [--json]"
    default "all"

    suboptions {
        option json_flag
    }
}

option reconnect {
    type "string"
    switch "reconnect"
    desc "reconnect VPN with given name (default:all)"
    hint "[all|connection name] [--json]"
    default "all"

    suboptions {
        option json_flag
    }
}

option status {
    type "string"
    switch "status"
    desc "get status of VPN connection with given name (default:all)"
    hint "[all|connection name] [--json]"
    default "all"

    suboptions {
        option json_flag
    }
}

option edit {
    type "string"
    switch "edit"
    desc "edit VPN connection with given name"
    hint "<connection name> <auto-connect|pre-exec-cmd|profile> <value>"

    suboptions {
        requires true
        type "mixed"

        option auto-connect {
            type "string"
            switch "auto-connect"
        }

        option pre-exec-cmd {
            type "string"
            switch "pre-exec-cmd"
        }

        option profile {
            type "string"
            switch "profile"
        }
    }
}

option remove {
    type "string"
    switch "remove"
    desc "remove VPN connection (sudo required)"
    hint "<connection name> [--force]"

    suboptions {
        option force_flag {
            type "boolean"
            switch "--force"
        }
    }
}

option import {
    type "boolean"
    switch "import"
    desc "import VPN connection (sudo required)"
    hint "<new-from-tblk|new-from-ovpn> <path>"

    suboptions {
        requires true
        type "single"

        option new_from_tblk {
            type "string"
            switch "--new-from-tblk"
        }

        option new_from_ovpn {
            type "string"
            switch "--new-from-ovpn"
        }
    }
}

option reload {
    type "boolean"
    switch "reload"
    desc "reload configuration (sudo required)"
}

option dns-override {
    type "boolean"
    switch "dns-override"
    desc "override DNS settings (sudo required)"
    hint "<enable <DNS serve IP list>|disable|status>"

    suboptions {
        requires true
        type "single"

        option enable {
            type "string"
            switch "--enable"
        }

        option disable {
            type "boolean"
            switch "--disable"
        }

        option status {
            type "boolean"
            switch "--status"
        }
    }
}

option script-security {
    type "boolean"
    switch "script-security"
    desc "enable/disable script security"
    hint "<enable|disable>"

    suboptions {
        requires true
        type "single"

        option enable {
            type "boolean"
            switch "--enable"
        }

        option disable {
            type "boolean"
            switch "--disable"
        }
    }
}

option version {
    type "boolean"
    switch "version"
    desc "print version"
}

option help {
    type "helper"
    switch "help"
    desc "print help message"
}
jjr840430 commented 6 years ago

Here is the description for NOS configuration of rvc. NOS configuration may have the following sections, suboption and option.

suboption: define commonly used suboptions in option. In the above example, json suboption is used in connect, disconnect, reconnect, status options. option: define standalone options

Each field in option or suboption section has the following purpose:

        switch "<switch1>|<switch2>"
  for example: `<-l|--verbose>`, `<config|-config>`
jjr840430 commented 6 years ago

Since NOS configuration is compiled with the main program, I suggest the following another section:

program {
        name "<program name>"
        version "<program version>"
        copyright "<copyright message>"
        homepage "<homepage URL>"
}

version, copyright and homepage may be used for command line to show version info.

jjr840430 commented 6 years ago

To integrate with NOC configuration, I suggest config section in NOS configuration:

config <config name> {
        type <configuration type>
        default <default value>
        env <environment variable>
        requires <true or false>
        subconfigs {
                  config <subconfig name> {
                          type <type of subconfig>
                          default <default value of subconfig>
                          env <environment variable>
                          requires <true or false>
                  }
        }
}

config name: it specifies the configuration name in NOS. it should be composed by only alphabetical characters, - and _. type: the type of configuration. it may have the following types: object: the configuration item has sub configurations string: the configuration has string type integer: the configuration has integer type array: the configuration has array type boolean: the configuration has boolean type float: the configuration has float type default: default value of the configuration requires: specify whether that configuration should be given in NOC configuration subconfigs: defines sub configurations belong into that configuration

If NOS has command line option to override the NOC configuration item, then it may be specified in option as the following:

option <option name> {
          ...
          config "<config path>"
          ...
}

config path means the path of configuration item. If configuration is subconfig, then config path should be specified by concating config section names using ..

Here is the example:

config global {
         ...
         subconfigs {
                 config log_directory {
                          ...
                 }
         }
         ...
}

option log_dir {
          ...
          config "global.log_directory"
          ...
}
drystone commented 6 years ago

I tend to agree with @ronaldtse that there is a semantic difference between options, subcommands and arguments. Options traditionally have the form -o or --option. Subcommands, if any will be the first n non-option arguments and the remaining non-option arguments are simply arguments for the command.

These could all be treated the same but I think the NOS language would be more intuitive if it can describe/model these concepts more naturally.

As I said earlier, there is no concept of type in NOC syntax (nor in command line arguments). All leaf values are strings and it is the responsibility of the program to convert these into meaningful values and to error out if the configuration values can't be parsed. Therefore I don't think the type fields are necessary other than to describe whether an option is optional an/or whether it requires an argument - and I think we can find a better language for this.

I'm off now but look forward to further discussion.

jjr840430 commented 6 years ago

@drystone Thanks for your comment. I also think that @ronaldtse 's opinion is meaningful. but sometimes it may be confused when determine which command line argument is specified as subcommand or option. if we may define command line arguments with static names as option or suboption, then it may reduce that confusing.

Also type in NOS and NOC configuration is for checking whether configuration item has the correct format for built-in types such as integer, string, boolean etc. It's possible to check them on the main program side but I think it may reduce the coding lines if nereon library checks the type of each arguments while initializing. We may override type-checking function in the main program side for specific data types such as date, IP address or other formats.

jjr840430 commented 6 years ago

@ronaldtse do you have any suggestions?

drystone commented 5 years ago

Since NOS configuration is compiled with the main program, I suggest the following another section:

program {
       name "<program name>"
       version "<program version>"
       copyright "<copyright message>"
       homepage "<homepage URL>"
}

version, copyright and homepage may be used for command line to show version info.

This is a good idea @jjr840430. Maybe not name as it's generally argv[0]. Unless there is any occasion when you'd want to report any name other than the name used to invoke the program itself?

ronaldtse commented 5 years ago

I think to distinguish "commands" (necessary, identifies operation) and "options" (optional, modifies existing operation), we can use the names command vs option. A command can allow multiple options, and a command can contain smaller commands too.

For example:

  1. git checkout -b <branch>, checkout is a command, -b is an option, <branch> is a command argument.

  2. rvc edit vpn1 --auto-connect enable, edit is command, vpn1 is command argument, --auto-connect is an option, enable is the option assignment for --auto-connect

  3. cp /tmp/1 /tmp2, both /tmp1/ and /tmp2/ are command arguments

  4. aws s3 ls <bucket-name> [options], s3 is command, ls is a command within s3, <bucket-name> is argument for ls command, [options] are options for the ls command.

ronaldtse commented 5 years ago
program {
       name "<program name>"
       version "<program version>"
       copyright "<copyright message>"
       homepage "<homepage URL>"
       license "<license URL>"
}

Let's also add license!

drystone commented 5 years ago

Options and commands are separate entities. (Sub)commands don't have leading dashes. @ronaldtse called them namespaces - Ie git rm is rm in the context of git, docker rm is rm in the context of docker

Suboptions are different. iptables, for example, has many options that are dependent on specific modules and only available if that module is specified.

iptables -A INPUT -i eth1 -s 10.0.0.0/8 -m limit --limit 5/m --limit-burst 7 -j LOG --log-prefix "LOG_A_FEW: "
iptables -A INPUT -m mac --mac-source 00:0F:EA:91:04:08 -j DROP

Whether we need either of these is undecided. Subcommands are already used by RVC so are probably necessary. Suboptions is less clear. I can see we might use something like

server -p 192.168.0.100:80 --max-connections 24 -p 127.0.0.1:80 --max-connections 4

Multiple args and repeatable options?

format --papersize a4 a5 legal // multiple args
format --papersize a4,a5,legal // one arg
format --papersize a4 --papersize a5 --papersize legal

There's plenty to think about! We need to determine exactly what commandline processing we want available for ongoing/future Ribose products. If we get this right we'll have a suite of products with ergonomic and uniform user interfaces. If we're too restrictive we'll struggle to get command lines to invoke the behaviour we require. And if we're too permissive we'll end up with an unwieldy library that's hard to use and maintain.

jjr840430 commented 5 years ago

Since NOS configuration is compiled with the main program, I suggest the following another section:

program {
       name "<program name>"
       version "<program version>"
       copyright "<copyright message>"
       homepage "<homepage URL>"
}

version, copyright and homepage may be used for command line to show version info.

This is a good idea @jjr840430. Maybe not name as it's generally argv[0]. Unless there is any occasion when you'd want to report any name other than the name used to invoke the program itself?

@drystone I think we may add API function to update program info.

jjr840430 commented 5 years ago

I think it's not clear between command and option. Here is an example:

./example -c test
./example -create test

The example utility has -c or -create to create something with a name test. Those have the same behavior and output. But someone could think -c as option, -create as command.

I think we need to use a popular term option in NOS syntax. It will work for all command line formats without any collisions.

If we need to separate them to command, subcommand, option, suboption, then we need to declare own command line formats and let programmers write NOS syntax based on them.

so I think we may assume only option and suboption. option and suboption is statically defined option strings by the programmer, and the remainings which are given at runtime are arguments.

In the above example, -c and -create are options which was statically defined by the programmer, and test is argument which was given at run-time.

drystone commented 5 years ago

Agreed. Program name can be user facing, eg. Riffol. Binary name is what is used to invoke and is argv[0] (with path component removed) eg. riffol

-create != --create

Long options have the form --option. Some programs ignore this and it's irritating :)

There is a potential problem though -create may be parsed as -c reate -c -r eate...-c -r -e -a -t -e`. I guess option names should take precedence. We'll be using option parsing libraries (such as getopt) so we'll have to rely on their behaviour in these cases.

If we need to separate them to command, subcommand, option, suboption, then we need to declare own command line formats and let programmers write NOS syntax based on them.

so I think we may assume only option and suboption. option and suboption is statically defined option strings by the programmer, and the remainings which are given at runtime are arguments.

command is implicit - it's the program itself. How would someone implement rvc edit ... without some form of subcommand?

jjr840430 commented 5 years ago
rvc --edit test --auto-connect disable

We may think edit as option, --auto-connect as suboption. And test and disable is argument for them.

Also, some command line utilities may not support getopt or getopt_long supported by libc. So we may think about -create option. It means that we don't need to rely on getopt or getopt_long standard functions.

For example, macOS has networksetup utility which has the following example:

networksetup -listnetworkserviceorder
ni4 commented 5 years ago

My opinion what could be improved on command line args config, probably this could be helpful in some way:

  1. Make things shorter and more flexible for options. We don't need to stick to short option name or long option name, or just two of them, this notation looks simpler (if it is supported by parse you are sticked to):
    option encrypt "-e",  "--encrypt", "encrypt" {
    usage "Encrypt file"
    }
  2. I don't think we need to distinguish between command/param/option via additional keywords. We could just set it as type:
    option encrypt "-e",  "--encrypt", "encrypt" {
    type command
    }
    option sym-algorithm "aes128", "aes256", "3des" {
    type switch
        default "aes128"
    }
    option userid "-u", "--userid" {
    type string
    }
  3. We could separate top-level (global) options from nested ones, which will be references from top-level or other options, i.e.:
    
    option encrypt "-e", "--encrypt", "encrypt" {
    type command, global
        usage "Encrypt file"
        suboptions {
                sym-algorithm?,
                userid+,
                source+,
                destination?
        }
    }

option decrypt "-d", "--decrypt", "decrypt" { type command, global usage "Decrypt file" suboptions { password?, source+, destination? } }

option password "-p", "--password" { type string, global }

option source { type path default stdin }

option destination { type path default stdout }


Symbols like "?", "+", "!" or keywords could be used to set whether further params are required/variable/can be stacked.
drystone commented 5 years ago

Thanks @ni4 .

If I understand correctly, you're advocating subcommands but as an option subtype. I think they're sufficiently different to have separate names. A subcommand is a set of options and an option cannot contain other options. If a subcommand is selected on a command line, all other subcommands (and their associated options) are ignored. Normal options aren't mutually exclusive in this way.

Suboptions, however, can be described using the option constraints you suggested. Perhaps requires and conflicts would be sufficient?

option global { ... }
command cmd {
    option cmdopt1 {
        .... 
        conflicts [cmdopt2]
    }
    option cmdopt1_1 {
        ...
        requires [cmdopt1]
    }
    option cmdopt2 { 
        ....
        conflicts [cmdopt1]
    }
    command nested {
        option nestedopt { ... }
    }
}

option encrypt "-e", "--encrypt", "encrypt" { ... } looks tidy but would be parsed into nested dicts that would need flattening:

option encrypt {
    "-e" {
        "--encrypt" {
            "encrypt" { ... }
        }
    }
}

... /can be stacked

Could you explain what 'stacked' means here?

ronaldtse commented 5 years ago

Sorry for the late reply, but I think by "stacked" @ni4 meant that an option can be used with another option. I think the graph sort of syntax (conflicts [opt], requires [opt]) could be extended (e.g., allows [opt]) to indicate which options are compatible with what other options.

Is this correct @ni4 ?

ni4 commented 5 years ago

@ronaldtse @drystone Sorry for a late response, missed notification. Yeah, @ronaldtse is right - so options may have other suboptions, and so on, without limitation of nesting levels. So we'll have tree, where root is command line, first level - commands/global options, and each of them may have subcomands/suboptions.

Also idea is to describe suboptions/subcommands just by their name to allow to reuse them - say, hash algorithm param with predefined list of constants may be reused for key generation, signing, whatever else. I.e. instead of

command cmd {
    option cmdopt1 {
        .... option description...
    }
}

have

command cmd {
    option cmdopt1 required;
}

option cmdopt1  {
     .... option description...
}
jjr840430 commented 5 years ago

@ronaldtse @drystone Do you have suggestions related to usage syntax from NOS?

drystone commented 5 years ago

@jjr840430 do you mean the syntax of the usage strings for individual options?