YAML configuration support

[skybert]

Hi there,

Unit seems to be an awesome app server that I see can find the bill for many serious. My only real gripe, is that I'd like to check the configuration into version control and a JSON document isn't my cup of tea in that regard:

$ cat unit.config

{
    "applications": {
        "example_python": {
            "type": "python 3.10",
..            
        }
    },
}

Of course, I could check in a YAML file:

applications:
  example_python:
    type: "python 3.10"

And then pre-process it before PUTing it to Unit:

$ json_xs -f yaml -t json < unit.yaml > unit.json 
$  curl \
    --method PUT \
    --data-binary @unit.json \
    --unix-socket /var/run/control.unit.sock \
    http://localhost/config 

But that makes it just that much harder for people to do, so I believe most folks will avoid or not even consider this approach.

Thus, I propose adding native YAML support so that you can do:

$  curl \
    --method PUT \
    --header 'Content-type: application/yaml' \
    --data-binary @unit.yaml \
    --unix-socket /var/run/control.unit.sock \
    http://localhost/config 

Preferably, comments and whitespace would be preserved, but this is not a must, GETing it back without them, would be more than good enough:

$  curl \
    --header 'Accept: application/yaml' \
    --unix-socket /var/run/control.unit.sock \
    http://localhost/config 

If the client doesn't specify content-type when reading or writing, Unit will assume application/json as before.

Cheers,

-Torstein

[ac000]

Hi there,

Unit seems to be an awesome app server that I see can find the bill for many serious. My only real gripe, is that I'd like to check the configuration into version control and a JSON document isn't my cup of tea in that regard:

Could you elaborate on why checking JSON into version control is an issue for you?

[skybert]

Could you elaborate on why checking JSON into version control is an issue for you?

Sure thing.

I think of conf as source code. I edit them by hand, indent them properly and write comments wherever I deem useful. I like to be able to plug in linters in pre-commit or pre-merge hooks to ensure they're sound and I appreciate never having non-intentional whitespace diffs.

JSON is a nice serialization of the configuration, it's easy to parse with jq, for instance. However, it has these shortcomings IMHO:

1) It's harder to write and easier to get wrong than conf, ini and yaml. In this wee example, I have to quote all the keys and all the string values. I must also remember to put in commas between the fields, as well as line the braces up correctly (the latter I don't mind).

{
  "user": {
    "name": "Lisa",
    "email": "lisa@exampel.com"
  }
}

The YAML version is faster to write and easier to read:

user:
  name: Lisa
  email: lisa@example.com

2) The second problem with JSON as the source configuration format, is that it doesn't support comments. Now, I know that Unit allows you to add comments in the JSON, and strips these on the way in, before serializing the configuration object. Still, this poses the problem that your editor and linters will complain (e.g. vim will mark the comments in red) and JSON tools like jq will refuse to parse it:

{
    /* hello world */
    "name": "john"
}

$ jq < /tmp/foo.json
jq: parse error: Invalid numeric literal at line 2, column 7

3) Third, I think it's worth considering that few other servers use JSON as the main format. In fact, I can't think of a single one that uses it (sure they exist of course): kubernetes, nginx, apache, lighthttpd, mariadb, sshd, systemd.

Cheers,

-Torstein

[lcrilly]

Hi @skybert

I think it's worth considering that few other servers use JSON as the main format. In fact, I can't think of a single one that uses it (sure they exist of course)

Like Unit, Caddy uses JSON as the canonical configuration format. But it provides adapters from several other formats. We might consider something similar - it's an interesting concept.

It seems unlikely that we would want to add native YAML support into the core of Unit, but adding format converters to the unitc CLI tool seems reasonable. Here's a patch that does that for YAML. Would this meet your needs?

diff -r 6380c3e7e6a5 tools/unitc
--- a/tools/unitc   Wed Sep 06 02:58:19 2023 +0100
+++ b/tools/unitc   Mon Sep 25 10:27:45 2023 +0100
@@ -12,6 +12,7 @@
 QUIET=0
 URI=""
 SSH_CMD=""
+YAML2JSON=""
 METHOD=PUT
 CONF_FILES=()

@@ -32,6 +33,21 @@
            shift
            ;;

+       "-Y" | "--YAML")
+           for y2j in "yq eval -P --output-format=json" "json_xs -f yaml -t json"; do
+                   hash ${y2j%% *} 2> /dev/null # Remove chars beyond space
+                   if [ $? -eq 0 ]; then
+                           YAML2JSON=$y2j
+                           break #for
+                   fi
+           done
+           if [ "$YAML2JSON" == "" ]; then
+               echo "${0##*/}: ERROR: jq(1) or json_xs(1) is required to apply YAML configuration"
+               exit 1
+           fi
+           shift
+           ;;
+
        "GET" | "PUT" | "POST" | "DELETE" | "INSERT" | "EDIT")
            METHOD=$OPTION
            shift
@@ -45,15 +61,20 @@
        *)
            if [ -f $1 ] && [ -r $1 ]; then
                CONF_FILES+=($1)
+               if [ "${1##*.}" == "yaml" ]; then
+                   echo "${0##*/}: INFO: converting $1 to JSON"
+                   shift; set -- "--yaml" "$@" # Force the --yaml option
+               fi
            elif [ "${1:0:1}" = "/" ] || [ "${1:0:4}" = "http" ] && [ "$URI" = "" ]; then
                URI=$1
+               shift
            elif [ "${1:0:6}" = "ssh://" ]; then
                UNIT_CTRL=$1
+               shift
            else
                echo "${0##*/}: ERROR: Invalid option ($1)"
                exit 1
            fi
-           shift
            ;;
    esac
 done
@@ -74,6 +95,7 @@
   EDIT          # Opens the URI contents in \$EDITOR
   INSERT        # Virtual HTTP method to prepend data to an existing array
   -q | --quiet  # No output to stdout
+  -y | --yaml   # Convert configuration data from YAML format into JSON

 Local options
   -l | --nolog  # Do not monitor the error log after applying config changes
@@ -249,6 +271,10 @@
            exit 3
        fi
    else
+       if [ "$YAML2JSON" != "" ]; then
+           cat ${CONF_FILES[@]} | $YAML2JSON > /tmp/${0##*/}.$$_json
+           CONF_FILES=(/tmp/${0##*/}.$$_json)
+       fi
        cat ${CONF_FILES[@]} | $SSH_CMD curl -X $METHOD --data-binary @- $UNIT_CTRL$URI 2> /tmp/${0##*/}.$$ | $OUTPUT
    fi
 fi

[skybert]

Hi @lcrilly ,

It seems unlikely that we would want to add native YAML support into the core of Unit, but adding format converters to the unitc CLI tool seems reasonable. Here's a patch that does that for YAML. Would this meet your needs?

If not getting it into the core, yes, I believe this is a good compromise and, it would work well for me. unitc is in PATH after apt-get installing Unit and it's ready to go.

If added to unitc, I guess the only challenge left is to tell the world. The documentation seems to favour curl in its examples, I don't see unitc mentioned on the configuration or control API pages, for instance, but perhaps I've skimmed through that too fast.

Cheers,

-Torstein

PS: That's a neat approach. btw (I tend to use command -v, but the hash is cool):

for y2j in "yq eval -P --output-format=json" "json_xs -f yaml -t json"; do
  hash ...
done

I'm sure you've considered this, and I'm just being annoying, but I think this is easier to read/maintain:

y2js=(
  "yq eval -P --output-format=json"
  "json_xs -f yaml -t json"
)

for y2j in "${y2js[@]}" ; do
  ..
done

Oh well, it requires bash4, so it breaks on lots of Mac without brew installed GNU tools.

[lcrilly]

@skybert thanks for your continued feedback. And yes, we strive for bash3 compatibility with macOS.

The current unitc tool is experimental, and expected to be replaced with a self-contained binary that doesn't rely on external dependencies. However, if we're happy that the current command line experience will be the specification for the stable thing, then we can start to update the documentation to provide alternatives to curl(1).

After further testing, I found that json_xs(1) did not reliably maintain the order of JSON objects. While that isn't a technical problem, it can spoil the experience when your expected config is rearranged. I also found that boolean values were not converted properly. Overall, the jq(1) tool looks like a better way.

I've put a YAML-enabled version of unitc here: https://github.com/lcrilly/unitc

This is likely to make it into the next release. Note that you can use --yaml to convert in both directions, or provide an interactive YAML editing experience if that's convenient, e.g.:

unitc /config edit --yaml

[skybert]

thanks for your continued feedback.

My pleasure.

And yes, we strive for bash3 compatibility with macOS.

Good to know. Guess there still is a substantial amount of developers that don't use brew for their core utils.

I found that json_xs(1) did not reliably maintain the order of JSON objects.

Interesting. I have used json_xs for many years and have never stumbled into this shortcoming. Probably because I've only used the output of json_xs as input to jq and querying the JSON structure has in my use cases never considered order.

I also found that boolean values were not converted properly.

Don't know if this is the thing that bit you, but we stumbled upon a change in the tool a while back, where booleans changed from yes/no to true/false or something along those lines. As a result, a good few build pipelines failed in spectacular ways. Good times.

This is likely to make it into the next release.

Cool!

Note that you can use --yaml to convert in both directions, or provide an interactive YAML editing experience if that's convenient, e.g.:

Nice.