st2 pack config <pack_name> does not handle object references

[nmaludy]

When running st2 pack config vsphere .... if you look at the example config (https://github.com/StackStorm-Exchange/stackstorm-vsphere/blob/master/vsphere.yaml.example) and the schema (https://github.com/StackStorm-Exchange/stackstorm-vsphere/blob/master/config.schema.yaml) it does not seem to be generating a config that matches the example:

[root@stackstorm ~]# st2 pack config vsphere
ssl_verify (boolean) [y]: y
vcenters.passwd (secret): ********
vcenters.host: hostname
vcenters.port (integer): 1234
vcenters.user: username
taskinfo.vsphere [default]: 
taskinfo.tasknum (integer) [1]: 
---
Do you want to preview the config in an editor before saving? [y]: n
---
Do you want me to save it? [y]: y
+----------+------------------------------------+
| Property | Value                              |
+----------+------------------------------------+
| pack     | vsphere                            |
| values   | {                                  |
|          |     "ssl_verify": true,            |
|          |     "vsphere": {},                 |
|          |     "vcenters": {                  |
|          |         "passwd": "password",      |
|          |         "host": "hostname",        |
|          |         "port": 1234,              |
|          |         "user": "username"         |
|          |     },                             |
|          |     "sensors": {                   |
|          |         "taskinfo": {              |
|          |             "vsphere": "default",  |
|          |             "tasknum": 1           |
|          |         }                          |
|          |     }                              |
|          | }                                  |
+----------+------------------------------------+

The yaml file that is generated looks like:

[root@stackstorm ~]# cat /opt/stackstorm/configs/vsphere.yaml 
sensors:
  taskinfo:
    tasknum: 1
    vsphere: default
ssl_verify: true
vcenters:
  host: hostname
  passwd: password
  port: 1234
  user: username
vsphere: {}

What i want it to look like:

[root@stackstorm ~]# cat /opt/stackstorm/configs/vsphere.yaml 
sensors:
  taskinfo:
    tasknum: 1
    vsphere: default
ssl_verify: true
vsphere: 
  some-name-i-could-enter:
    host: hostname
    passwd: password
    port: 1234
    user: username

[lakshmi-kannan]

@lhill and @nmaludy: I looked into this and it appears we aren't strictly following JSONSchema standards which makes this hard to implement with standard jsonschema ref resolvers.

For example, https://spacetelescope.github.io/understanding-json-schema/structuring.html. jsonschema wants people to group inline refs into a "definitions"group. This is super critical. I could actually look for "ref" keyword in "properties" or "patternProperties" and then appropriately resolve the schema but when the parser hits the ref like vcenter here https://github.com/StackStorm-Exchange/stackstorm-vsphere/blob/master/config.schema.yaml#L15, parser won't be able to tell that it was a ref and not an actual object. So basically what I am saying is that the config schema handler today 1. doesn't handle ref 2. the config.schema doesn't really use the JSONSchema strict syntax. I can think of some hacky ways to fix this but ideally I'd want us to think more to see if we can do something better in terms of user simplicity and following jsonschema syntax.

I don't have a good workaround yet. Let me try playing with this more.

[LindsayHill]

I am shocked, shocked to hear that this is complicated.

I suspect that a short-term workaround will be to drop into an editor with the example config

[Kami]

@lakshmi-kannan Yeah, the ref thing I did was a bit of a hack / work-around.

The reason for that is that we don't expose full raw JSON schema functionality for parameters. We basically takes stuff from parameters and put it into "definitions".

See https://github.com/StackStorm/st2/issues/3276 for more details (so we could do some "magic" and put things under special attribute under definitions or similar).

[blag]

Instead of using JSONSchema references, could we use YAML anchors and references to solve this problem?

There is a PR for the jira pack (StackStorm-Exchange/stackstorm-jira#19) that attempts to implement multiple profiles in the Jira pack config. If I understand this issue correctly, that could also be used here.

See my comment here for a primer.

This is the final config schema.

That PR ran into an issue they could not work around, and I have tried to add a few test cases to our test code, but I couldn't figure out where I should add pack config fixtures and I've been working on unrelated higher priority tasks.

@nmaludy Would you be interested in trying YAML anchors and references to see if that would work for you? Please ping me if you need help.