Dynamically Parse Object Arrays

[introsuit]

We are encountering an issue where we are unable to dynamically parse arrays of objects from a YAML configuration file.

Example YAML Configuration:

push:
  fcm:
    - name: "app1"
      project-id: "foo"
      service-account: "bar"

    - name: "app2"
      project-id: "baz"
      service-account: "qux"

When attempting to parse the above structure using ffyaml, we are unable to convert this configuration into a struct array. For instance, using fs.Var(&cfg.Push, "push", "") won't do the trick. The logic traversing the structure skips the array because it assumes there is nothing to parse.

It appears that while ff can handle nested structures, it struggles with arrays of objects or maps. The traverse_map() function (see source code) recursively traverses the configuration until it finds a primitive type in the nested structure, rather than providing an option to handle parsing the entire structure.

We are seeking thoughts or proposals for future implementation to address this issue. Any suggestions on how to effectively handle the parsing of such nested arrays of objects would be greatly appreciated.

[peterbourgon]

I guess cfg.Push is something like

type Push struct {
    FCM []struct {
        Name           string
        ProjectID      string
        ServiceAccount string
    }
}

?

[peterbourgon]

I don't think ffyaml is a fit for this use case. All of the ff— packages expect files where each (fully-expanded) identifier is unique, and corresponds to a single flag in the flag set. Said another way, you can't put anything in a ffyaml-parsed YAML config file that can't be equivalently expressed with individual command-line flags. I'm not sure how you'd do that with repeated values, like you have there. If you have that YAML in an e.g. config.yaml file somewhere, then I think you want to define a string flag for a push config file (or whatever), parse the flags, and then decode the specified file into the cfg.Push struct with a regular YAML parser.

[introsuit]

I guess cfg.Push is something like

type Push struct {
    FCM []struct {
        Name           string
        ProjectID      string
        ServiceAccount string
    }
}

?

Correct.

Command line version perhaps could look something like this:

--push.fcm[0].name="app1" --push.fcm[0].project-id="foo" --push.fcm[0].service-account="bar"
--push.fcm[1].name="app2" --push.fcm[1].project-id="baz" --push.fcm[1].service-account="qux"

I think you want to define a string flag for a push config file (or whatever)

Yes, we had that thought as well as a workaround, but it's not ideal.

[glerchundi]

What about something like this @peterbourgon?

ffval.List[FCMConfig]
fs.Var(ffval.NewList(&v.Push.FCM), "push.fcm", "collection of structs (repeatable)")

We can also request ff tags in the struct to be mandatory in these cases, for example:

type FCMConfig struct {
  Name string `ff:"shortname: n, longname: name, usage: name string"`
  ...
}

And the access to the nested properties would be as @introsuit proposed:

--push.fcm[0].name="app1" --push.fcm[1].name="app2"

[peterbourgon]

Command line version perhaps could look something like this:

--push.fcm[0].name="app1" --push.fcm[0].project-id="foo" --push.fcm[0].service-account="bar"
--push.fcm[1].name="app2" --push.fcm[1].project-id="baz" --push.fcm[1].service-account="qux"

What would be the result of --push.fcm[999].project-id="x"? An array of 999 FCMs in the Push value, the first 998 of which having zero values for Name, ProjectID, and ServiceAccount; and the 999th having zero values for Name and ServiceAccount, but ProjectID set to x?

[glerchundi]

No, it would dramatically fail if conditions weren't met. That is, all objects must be present in this static way of defining configuration. WDYT?

[peterbourgon]

OK, so what would be the result of parsing the following?

• --push.fcm[999].name=a — presumably you would expect Parse to return an error here?
• --push.fcm[0].name=a --push.fcm[1].name=b — is it OK to specify index 1 if index 0 is only partially-specified?
• --push.fcm[0].name="" --push.fcm[1].name=x — what if we explicitly specify zero values? is that OK?
• --push.fcm[1].name=a --push.fcm[0].name=b — and does order matter?

These are all kind of rhetorical questions. Whatever you might decide is the best answer for each/any of them, it's guaranteed that someone else will prefer the opposite.

The ffjson/ffyaml/ffenv/etc. packages are not intended to be general-purpose parsers for arbitrary data into arbitrary structures. They're intended to be helpers for parsing a very narrowly-defined subset of their corresponding languages into explicitly-defined and corresponding flag values. One-to-one. Dynamic arrays of arbitrary and/or nested values are definitely out of scope.