ConvertTo-Toml does not produce consistent content above a depth > 2

Description

I am attempting to automate provisioning of a TOML configuration file for Grafana's LDAP integration with Active Directory, and am encountering some issues with producing the required output. Specifically, when attempting to generate the new TOML file, anything above a depth of 2 results in content that does not conform to the TOML standard.

Not sure if this is the PS Module or the .NET Assembly that is the root cause.

Example Configuration

The following is an example configuration. This configuration consists of two Domains, each with multiple hosts defined (for redundancy), multiple search_base_dns as well as multiple group_mappings per server.

# --- Domain 1 ---
[[servers]]

host = "dom1dc1.domain1.net dom1dc2.domain1.net"
port = 636
use_ssl = true
start_tls = false

bind_dn = "svc.ldap@domain1.net"

search_filter = "(mail=%s)"
search_base_dns = ["ou=unit1,dc=domain1,dc=net", "ou=unit2,dc=domain1,dc=net"]

[servers.attributes]
member_of = "memberOf"
email = "mail"
name = "givenName"
surname = "sn"
username = "sAMAccountName"

[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 1
grafana_admin = true

[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 2
grafana_admin = true

[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 3
grafana_admin = true

[[servers.group_mappings]]
group_dn = "CN=App_Org2,OU=Org2,OU=unit2,dc=domain1,dc=net"
org_role = "Viewer"
grafana_admin = false
org_id = 2

# --- Domain 2 ---
[[servers]]

host = "dom2dc1.domain2.net dom2dc2.domain2.net"
port = 636
use_ssl = true
start_tls = false

bind_dn = "svc.ldap@domain2.net"

search_filter = "(mail=%s)"
search_base_dns = ["ou=unit1,dc=domain2,dc=net", "ou=unit2,dc=domain2,dc=net"]

[servers.attributes]
member_of = "memberOf"
email = "mail"
name = "givenName"
surname = "sn"
username = "sAMAccountName"

[[servers.group_mappings]]
group_dn = "CN=App_Org3,OU=Org3,OU=unit2,dc=domain2,dc=net"
org_role = "Viewer"
grafana_admin = false
org_id = 3

Expected Behaviour

When using ConvertTo-Toml, with a depth of 5 (as is required for this configuration), the output should be TOML compliant, and be re-usable.

Example:

$tomldata = Get-Content 'ldap.toml' | ConvertFrom-Toml
$tomldata | ConvertTo-Toml

Should produce:

[[servers]]
host = "dom1dc1.domain1.net dom1dc2.domain1.net"
port = 636
use_ssl = true
start_tls = false
bind_dn = "svc.ldap@domain1.net"
search_filter = "(mail=%s)"
search_base_dns = ["ou=unit1,dc=domain1,dc=net", "ou=unit2,dc=domain1,dc=net"]
[servers.attributes]
member_of = "memberOf"
email = "mail"
name = "givenName"
surname = "sn"
username = "sAMAccountName"
[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 1
grafana_admin = true
[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 2
grafana_admin = true
[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 3
grafana_admin = true
[[servers.group_mappings]]
group_dn = "CN=App_Org2,OU=Org2,OU=unit2,dc=domain1,dc=net"
org_role = "Viewer"
grafana_admin = false
org_id = 2
[[servers]]
host = "dom2dc1.domain2.net dom2dc2.domain2.net"
port = 636
use_ssl = true
start_tls = false
bind_dn = "svc.ldap@domain2.net"
search_filter = "(mail=%s)"
search_base_dns = ["ou=unit1,dc=domain2,dc=net", "ou=unit2,dc=domain2,dc=net"]
[servers.attributes]
member_of = "memberOf"
email = "mail"
name = "givenName"
surname = "sn"
username = "sAMAccountName"
[[servers.group_mappings]]
group_dn = "CN=App_Org3,OU=Org3,OU=unit2,dc=domain2,dc=net"
org_role = "Viewer"
grafana_admin = false
org_id = 3

or something similar.

Actual Behaviour

When using ConvertTo-Toml, with a depth of 5 (as is required for this configuration), the output below is observed.

Example:

$tomldata = Get-Content 'ldap.toml' | ConvertFrom-Toml
$tomldata | ConvertTo-Toml

servers = [{host = "dom1dc1.domain1.net dom1dc2.domain1.net", port = 636, use_ssl = true, start_tls = false, bind_dn = "svc.ldap@domain1.net", search_filter = "(mail=%s)", search_base_dns = ["ou=unit1,dc=domain1,dc=net", "ou=unit2,dc=domain1,dc=net"], attributes = {member_of = "memberOf", email = "mail", name = "givenName", surname = "sn", username = "sAMAccountName"}, group_mappings = [{group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net", org_role = "Admin", org_id = 1, grafana_admin = true}, {group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net", org_role = "Admin", org_id = 2, grafana_admin = true}, {group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net", org_role = "Admin", org_id = 3, grafana_admin = true}, {group_dn = "CN=App_Org2,OU=Org2,OU=unit2,dc=domain1,dc=net", org_role = "Viewer", grafana_admin = false, org_id = 2}]}, {host = "dom2dc1.domain2.net dom2dc2.domain2.net", port = 636, use_ssl = true, start_tls = false, bind_dn = "svc.ldap@domain2.net", search_filter = "(mail=%s)", search_base_dns = ["ou=unit1,dc=domain2,dc=net", "ou=unit2,dc=domain2,dc=net"], attributes = {member_of = "memberOf", email = "mail", name = "givenName", surname = "sn", username = "sAMAccountName"}, group_mappings = [{group_dn = "CN=App_Org3,OU=Org3,OU=unit2,dc=domain2,dc=net", org_role = "Viewer", grafana_admin = false, org_id = 3}]}]

Environment

I am using PowerShell Core 7.3.9 on a Mac as well as Linux and Windows Server 2016. For full validation, I have also tried this in PowerShell 5.1 on Windows Server 2016.

get-module pstoml

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     0.3.0                 pstoml                              {ConvertFrom-Toml, ConvertTo-Toml}

Please feel free to reach out if you require additional information.

Is the problem that the output is in the inline format rather than expanded out? When I convert the TOML string provided in the Actual Behaviour I get the following JSON structure back

{
    "servers": [
      {
        "bind_dn": "svc.ldap@domain1.net",
        "host": "dom1dc1.domain1.net dom1dc2.domain1.net",
        "port": 636,
        "search_base_dns": [
          "ou=unit1,dc=domain1,dc=net",
          "ou=unit2,dc=domain1,dc=net"
        ],
        "search_filter": "(mail=%s)",
        "start_tls": false,
        "use_ssl": true,
        "group_mappings": [
          {
            "grafana_admin": true,
            "group_dn": "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net",
            "org_id": 1,
            "org_role": "Admin"
          },
          {
            "grafana_admin": true,
            "group_dn": "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net",
            "org_id": 2,
            "org_role": "Admin"
          },
          {
            "grafana_admin": true,
            "group_dn": "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net",
            "org_id": 3,
            "org_role": "Admin"
          },
          {
            "grafana_admin": false,
            "group_dn": "CN=App_Org2,OU=Org2,OU=unit2,dc=domain1,dc=net",
            "org_id": 2,
            "org_role": "Viewer"
          }
        ],
        "attributes": {
          "email": "mail",
          "member_of": "memberOf",
          "name": "givenName",
          "surname": "sn",
          "username": "sAMAccountName"
        }
      },
      {
        "bind_dn": "svc.ldap@domain2.net",
        "host": "dom2dc1.domain2.net dom2dc2.domain2.net",
        "port": 636,
        "search_base_dns": [
          "ou=unit1,dc=domain2,dc=net",
          "ou=unit2,dc=domain2,dc=net"
        ],
        "search_filter": "(mail=%s)",
        "start_tls": false,
        "use_ssl": true,
        "group_mappings": [
          {
            "grafana_admin": false,
            "group_dn": "CN=App_Org3,OU=Org3,OU=unit2,dc=domain2,dc=net",
            "org_id": 3,
            "org_role": "Viewer"
          }
        ],
        "attributes": {
          "email": "mail",
          "member_of": "memberOf",
          "name": "givenName",
          "surname": "sn",
          "username": "sAMAccountName"
        }
      }
    ]
  }

The structure of the data matches up with the expected format just that the data is inlined. I can certainly see this format is not the desired default and this module should provide a way to specify what format it should be in but I just want to verify whether the issue is that or something else.

It is not just that the data is inlined, if I re-convert the data, the structures have different properties/types than it does the first time I read it from the original toml file, which, should ideally not be the case. For example, when you output the list of "mapping_rules" from the initial read, the data is presented in the expected hash/table format, however, on a subsequent re-read, they become "key/value" pairs instead, meaning the code the handle the data could throw up unexpected results due to this change.

See below:

$ldapconf = get-content .\ldap.toml | convertfrom-toml
$ldap2conf = $ldapconf | convertto-toml -depth 5 | convertfrom-toml

$ldapconf.servers[0].group_mappings

Name                           Value
----                           -----
group_dn                       CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net
org_role                       Admin
org_id                         1
grafana_admin                  True
group_dn                       CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net
org_role                       Admin
org_id                         2
grafana_admin                  True
group_dn                       CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net
org_role                       Admin
org_id                         3
grafana_admin                  True
group_dn                       CN=App_Org2,OU=Org2,OU=unit2,dc=domain1,dc=net
org_role                       Viewer
grafana_admin                  False
org_id                         2

$ldapconf.servers[0].group_mappings.gettype()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     OrderedDictionary[]                      System.Array

$ldap2conf.servers[0].group_mappings.gettype()
InvalidOperation: You cannot call a method on a null-valued expression.
$ldap2conf.servers[0].group_mappings
$ldap2conf.servers[0]

Key             Value
---             -----
host            dom1dc1.domain1.net dom1dc2.domain1.net
port            636
use_ssl         True
start_tls       False
bind_dn         svc.ldap@domain1.net
search_filter   (mail=%s)
search_base_dns {ou=unit1,dc=domain1,dc=net, ou=unit2,dc=domain1,dc=net}
attributes      {[member_of, memberOf], [email, mail], [name, givenName], [surname, sn]…}
group_mappings  {[group_dn, CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net] [org_role, Admin] [org_id, 1] [grafana_admin, True], [group_dn, CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net] [org_role, Admin] [org_id, 2] [grafana_admin, True], [group_dn, CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net] [org_role, Admin] [org_id, 3] [grafana_adm…

$ldap2conf.servers[0]["group_mappings"]

Key           Value
---           -----
group_dn      CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net
org_role      Admin
org_id        1
grafana_admin True
group_dn      CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net
org_role      Admin
org_id        2
grafana_admin True
group_dn      CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net
org_role      Admin
org_id        3
grafana_admin True
group_dn      CN=App_Org2,OU=Org2,OU=unit2,dc=domain1,dc=net
org_role      Viewer
grafana_admin False
org_id        2

So the content is there, but the structure of the data is different.

Also, I don't quite get the same behaviour with the JSON conversion, seeing the "key" and "value" keys in the JSON data instead.

Hope that clarifies.

Thanks for the information, I'll have a play around and see what I can do. When playing around with it briefly yesterday it unfortunately didn't seem like I have control over when Tomlyn inlined an array or table and I couldn't make sense as to what the rules behind it was but at least the data was still correct (just the presentation changed). I did come across a few problems where certain lists weren't serialized properly so I'll look into your last results and see what's happening.

Thanks for looking into this.

I've fixed a few problems I found in https://github.com/jborean93/PSToml/pull/8. While I can't fully control when an inline array of tables or just an array of tables is used with Tomlyn the changes in that PR have made it less strict on when it will use the inline format. With the changes there the $tomldata | ConvertTo-Toml -Depth 5 becomes

[[servers]]
host = "dom1dc1.domain1.net dom1dc2.domain1.net"
port = 636
use_ssl = true
start_tls = false
bind_dn = "svc.ldap@domain1.net"
search_filter = "(mail=%s)"
search_base_dns = ["ou=unit1,dc=domain1,dc=net", "ou=unit2,dc=domain1,dc=net"]
attributes = {member_of = "memberOf", email = "mail", name = "givenName", surname = "sn", username = "sAMAccountName"}
[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 1
grafana_admin = true
[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 2
grafana_admin = true
[[servers.group_mappings]]
group_dn = "CN=App_Admin,OU=Groups,OU=unit1,dc=domain1,dc=net"
org_role = "Admin"
org_id = 3
grafana_admin = true
[[servers.group_mappings]]
group_dn = "CN=App_Org2,OU=Org2,OU=unit2,dc=domain1,dc=net"
org_role = "Viewer"
grafana_admin = false
org_id = 2
[[servers]]
host = "dom2dc1.domain2.net dom2dc2.domain2.net"
port = 636
use_ssl = true
start_tls = false
bind_dn = "svc.ldap@domain2.net"
search_filter = "(mail=%s)"
search_base_dns = ["ou=unit1,dc=domain2,dc=net", "ou=unit2,dc=domain2,dc=net"]
attributes = {member_of = "memberOf", email = "mail", name = "givenName", surname = "sn", username = "sAMAccountName"}
[[servers.group_mappings]]
group_dn = "CN=App_Org3,OU=Org3,OU=unit2,dc=domain2,dc=net"
org_role = "Viewer"
grafana_admin = false
org_id = 3

Notably the attributes is an inline table and unfortunately I have no clue as to how Tomlyn decides to use the inline vs explicit format but at least in this case it's a bit more readable.

As for your other issue there was a problem when deserializing a TOML table that contained an array of tables or arrays. With the new changes in the PR it will actually have an ordered dictionary or array respectively so $ldap2conf.servers[0].group_mappings.gettype() will work.

Hi @jborean93 ,

Thanks for looking at this. Are you planning on publishing a new version of the module to PSGallery, or will I need to update manually?

Regards, JohnB

Thanks for the reminder, v0.3.1 has been pushed to the gallery https://www.powershellgallery.com/packages/PSToml/0.3.1

Thanks for the quick response!

jborean93 / PSToml