TOML parsing glitches concerning dotted keys

Description

The parsing of the TOML config file appears glitchy. Adding a setting to the config file fails (instead taking the default value) depending on whether I use the dotted key syntax (and where it is placed in the file).

I experienced this problem while attempting to configure a container registry mirror for AWS EKS nodes, using the most current containerd/cri mirror config syntax.

Steps to reproduce the issue

1. If I update /etc/containerd/config.toml to append (at the end) the line:

plugins."io.containerd.grpc.v1.cri".registry.config_path = "/etc/containerd/certs.d"

then it fails. According to containerd config dump the CRI registry config path instead remains set to "", which is the default value.

2. If I instead append:

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"

then it succeeds.

3. Alternatively, if I use the dotted key syntax as per (1), but insert it earlier in the file (before all of the other tables, including the plugins."io.containerd.grpc.v1.cri" super-table, are defined) then it succeeds.

Describe the results you received and expected

I expected the dotted key syntax (1) to succeed. All valid TOML syntaxes should be equivalent.

What version of containerd are you using?

containerd github.com/containerd/containerd 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1

Any other relevant information

My test was based from the latest AWS EKS AMI release (v20221112).

Show configuration if it is related to CRI plugin.

Here is my full config.toml for reference:

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"

#plugins."io.containerd.grpc.v1.cri".registry.config_path = "/etc/containerd/certs.d"    # <----- 3 ok

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "<sanitised>"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

plugins."io.containerd.grpc.v1.cri".registry.config_path = "/etc/containerd/certs.d"    # <----- 1 ignored

#[plugins."io.containerd.grpc.v1.cri".registry]                                         # <----- 2 ok
#config_path = "/etc/containerd/certs.d"                                                #

Hy I looked into this. I'm not a TOML expert but I think this has something to do with how TOML defines tables. If you look at the TOML documentation it states that:

Dotted keys create and define a table for each key part before the last one, provided that such tables were not previously created.

And:

Since tables cannot be defined more than once, redefining such tables using a [table] header is not allowed. Likewise, using dotted keys to redefine tables already defined in [table] form is not allowed. The [table] form can, however, be used to define sub-tables within tables defined via dotted keys.

Also take a look at the last two example in https://toml.io/en/v1.0.0#table where the above is described.

I hope this helps you.

containerd / containerd