Beats configuration "dotted" keys issue

[aleksmaus]

It looks like the "dotted" keys are not supported in the configuration as of July 27th, 2021.

Trying to introduce the ecs_mapping for osquery_manager integration.

The data stream template (stream.yml.hbs) looks like this:

query: {{query}}
{{#if interval}}
interval: {{interval}}
{{/if}}
id: {{id}}
{{#if platform}}
platform: {{platform}}
{{/if}}
{{#if version}}
version: {{version}}
{{/if}}
{{#if ecs_mapping}}
ecs_mapping:
{{#each ecs_mapping}}
   '{{@key}}':
      {{#each this}}
         '{{@key}}': {{this}} 
      {{/each}}
{{/each}}
{{/if}}

Submitting the integration configuration payload with ecs_mapping that looks like the following:

....
                        "ecs_mapping": {
                            "value": {
                                "user.custom.shoeSize": {
                                    "value": 45
                                },
                                "user.id": {
                                    "field": "uid"
                                },
                                "user.name": {
                                    "field": "username"
                                }
                            }
                        }
....

Kibana renders the stream configuration with ecs_mapping like the following

    streams:
      - id: users
        data_stream:
          dataset: osquery_manager.result
          type: logs
        query: select * from users limit 2
        interval: 60
        ecs_mapping:
          user.custom.shoeSize:
            value: 45
          user.id:
            field: uid
          user.name:
            field: username

The agent receives the ecs_mapping blob correctly as JSON

{
    "ecs_mapping": {
        "user.custom.shoeSize": {
            "value": 45
        },
        "user.id": {
            "field": "uid"
        },
        "user.name": {
            "field": "username"
        }
    }
} 

The beats configuration YAML update is "broken", the beat received the configuration where the keys with dots are unwrapped into the nested YAML tree:

    ecs_mapping:
      user:
        custom:
          shoeSize:
            value: 45
        id:
          field: uid
        name:
          field: username      

Expecting the keys with dots to be preserved as is and the original structure of the document to be unchanged (flat mapping document with the dotted keys".

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ruflin]

@michalpristas @blakerouse @urso I assume this building of the tree somewhere happens between Elastic Agent when the config is shipped down to the process. Maybe go-ucfg is applying some magic here? Is there a reason we "dedot" configs?

[aleksmaus]

go-ucfg

looks like default options use ucfg.PathSep(".") which will split the key

[ruflin]

I wonder if we should disable this in general. Maybe worth trying to remove it and see what happens? Otherwise we could just disabled it for osquerybeat?

[blakerouse]

I don't think we can just remove it. It would break parsing other parts of the elastic-agent.yml.

[aleksmaus]

{
        "foo.bar": {
            "value": 45
        }
        "foo": {
            "bar": {
                "value": 23
            }
        }
}

so with the JSON like this for example we will loose some values after converting to YAML? the last one wins? I understand the compatibility issues with existing yaml configuration. but does it have to be yaml a year from now for example? maybe a config format can be a part of the spec? this possibly would allow to skip yaml conversion for the beats/apps that can consume JSON configuration as is. just thinking out loud.

[ruflin]

@aleksmaus Did we ever make progress on this issue? How did you solve it?

[aleksmaus]

@aleksmaus Did we ever make progress on this issue? How did you solve it?

as far as I understood from the replies that this can't be easily fixing without breaking backwards compatibility. we just adopted what we get in osquerybeat in this particular case, and flattened the tree in osquerybeat code.

[javabrett]

I wonder if I've hit a variant of this issue - my case is different in that I'm just starting with a Beats metricbeat module prometheus configuration, as module configuration in YAML only, so no JSON. I guess the config file is still processed by ucfg with ucfg.PathSep("."), and this is preventing me from expressing config keys which contain a .:

- module: prometheus
  period: 60s
  hosts: ["https://endpoint.with.https:443"]
  metricsets: ["query"]
  queries:
  - name: export
    path: /path/to/export
    params:
      foo.bar: baz
  username: "someuser"
  password: "somesecretpassword"

... results in a bad interpretation of key foo.bar resulting in a query https://endpoint.with.https:443/path/to/export?foo=map%5Bbar%3Abaz%5D or decoded, foo=map[bar:baz].

Are there any known escaping or quoting workarounds for this?

Also posted to https://discuss.elastic.co/t/how-can-i-represent-a-config-key-containing-a-dot-period-full-stop/301907

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[aleksmaus]

My memories are vague about this issue, it's been awhile. I think for osquerybeat we "worked around" this limitation. Could potentially be closed.

[ruflin]

I think the problem that how the configs are treated is not always the same still persists. @joshdover Reminds me of the standalone elastic agent experience where dots sometimes work and sometimes don't.

[bartoszcisek]

This issuer might be related to escaping URLs in query argument https://github.com/elastic/elastic-agent/issues/3421

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!