[Fleet] Allow multiple inputs configuration for the integration configuration

[aleksmaus]

Problem statement: The osquerybeat integration configuration should support configuring multiple queries with different execution intervals. Currently there is no way to do this with kibana. After discussion with @ruflin he suggested to start with opening Kibana issue.

Describe a specific use case for the feature:

Here is an example screenshot from our initial proof of concept:

The manifest for this particular integration:

title: OSQuery queries
type: logs
release: experimental
streams:
  - input: osquery
    title: OSQuery queries configuration
    description: OSQuery queries configuration
    vars:
      - name: query
        type: text
        title: The query to be executed.
        multi: true
        required: true
        show_user: true
        default:
          - select * from users
      - name: period
        type: text
        title: Execute every (leave blank for on-time query)
        required: false
        show_user: true
        default:
          - 15m
      - name: id
        type: text
        title: Query ID (should be unique per query)
        required: true
        show_user: true
        default:
          - 123-456

Possible solution:

The possible solution could be to allow the multi flag at the input level

streams:
  - input: osquery
    multi: true

and added + icon or something on Kibana UI allowing to add more inputs.

Thoughts?

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

Support to specify the same input multiple times would also help the Uptime use case (specify more then 1 ping at the time) or the "Custom logs" to have multiple paths. @jen-huang You might know best if this is feasible with the current structure.

[jen-huang]

It's not feasible with the current structure unless maybe instead of exporting 3 vars, the package exports 1 YAML var and allow the user to configure any number of queries in YAML format. But this isn't great UX so maybe a multi option makes sense here.

I'd like to know what we'd like the final input block in elastic-agent.yml to look like for multiple queries? Would each query config become its own stream with it's own ID? Currently we generate an ID for each input stream, as in this system agent YAML example:

inputs:
  - id: 5b1d9a0a-043e-45f7-8673-4a15499f5d85
    name: system-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 0.9.1
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth-5b1d9a0a-043e-45f7-8673-4a15499f5d85
        data_stream:
          dataset: system.auth
          type: logs
        paths:
        ...
      - id: logfile-system.syslog-5b1d9a0a-043e-45f7-8673-4a15499f5d85
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
        ...

[ruflin]

My expectation would be that the full input would be repeated: (please ignore same ids)

  - id: 5b1d9a0a-043e-45f7-8673-4a15499f5d85
    name: system-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 0.9.1
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth-5b1d9a0a-043e-45f7-8673-4a15499f5d85
        data_stream:
          dataset: system.auth
          type: logs
        paths:
  - id: 5b1d9a0a-043e-45f7-8673-4a15499f5d85
    name: system-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 0.9.1
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.syslog-5b1d9a0a-043e-45f7-8673-4a15499f5d85
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
        ...

[jamiehynds]

The Security Integrations team are in the process of migrating winlogbeat functionality to Elastic Agent packages. We are currently focused on allowing users to ingest custom windows event logs, see proposal doc here.

We have a strong dependency on this multiple inputs issue, as without it, a user would have to enable one package for each custom windows event channel. e.g. ingesting AppLocker, DNS, and DHCP event channels would require a user to enable three packages. Multiple inputs would also open up the possibility of adding custom events to the existing Windows package, without the need for a separate custom package.

Can we raise the priority on the Kibana side? As mentioned above, osquery also has a dependency and I understand Uptime do too.

[mostlyjason]

Sounds like a good solution to me. Would help for several use cases mentioned above.

[ph]

I will put tentative 7.12 on this, @jamiehynds would be nice to collaborate on the implementation.

[ph]

@jamiehynds @aleksmaus I propose we collaborate to fit it in the roadmap, can @aleksmaus or @jamiehynds make a PR to add it to the package spec once the manifest and usage is formalize we can move forward with the implementation.

[jamiehynds]

@leehinman would you mind creating the PR as per PH above?

[jamiehynds]

@ph @aleksmaus - @leehinman has submitted a PR to the package-spec: https://github.com/elastic/package-spec/pull/110

[jen-huang]

@jamiehynds @leehinman Will OSquery package be updated to use the package spec changes for 7.13? Will we need to do this Kibana work for 7.13 too? cc @ph

[ph]

@jamiehynds I thought this feature was not a requirement for osquery? cc @mostlyjason