Support the ingest of the source address and the source port separately

[narph]

Related https://github.com/elastic/beats/pull/9460 https://github.com/elastic/ecs/pull/247

We have introduced log.source.address for TCP/UDP input which is contains both the IP and the port like 127.0.0.1:8080. However, neither the IP nor the port are collected separately, which means that it usually involves a dissect operation to get either the IP or the port separately for data analysis (e.g. aggregation of unique IPs).

- if.has_fields: [log.source.address]
    then:
      - dissect:
          field: log.source.address
          target_prefix: log.source
          tokenizer: '%{ip}:%{port}'
      - convert:
          ignore_missing: true
          fields:
            - from: log.source.port
              type: integer

Can we consider introducing separate fields for ip/port?

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

I think these fields should be proposed to be part of ECS before Filebeat implements them.

[efd6]

I didn't see that log.source wasn't an ECS field. Why are we using it?

[andrewkroh]

I think its usage predates ECS. I don't recall whether anyone proposed added it to ECS in the past. But since Beats are adding fields into the log.* namespace there is a chance of a future collision.

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[aleksmaus]

Hi!

Looks like this ticket is still in our backlog. Trying to understand what to do with this request.

Example of TCP log:

  "log": {
    "source": {
      "address": "127.0.0.1:49666"
    }
  }

It doesn't look like we can change log.source.address anymore, it has to have both host/ip and port. It's not ECS field. It looks like there is a debate about adding more fields under log.source such as ip and port and possible concern about conflicts now or in the future.

Would the addition of ip and port solve this issue?

  "log": {
    "source": {
      "address": "127.0.0.1:49666",
      "ip": "127.0.0.1",
      "port": 49666
    }
  }

What fields should we use instead if above is not acceptable? @andrewkroh

[efd6]

I think this is not doable. I had a branch to do it somewhen, but the ECS definitions precluded doing it.

[aleksmaus]

I think this is not doable. I had a branch to do it somewhen, but the ECS definitions precluded doing it.

Understood. What is the appropriate resolution to this ticket?

[efd6]

I think it can be closed, but please check with Andrew.

[aleksmaus]

@andrewkroh please cast your vote.