Amend total event duration (request time) field and value with Filebeat's aws/elb ingest pipeline

[georgivalentinov]

Describe the enhancement:

The aws module of Filebeat and its elb fileset make for a great AWS ELB log harvesting tool. Logs are parsed and useful. There's info on the following duration intervals, part of a request's lifecycle:

• The total time in seconds since the connection or request is received until it is sent to a registered backend (aws.elb.request_processing_time.sec).
• The total time in seconds since the connection is sent to the backend till the backend starts responding (aws.elb.backend_processing_time.sec).
• The total time in seconds since the response is received from the backend till it is sent to the client (aws.elb.response_processing_time.sec).
• For NLB (ELBv2) only: the total time of the connection in milliseconds, since it is opened till it is closed (aws.elb.connection_time.ms).

What's missing is the total request time (connection time, event duration) for the ALB (ELBv2) type, and potentially for the ELBv1 types as well.

One way to compute and add it I guess is as follows:

--- a/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml
+++ b/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml
@@ -173,6 +173,14 @@ processors:
       value: '{{ @timestamp }}'
       ignore_empty_value: true

+  - script:
+      lang: painless
+      if: ctx?.event?.start != null && ctx?.event?.end != null
+      source: >-
+        Instant start = Instant.parse(ctx.event.start);
+        Instant end = Instant.parse(ctx.event.end);
+        ctx.event.duration = end.getNano() - start.getNano();
+
   - geoip:
       field: 'source.ip'
       target_field: 'source.geo'

where event.start is the time LB "received the request" and event.end is "the time when the load balancer generated a response to the client". Both are already present.

Tested this pipeline computation in a local setup and seems functional and correct.

Describe a specific use case for the enhancement or feature:

Really useful for tracking down slow, fast, or anything else related to requests duration.

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[raychinov]

Hey, this is still needed. Could someone take a look at the suggestion? 🙏🏼