Logs in Datadog for NGINX+ should parse

[tpendragon]

Right now we can't extract fields from the NGINX logs in datadog. We probably need to add a parser to https://app.datadoghq.com/logs/pipelines

This is critical because without it we can't use facets to do things like measure how many user agents are coming in for a specific service, what IPs they're going to, or who the client IPs are.

[acozine]

Could we configure the nginx+ logs to use JSON? Datadog automatically parses JSON files, and this might be an easier lift (and more futureproofed) than writing our own parsing rules.

See this SO post and this datadog docs page.

[acozine]

Some of the scenarios I can think of are possible now. For example, this view shows all log messages related to the /catalog path in the last fifteen minutes, and from there we can use facets to see requests from a single client IP to that path. What are the other use cases? Can we add acceptance criteria?

[acozine]

Ah, we DO have a pipeline for nginx/nginxplus enabled. Is this ticket done, then?

[tpendragon]

https://app.datadoghq.com/logs?query=service%3Anginxplus&cols=host%2Cservice&event=AgAAAY6BE9B2c_CY8AAAAAAAAAAYAAAAAEFZNkJFOUYwQUFETVpMQTJDamxsdEFFNQAAACQAAAAAMDE4ZTgxMTMtZDU5NS00YWVkLWFmNWUtNmY3OWEzNmNjM2Qx&index=%2A&messageDisplay=inline&storage=hot&stream_sort=desc&viz=stream

When I look at nginxplus logs I don't get any event attributes, I'm assuming our log format is a little different from whatever the default is.