Update Nginx to mimic ops-agent

[Mrod1598]

Updated regexes to mimic ops-agent

Example Access logs:

::1 - - [26/Aug/2021:16:49:43 +0000] "GET / HTTP/1.1" 200 10701 "-" "curl/7.64.0"
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

Example Access output:

{
   "timestamp":"2000-10-10T13:55:36-07:00",
   "severity":30,
   "severity_text":"200",
   "labels":{
      "file_name":"example.log",
      "log_type":"nginx.access",
      "plugin_id":"nginx"
   },
   "record":{
      "host":"-",
      "http_request_protocol":"HTTP/1.0",
      "http_request_referer":"",
      "http_request_remoteIp":"127.0.0.1",
      "http_request_requestMethod":"GET",
      "http_request_requestUrl":"/apache_pb.gif",
      "http_request_responseSize":"2326",
      "http_request_status":"200",
      "http_request_userAgent":"",
      "user":"frank"
   }
}{
   "timestamp":"2021-08-26T16:49:43Z",
   "severity":30,
   "severity_text":"200",
   "labels":{
      "file_name":"example.log",
      "log_type":"nginx.access",
      "plugin_id":"nginx"
   },
   "record":{
      "host":"-",
      "http_request_protocol":"HTTP/1.1",
      "http_request_referer":"-",
      "http_request_remoteIp":"::1",
      "http_request_requestMethod":"GET",
      "http_request_requestUrl":"/",
      "http_request_responseSize":"10701",
      "http_request_status":"200",
      "http_request_userAgent":"curl/7.64.0",
      "user":"-"
   }
}

Example Error log

2021/08/26 16:50:17 [error] 29060#29060: *2191 open() "/var/www/html/forbidden.html" failed (13: Permission denied), client: ::1, server: _, request: "GET /forbidden.html HTTP/1.1", host: "localhost:8080"

Example Error Log output

{
   "timestamp":"2021-08-26T16:50:17-04:00",
   "severity":60,
   "severity_text":"error",
   "labels":{
      "file_name":"example.log",
      "log_type":"nginx.error",
      "plugin_id":"nginx"
   },
   "record":{
      "client":"",
      "connection":"2191",
      "host":"",
      "message":"",
      "pid":"29060",
      "referer":"",
      "request":"",
      "server":"",
      "subrequest":"",
      "tid":"29060",
      "upstream":""
   }
}

[schmikei]

@jsirianni do we need to worry about mappings for this? Or is that process automated now?

[jsirianni]

@jsirianni do we need to worry about mappings for this? Or is that process automated now?

Yes we absolutely need to consider mappings.

I'm a bit out of the loop, what is the motivation for gaining parity with ops agent? Mappings are difficult to change once implemented.

[jsirianni]

@Mrod1598 Couple things

1. Can you provide input and output examples when running with Stanza v1?
2. Does this same change need to be made to the OTEL branch? I suspect it does. https://github.com/observIQ/stanza-plugins/tree/updated-fields

[Mrod1598]

@Mrod1598 Couple things

1. Can you provide input and output examples when running with Stanza v1?
2. Does this same change need to be made to the OTEL branch? I suspect it does. https://github.com/observIQ/stanza-plugins/tree/updated-fields

Added examples, and probably, going to chat with Josh about some this and come back to this.

[jsirianni]

Another note. We will need to update the observiq log format on our doc site and in the plugin doc.

[Mrod1598]

Another note. We will need to update the observiq log format on our doc site and in the plugin doc.

is this still required?

[jsirianni]

Another note. We will need to update the observiq log format on our doc site and in the plugin doc.

is this still required?

No, but we should track this.