Support logging incoming requests

[jsoref]

It'd be nice if there was an optional way to log incoming requests.

Prior to #11821, it was probably possible to write a lua plugin for the rewrite event which logged the incoming request.

Without that hook, I don't think it's possible.

I'm reasonably confident that it is possible today for a request to be received by ingress-nginx, sent along to a backend, and then have the ingress-nginx pod killed by kubernetes (for whatever reason) prior to the response being returned to ingress-nginx and thus have no logs showing the request arriving.

The reason I'm looking for this feature is that I have a number of logged consumers who claim they sent requests to my system (whose dns points to my public ip address, where that ip address is routed to ingress-nginx, and for which there are ingress objects configured to send the traffic to backends, and for which i have a default backend) and while I have many logs from ingress-nginx about other request-responses, I can't find record of these requests. I'd like to rule out the possibility that the request was received by ingress-nginx and somehow dropped on the floor without a response being returned (or without it logging the response -- which I think is even less likely). As noted, it's technically possible for this to happen, at least in the case of the ingress-nginx pod dying before sending a response.

No

No

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[chengjoey]

It seems that there is only one possibility that incoming requests are not logged, that is that the nginx-pod is killed when no response is received

[jsoref]

fwiw, i'm going to try experimenting with /etc/nginx/lua/plugins/lua-logging/main.lua:

local ngx = ngx
local cjson_encode = require("cjson").encode

local string_format = string.format
local ngx_log = ngx.log

local _M = {}

function _M.rewrite()
  local headers = {}
  local data = {}
  data.debugging = true
  local h, err = ngx.req.get_headers()

  if err == "truncated" then
    data.truncated = true
  end
  for k, v in pairs(h) do
    headers[k] = v
  end
  data.headers=headers

  ngx_log(ngx.NOTICE, cjson_encode(data))
end

return _M

Enabled by adding "plugins": "lua-logging" to the ingress-nginx configuration configmap. -- Obviously I won't be able to do something like this with the next release, hence the feature request.

The ability to turn on/off a request log (as opposed to the response log that is included out of the box) would be handy for proving to clients "yes, we really didn't receive your request" or for DevOps folks to say "hey, that request was received, but then the ingress pod died -- we should find out why". Or in case we end up in a third state "Hey, we received that request, there's no logged response and the pod didn't die -- we really need to investigate, because something's clearly not working properly".

I hope to have an answer to the problem I'm debugging soon enough now that I have a logging system that can work -- but it would be rather unfortunate if I had no way to deploy something like this later (i.e. after upgrading to the next release).