bug: Unable to configure the format of `http-logger`

jinnyu commented 2 years ago

Current Behavior

http-logger lua report error.

Expected Behavior

Send the request to the configured service correctly

Error Logs

2022/06/29 17:20:13 [error] 49#49: *1586203 failed to run log_by_lua*: /usr/local/apisix/apisix/utils/log-util.lua:46: attempt to call method 'byte' (a nil value)
stack traceback:
        /usr/local/apisix/apisix/utils/log-util.lua:46: in function 'create_obj_fun'
        /usr/local/apisix/apisix/core/lrucache.lua:95: in function 'lru_log_format'
        /usr/local/apisix/apisix/utils/log-util.lua:57: in function 'get_custom_format_log'
        /usr/local/apisix/apisix/plugins/http-logger.lua:167: in function 'phase_func'
        /usr/local/apisix/apisix/plugin.lua:771: in function 'run_plugin'
        /usr/local/apisix/apisix/plugin.lua:804: in function 'run_global_rules'
        /usr/local/apisix/apisix/init.lua:311: in function 'common_phase'
        /usr/local/apisix/apisix/init.lua:710: in function 'http_log_phase'
        log_by_lua(nginx.conf:328):2: in main chunk while logging request, client: 172.17.132.35, server: _, request: "GET /rest/2.0/ocr/v1/idcard HTTP/1.1", upstream: "https://somehost.com", host: "10.26.20.220:4006"

Steps to Reproduce

enable http-logger globally

plugin config

{
"concat_method": "new_line",
"disable": false,
"include_req_body": true,
"include_resp_body": true,
"uri": "report server"
}

log_format config

{
"log_format": {
"streams": [
  {
    "stream": {
      "serviceName": "$service_name",
      "routeName": "$route_name",
      "host": "$host",
      "balancer": "$balancer_ip:$balancer_port"
    },
    "values": [
      [
        "$msec",
        "{\"ts\":\"$msec\",\"status\":\"$status\",\"request\":{\"ua\":\"$http_user_agent\",\"body-size\":\"$body_bytes_sent\",\"request\":\"$request\"},\"upstream\":{\"addr\":\"$upstream_addr\",\"host\":\"$upstream_host\",\"uri\":\"$upstream_uri\",\"status\":\"$upstream_status\",\"responseTime\":\"$upstream_response_time\"}}"
      ]
    ]
  }
]
}
}

do request to apisix

if enable plugin only, no log_format config, plugin can work correctly with default format, but when log_format is configured, an error will be reported

Environment

APISIX version (run apisix version): 2.13.0 and 2.14.1
Operating system (run uname -a): offical docker image
OpenResty / Nginx version (run openresty -V or nginx -V):
etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.14.8
APISIX Dashboard version, if relevant: 2.13
Plugin runner version, for issues related to plugin runners:
LuaRocks version, for installation issues (run luarocks --version):

tzssangglass commented 2 years ago

log_format config

Can you give the full command to configure log_format, the format you gave looks strange and I don't remember keywords like streams.

jinnyu commented 2 years ago

curl --location --request PUT 'http://a6:9080/apisix/admin/plugin_metadata/http-logger' \
--header 'X-API-KEY: 123456789' \
--header 'Content-Type: application/json' \
--data-raw '{
  "log_format": {
    "streams": [
      {
        "stream": {
          "serviceName": "$service_name",
          "routeName": "$route_name",
          "host": "$host",
          "balancer": "$balancer_ip:$balancer_port"
        },
        "values": [
          [
            "$msec",
            "{\"ts\":\"$msec\",\"status\":\"$status\",\"request\":{\"ua\":\"$http_user_agent\",\"body-size\":\"$body_bytes_sent\",\"request\":\"$request\"},\"upstream\":{\"addr\":\"$upstream_addr\",\"host\":\"$upstream_host\",\"uri\":\"$upstream_uri\",\"status\":\"$upstream_status\",\"responseTime\":\"$upstream_response_time\"}}"
          ]
        ]
      }
    ]
  }
}'

I want to send content in json format Should I escape all characters?

jinnyu commented 2 years ago

http-logger seems that cannot handle JSON content of nested types. But without config log_format, why plugin can process nested content correctly?

The log_foramt above is Loki's format, I want to config log_format to Loki's format, so that plugin can be send log to Loki directly without more process.

tzssangglass commented 2 years ago

The log_foramt above is Loki's format

I think that's the wrong format, you can take a look at: https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/http-logger.md

tokers commented 2 years ago

@jinnyu FYI:

Log format declared as key value pairs in JSON format. Values only support strings. APISIX or Nginx variables can be used > by prefixing the string with $.

jinnyu commented 2 years ago

Thank you!~ I missed that note.

Are there plans to support nested formats?

yujinchoi-94 commented 1 year ago

@tokers Hi, I also want to use http-logger to push log data to loki. Any plans with supporting nested json format?

apache / apisix