filebeat import nginx json log errr

[macaty]

Which OpenObserve functionalities are the source of the bug?

ingestion

Is this a regression?

Yes

Description

1. config file [root@BA-xx filebeat]# more testfilebeat.yml setup.ilm.enabled: false setup.template.enabled: true setup.template.name: "nginx-log" setup.template.pattern: "nginx-log-*" setup.template.overwrite: true

filebeat.inputs:

• type: log enabled: true paths:
  • /home/xxx/nginx/logs/http.wgxxljobadmin.xxx.cn.access.log

output.elasticsearch: hosts: ["http://172.31.xx.50:5080"] timeout: 10 path: "/api/default/" index: default username: "root@example.com" password: "xxxx"

2. log example

[root@BA-xxx filebeat]# more /home/xxx/nginx/logs/http.wgxxljobadmin.xxx.cn.access.log {"@timestamp":"2023-08-04T00:00:01+08:00","remote_addr": "172.31.37.134", "upstream_addr": "172.31.37.131:36051", "remote_user": "", "host": "wgxxljobadmin.xxx.cn", "time_local": "04/Aug/2023:00:00:01 +0800", "ms ec": "1691078401.337", "request": "POST /xxl-job-admin/api/callback HTTP/1.1", "status": 200, "body_bytes_sent": 49, "http_referer": "", "http_user_agent": "Java/1.8.0_212", "http_x_forwarded_for": "","http_cookie": "", "request_time": 0.006, "upstream_response_time": "0.006", "upstream_cache_status": "","http_Cdn_Tag": "" } {"@timestamp":"2023-08-04T00:00:04+08:00","remote_addr": "172.31.37.135", "upstream_addr": "172.31.37.132:36051", "remote_user": "", "host": "wgxxljobadmin.xxx.cn", "time_local": "04/Aug/2023:00:00:04 +0800", "ms ec": "1691078404.027", "request": "POST /xxl-job-admin/api/callback HTTP/1.1", "status": 200, "body_bytes_sent": 49, "http_referer": "", "http_user_agent": "Java/1.8.0_212", "http_x_forwarded_for": "","http_cookie": "", "request_time": 0.008, "upstream_response_time": "0.008", "upstream_cache_status": "","http_Cdn_Tag": "" } {"@timestamp":"2023-08-04T00:00:09+08:00","remote_addr": "172.31.37.141", "upstream_addr": "172.31.37.133:36051", "remote_user": "", "host": "wgxxljobadmin.xxx.cn", "time_local": "04/Aug/2023:00:00:09 +0800", "ms ec": "1691078409.419", "request": "POST /xxl-job-admin/api/registry HTTP/1.1", "status": 200, "body_bytes_sent": 49, "http_referer": "", "http_user_agent": "Java/1.8.0_212", "http_x_forwarded_for": "","http_cookie": "", "request_time": 0.004, "upstream_response_time": "0.004", "upstream_cache_status": "","http_Cdn_Tag": "" }

Please provide a link to a minimal reproduction of the bug

No response

Please provide the exception or error you saw

3. filebeat got an error
2023-08-04T23:36:31.294+0800    ERROR   [publisher_pipeline_output] pipeline/output.go:154  Failed to connect to backoff(elasticsearch(http://172.31.38.50:5080/api/default/)): Connection marked as failed because the onConnect callback failed: error loading template: failure while checking if template exists: 401 Unauthorized: Unauthorized Access
2023-08-04T23:36:31.294+0800    INFO    [publisher_pipeline_output] pipeline/output.go:145  Attempting to reconnect to backoff(elasticsearch(http://172.31.xx.50:5080/api/default/)) with 3 reconnect attempt(s)
2023-08-04T23:36:31.294+0800    INFO    [publisher] pipeline/retry.go:219   retryer: send unwait signal to consumer
2023-08-04T23:36:31.294+0800    INFO    [publisher] pipeline/retry.go:223     done
2023-08-04T23:36:31.295+0800    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.17.1

Please provide the version you discovered this bug in (check about page for version information)

filebeat version 7.17.1
openobserve verstion 0.5.1

Anything else?

No response

[macaty]

if use the exact root password instead of token , will got another error : 2023-08-05T00:35:12.087+0800 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://172.31.38.50:5080/api/default/)): Connection marked as failed because the onConnect callback failed: error loading template: failure while checking if template exists: 404 Not Found:

[hengfeiyang]

i checked with the versions:

filebeat version 7.17.1
openobserve verstion 0.5.1

It works in my local, can you clean data and try again? if it still report error, can you give more logs?

[macaty]

thx，i change another version of filebeat（filebeat-8.9.0），and it works。