Open ingoles opened 4 years ago
I changed configuration file as below but still no logs displayed from Startup.log. Still getting logs from td-agent.log on kibana dashboard.
<system>
log_level debug
</system>
<source>
@type tail
path /mnt/Log/Startup.log
pos_file /mnt/Log/Startup.log.pos
read_from_head true
<parse>
@type multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
</parse>
tag log
</source>
<match *log*>
@type elasticsearch
host localhost
port 9200
include_tag_key true
tag_key @log_name
logstash_format true
flush_interval 10s
</match>
Can you please elaborate what is wrong in configuration.? Why I am still not getting desired logs on kibana.
Actually it worked when I appended one line in Startup.log. Then all previously present data showed up on kibana dashbord. But again when my application was not appending any lines in log file at that time again it started showing only below line from td-agent.log continuously on kibana
"tailing paths: target = /mnt/Log/Startup.log | existing = /mnt/Log/Startup.log"
"tailing paths: target = /mnt/Log/Startup.log | existing = /mnt/Log/Startup.log"
I want to display only content from Startup.log. and when application stops appending logs into Startup.log, it should not show content from td-agent log file.
It there any way I can configure this? kindly inform.
I am facing one more issue while creating two different indices for two different sources. I will not create separate thread for this as it would be better to keep track in the same thread here.Below is my configuration file
<system>
log_level debug
</system>
<source>
@type tail
@log_level debug
path /mnt/appln/ncnsintusr/ncns/interface/sol-jms-test/file-loader-logs/Startup.log
pos_file /mnt/appln/ncnsintusr/ncns/interface/sol-jms-test/file-loader-logs/Startup.log.pos
#read_from_head true
<parse>
@type multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
</parse>
tag fileloader.log
</source>
<source>
@type tail
@log_level debug
path /mnt/appln/ncnsintusr/ncns/interface/sol-jms-test/fixml-gateway-logs/Startup.log
pos_file /mnt/appln/ncnsintusr/ncns/interface/sol-jms-test/fixml-gateway-logs/Startup.log.pos
#read_from_head true
<parse>
@type multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
</parse>
tag fixml
</source>
<match fileloader.log>
@type elasticsearch_dynamic
host localhost
port 9200
logstash_format true
include_timestamp true
logstash_prefix fileloader
logstash_dateformat %Y.%m.%d
time_key_format %Y-%m-%dT%H:%M:%S.%N%z
include_tag_key true
type_name access_log
target_index_key @target_index
#tag_key @log_name
flush_interval 1s
</match>
<match **>
@type elasticsearch
host localhost
port 9200
logstash_format true
include_timestamp true
logstash_prefix fixml
logstash_dateformat %Y.%m.%d
time_key_format %Y-%m-%dT%H:%M:%S.%N%z
#include_tag_key true
type_name access_log
#tag_key @log_name
flush_interval 1s
</match>
index for fixml is getting created as it has matches with pattern ** but fileloader index is not getting created though i have given match tag in correct sequence. Can someone reply to at least single query mentioned above. I am new to using td-agent and tried all possible configurations till now but no success.
I have installed EFK on redhat linux system(version 7.6). Stable distribution of fluentd, that is td-agent is used instead of fluentd. Path of log file(/mnt/Log/Startup.log) is configured in td-agent.conf file. But on kibana dashboard it displays content from td-agent.log instead of log file(Startup.log).
td-agent.log:
Though above log shows that is it tailing the Startup.log. Still it shows content from td-agent.log on kibana dashboard and not the content from configured log file.
td-agent.conf :
Version details:
Elasticsearch version 7.8.1 Fluentd (td-agent) version 3.8.0 Kibana version 7.8.1
I have installed all above tools using rpms:
elasticsearch-7.8.1-x86_64.rpm td-agent-3.8.0-0.el7.x86_64.rpm kibana-7.8.1-x86_64.rpm
curl -X GET "localhost:9200/?pretty" gives:
{ "name" : "ncnsidapp2", "cluster_name" : "elasticsearch", "cluster_uuid" : "MnfFXTyZT0ahp9u4eLSW2A", "version" : { "number" : "7.8.1", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89", "build_date" : "2020-07-21T16:40:44.668009Z", "build_snapshot" : false, "lucene_version" : "8.5.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }
What could be the issue here? Please help.