Closed jerome83136 closed 2 years ago
You should use the file input plugin without tail. It's tested against lot of log rotation implementation.
Regards,
Bertrand
On Tue, Apr 26, 2016 at 11:40 AM, jerome83136 notifications@github.com wrote:
Hi,
I have this kind of errors in my node-logstash logs:
[Tue, 26 Apr 2016 01:08:53 GMT] ERROR tail: /central_logs/input/prod/webservers/zp2web02/apache/prospect/access_FHM_log: file truncated
That seems to happen when I use the "tail" option in the file input plugin. My input files are Apache's access_logs and they are rotating at 00:00 (before, they were rotating at 03:00 am)
But the errors on noge-logstash's logs are happening at ~01:30 am
Here is the file input section of my node-logstash config file:
input { file { use_tail => true start_index => 0 path => '/central_logs/input/prod/webservers/zp2web0?/apache/prospect/access_FH?_log' type => 'prospect' } file { use_tail => true start_index => 0 path => '/centrallogs/input/prod/webservers/zp2web0?/apache/webshop/access*MALE_log' type => 'webshop' } }
Are these errors harmful ? Did I do something wrong ?
Many thanks for your help
Best regards
Jérôme
— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/125
OK, will try this tomorrow Thank you Le 26 avr. 2016 8:42 PM, "Bertrand Paquet" notifications@github.com a écrit :
You should use the file input plugin without tail. It's tested against lot of log rotation implementation.
Regards,
Bertrand
On Tue, Apr 26, 2016 at 11:40 AM, jerome83136 notifications@github.com wrote:
Hi,
I have this kind of errors in my node-logstash logs:
[Tue, 26 Apr 2016 01:08:53 GMT] ERROR tail:
/central_logs/input/prod/webservers/zp2web02/apache/prospect/access_FHM_log: file truncated
That seems to happen when I use the "tail" option in the file input plugin. My input files are Apache's access_logs and they are rotating at 00:00 (before, they were rotating at 03:00 am)
But the errors on noge-logstash's logs are happening at ~01:30 am
Here is the file input section of my node-logstash config file:
input { file { use_tail => true start_index => 0 path => '/central_logs/input/prod/webservers/zp2web0?/apache/prospect/access_FH?_log' type => 'prospect' } file { use_tail => true start_index => 0 path => '/centrallogs/input/prod/webservers/zp2web0?/apache/webshop/access*MALE_log' type => 'webshop' } }
Are these errors harmful ? Did I do something wrong ?
Many thanks for your help
Best regards
Jérôme
— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/125
— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/125#issuecomment-214844845
Hi, I have disabled "tail" in my file input plugin's configuration.
But after ~1 hour it seems my output files are not written anymore.
The last entries in my node-logstash logs are:
[Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web02/apache/webshop/access_MALE_log at 0 fd 22
[Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web03/apache/webshop/access_FEMALE_log at 0 fd 23
[Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web03/apache/webshop/access_MALE_log at 0 fd 24
[Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web04/apache/webshop/access_MALE_log at 0 fd 25
[Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web04/apache/webshop/access_FEMALE_log at 0 fd 26
It seems node-logstash just stops writing logs in my output files after a certain time.
What can I do ?
Thanks
Best regards Jérôme
Can you provide the config (with outputs) and some infos about the amount of data ?
On Wed, Apr 27, 2016 at 5:17 PM, jerome83136 notifications@github.com wrote:
Hi, I have disabled "tail" in my file input plugin's configuration.
But after ~1 hour it seems my output files are not written anymore.
The last entries in my node-logstash logs are:
[Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web02/apache/webshop/access_MALE_log at 0 fd 22 [Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web03/apache/webshop/access_FEMALE_log at 0 fd 23 [Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web03/apache/webshop/access_MALE_log at 0 fd 24 [Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web04/apache/webshop/access_MALE_log at 0 fd 25 [Wed, 27 Apr 2016 08:00:00 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web04/apache/webshop/access_FEMALE_log at 0 fd 26
It seems node-logstash just stops writing logs in my output files after a certain time.
What can I do ?
Thanks
Best regards Jérôme
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/125#issuecomment-215117314
Hi,
Here is the output section of the config file:
output {
if [tags] != '_grokparsefailure' {
file {
idle_timeout => 86400
path => '/central_logs/output/prod/webservers/#{type}/#{type}_#{p0x}.#{now:YYYYMMDD}.log'
}
}
}
There are ~ 2,3 Millions of logs lines every day.
Thanks for your help
Regards Jérôme
Hello, I have configured my input plugin to not use the "tail" mode. But, I can't get that configuration working more than few hours.
I start logstash, it reads the input log files from the beginning and it sends the output to the configured output files; good. But after some hours, node-logstash stops writing the output files.
In the node-logstash logs, there is no recent entry. the last entry is related to the node-logstash startup; when it starts reading input logs files:
[Tue, 24 May 2016 14:18:13 GMT] INFO Start reading /central_logs/input/prod/webservers/zp2web04/apache/webshop/access_FEMALE_log at 0 fd 28
Do you now what could be wrong ? a Thanks for your help Best regards Jérôme
Hello, After some investigations; it seems node-logstash stops writing output files at a very precise time: the time at which I have started it.
For example: if I start node-logstash today at 24/05/2016 16:18pm; it will read my input log files from the begining (~03:00am) and the last log entry on the output files will have a timestamp of 24/05/2016 16:18pm
So it seems node-logstash will read my input logs files but it will not follow new entries into these files.
Strangely when I change the file-input plugin to use the "tail" mode; I get an opposite behavior. node-logstash starts reading the input logs files from the current last line and will follow new lines. But it will not start reading the input log files from the beginning and so I will loose previous logs entries.
Do you now why I'm getting these behaviors ?
Thanks for your help
Best regards Jérôme
Hi,
I have this kind of errors in my node-logstash logs:
[Tue, 26 Apr 2016 01:08:53 GMT] ERROR tail: /central_logs/input/prod/webservers/zp2web02/apache/prospect/access_FHM_log: file truncated
That seems to happen when I use the "tail" option in the file input plugin. My input files are Apache's access_logs and they are rotating at 00:00 (before, they were rotating at 03:00 am)
But the errors on noge-logstash's logs are happening at ~01:30 am
Here is the file input section of my node-logstash config file:
Are these errors harmful ? Did I do something wrong ?
Many thanks for your help
Best regards
Jérôme