search

shinsaka / fluent-plugin-elb-log

Amazon ELB log input plugin
MIT License
39 stars 23 forks source link

ELB logs not reflecting in ElasticSearch for a custom index. #16

Open paritosh16 opened 9 years ago

paritosh16 commented 9 years ago

I have the following configuration for grabbing ELB logs from S3.

<source>
  type elb_log
  access_key_id     
  secret_access_key 
  region            
  s3_bucketname    
  s3_prefix         
  timestamp_file   
  buf_file          
  refresh_interval  
  tag               
</source>

I have defined the mapping for the ELB logs in elasticSearch. The plugin doesn't process the ELB logs for a custom index that I define. The log statements like Processing 0 objects reflect into ES.

To be honest, I really don't know if this is a issue at ES level or plugin level. The plugin logs are getting into ES but ELB logs aren't. Thank you for the help.

shinsaka commented 9 years ago

Hi,

I'm not sure about ElasticSearch plugin. Can you get fluentd's log some information?

paritosh16 commented 9 years ago

These are the fluentd's logs that are getting into ElasticSearch. 

2015-10-13 02:38:42 +0000 [info]: processing 0 object(s).
2015-10-13 02:43:42 +0000 [info]: processing 0 object(s).
2015-10-13 02:48:42 +0000 [info]: processing 0 object(s).
2015-10-13 02:53:42 +0000 [info]: processing 0 object(s).
2015-10-13 02:58:42 +0000 [info]: processing 0 object(s).
2015-10-13 03:03:42 +0000 [info]: processing 1 object(s).
2015-10-13 03:08:43 +0000 [info]: processing 0 object(s).
2015-10-13 03:13:43 +0000 [info]: processing 0 object(s).
2015-10-13 03:18:43 +0000 [info]: processing 0 object(s).

I can see nothing in the logs that suggest me that fluentd is failing. But I think it is because fluent logs are getting through and the ones from S3 aren't.

martinssipenko commented 6 years ago

I have similar issue, and I can say that this is because of custom mappings and very much depends on mapping itself.