logstash-plugins / logstash-output-s3

Apache License 2.0
58 stars 151 forks source link

unable to push kubernetes logs to s3 bucket #207

Open suveerrao opened 5 years ago

suveerrao commented 5 years ago

I am trying to send my kubernetes logs to s3 using logstash(v5.4.3) and logstash-output-s3 (5.0.7). logstash is creating logstash-programmatic-access-test-object but not the actual log files. Below are my input and output configurations. Am I taking a wrong input plugin ? please do suggest.

input {
  file {
    # kubelet created symlinks to docker container logs
    path => "/var/log/containers/*.log"
    tags => ["kubernetes", "docker"]
    sincedb_path => "/var/log/containers/sincedb-kubernetes"
  }
}

output {
  if "kubernetes" and "docker" in [tags] {
     s3{
      access_key_id => "XXXXXXXXXX"
      secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXXX"
      region => "us-east-1"
      bucket => "nonapp-logs"
      size_file => 2048
      time_file => 5
      prefix => "non-app-logs-%{+YYYY.MM.dd}"
      canned_acl => "private"
      server_side_encryption => true
    }
  }
}