We have deployed vault docker container in a docker-compose along with aws-for-fluent-bit image where we would like to store vault audit logs in a cloudwatch log group.
we have enabled vault audit logs for a filepath and fluentbit tails this file and send logs to cloudwatch loggroup.
Initially after we enable audit logs, we only see the logs where there is an activity in vault like a user generating credentials for a db or updating any configuration or login and logout.
After few hours we observed that vault logs were streamed to cloudwatch even if there is no activity, streaming is continuous like a flood with a "display_name": "token-terraform".
we don't have any token issued with token-terraform or any user with token-terraform.
with this issue we were unable to search the real audit events logs.
Here is some info about images
[ec2-user@ip-xx-xx-xx-xx hashicorp]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
28e8b6eefcd6 vault:latest "docker-entrypoint.s…" 3 days ago Up 26 hours 0.0.0.0:443->8200/tcp, :::443->8200/tcp hashicorp-vault-1
e4f1acec507f amazon/aws-for-fluent-bit:latest "/bin/sh -c /entrypo…" 3 days ago Up 26 hours 2020/tcp hashicorp-fluent-bit-1
[ec2-user@ip-xx-xx-xx-xx hashicorp]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
vault latest 7193130a202b 12 days ago 221MB
amazon/aws-for-fluent-bit latest d862dfcfa5b7 13 days ago 339MB
[ec2-user@ip-xx-xx-xx-xx hashicorp]$
cat docker-compose.yml
version: '3'
services:
vault:
image: vault:latest
restart: always
ports:
- 443:8200
volumes:
- ./vault/config:/vault/config
- ./vault/policies:/vault/policies
- ./vault/data:/vault/data
- ./vault/logs:/vault/logs
- ./vault/file:/vault/file
environment:
- VAULT_ADDR=https://vault.example.com
- VAULT_API_ADDR=https://vault.example.com
- VAULT_ADDRESS=https://vault.example.com
cap_add:
- IPC_LOCK
command: vault server -config=/vault/config/vault.json
fluent-bit:
image: amazon/aws-for-fluent-bit:latest
restart: always
volumes:
- ./vault/logs:/var/log/vault/logs
- ./fluent-bit/etc:/fluent-bit/etc/
Vault config:
cat vault.json
{
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_cert_file": "/vault/data/certs/server.crt",
"tls_key_file": "/vault/data/certs/server.key"
}
},
"backend": {
"file": {
"path": "/vault/file"
}
},
"default_lease_ttl": "168h",
"max_lease_ttl": "0h",
"api_addr": "https://vault.example.com:8200",
"ui": "true"
}
Fluent-bit config:
cat fluent-bit.conf
[SERVICE]
flush 5
log_Level info
[INPUT]
Name tail
Tag audit
Path /var/log/vault/logs/audit.log
Mem_Buf_Limit 8MB
Refresh_Interval 30
Parser json
[OUTPUT]
Name cloudwatch
Match audit
region eu-central-1
log_group_name /aws/some_log_group_name/var/log
log_stream_prefix vault_
auto_create_group true
log_retention_days 365
log_key log
We have deployed vault docker container in a docker-compose along with aws-for-fluent-bit image where we would like to store vault audit logs in a cloudwatch log group.
we have enabled vault audit logs for a filepath and fluentbit tails this file and send logs to cloudwatch loggroup.
Initially after we enable audit logs, we only see the logs where there is an activity in vault like a user generating credentials for a db or updating any configuration or login and logout.
After few hours we observed that vault logs were streamed to cloudwatch even if there is no activity, streaming is continuous like a flood with a "display_name": "token-terraform". we don't have any token issued with token-terraform or any user with token-terraform.
with this issue we were unable to search the real audit events logs.
Here is some info about images