3JIou-home
commented
2 years ago Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request * If you are interested in working on this issue or have submitted a pull request, please leave a commentVector Version
vector 0.19.0 (x86_64-unknown-linux-gnu da60b55 2021-12-28)Vector Configuration File
[sources.http_REDACTED] data_dir = "/var/opt/vector/dbs" type = "file" include = ["/var/opt/logs-http-cf/json/REDACTED/*/*/*.json"] read_from = "beginning" ignore_older_secs = 43200 remove_after_secs = 1 oldest_first = true ignore_checkpoints = true fingerprint.strategy = "device_and_inode" [sinks.kafka_REDACTED] type = "kafka" inputs = [ "http_REDACTED" ] bootstrap_servers = "kafka-brokers-list" message_timeout_ms = 300_000 socket_timeout_ms = 60_000 topic = "REDACTED" compression = "gzip" encoding.codec = "text" buffer.max_events = 1000Debug Output
Expected Behavior
Vector periodically releasing memory
Actual Behavior
Vector memory usage increases overtime
Additional Context
Hello, we are migrating some simple data pipelines from logstash to vector, these pipelines just reads json files and send the events in the json to a kafka topic, the files are cloudflare http requests and firewall events logs, so we have something around 15 to 20 pipelines.
At the moment we migrated two of those pipelines using the configuration shared before, we are using one
.tomlfor each source, and it is working as expected, the only issue we find is that the memory usage of the vector process is increasing with the time and if we let the service running for a couple of days it will eventually consume all the memory on the server.The logs are collected using custom python scripts that only download the json files and put them on a folder for vector to consume, the python scripts are called using crontab each minute, vector runs as a systemd process.
Vector is the only service (besides the systems services) running on this server, the load and cpu usage is pretty low, the server runs on GCP and have 8 vCPUs and 8 GB (7.63 GB), with vector stopped the memory usage is around 900 MB, when we start vector the memory starts increasing and will only be released if we restart the vector service, we tried reload with
kill -1 PID, but it didn't have any effect.
References
did u try run vector with -qqq cli arguments? i have a strong feeling that this problem into internal log level
leandrojmp
jszwedko
Community Note
Vector Version
Vector Configuration File
Debug Output
Expected Behavior
Vector periodically releasing memory
Actual Behavior
Vector memory usage increases overtime
Additional Context
Hello, we are migrating some simple data pipelines from logstash to vector, these pipelines just reads json files and send the events in the json to a kafka topic, the files are cloudflare http requests and firewall events logs, so we have something around 15 to 20 pipelines.
At the moment we migrated two of those pipelines using the configuration shared before, we are using one
.tomlfor each source, and it is working as expected, the only issue we find is that the memory usage of the vector process is increasing with the time and if we let the service running for a couple of days it will eventually consume all the memory on the server.The logs are collected using custom python scripts that only download the json files and put them on a folder for vector to consume, the python scripts are called using crontab each minute, vector runs as a systemd process.
Vector is the only service (besides the systems services) running on this server, the load and cpu usage is pretty low, the server runs on GCP and have 8 vCPUs and 8 GB (7.63 GB), with vector stopped the memory usage is around 900 MB, when we start vector the memory starts increasing and will only be released if we restart the vector service, we tried reload with
kill -1 PID, but it didn't have any effect.References