Open nabokihms opened 10 months ago
nabokihms
commented
10 months ago Probably, this is due to the vector not graceful process termination. Maybe it is better to switch to the time-based locking for vector: set the lock for, for example, 5 minutes and try to renew it every second minute. If the lock is not renewed, the vector process is allowed to take it.
tobz
commented
10 months ago Yep, this is definitely a limitation and something that isn't handled terribly well with non-graceful shutdown.
We would likely want to make it work somewhat more like a PID file, where any Vector process that wants to take the lock will confirm that, if the lock file already exists, that the PID referenced within it is still running... otherwise, it would delete the lock file and take it over.
(Just to comment on it: I would want to avoid a time-based solution because I don't want another process trying to take over the buffer if there was a simple bug with the background task responsible for updating lock liveness. If there's a stalled Vector process, something might still be happening to the disk buffer, so we can only be sure that nothing is touching it if the PID itself is gone: process not running, can't possibly be modifying the disk buffer.
nabokihms
commented
10 months ago Fantastic idea, thanks for clarifying, @tobz.
How will it work in a container? Will vector need access to the host pid namespace?
nabokihms
commented
10 months ago I assume the lock is not required for the vector running in the Kubernetes cluster because the DaemonSet controller can handle that two pods are not running on the same node. As a workaround, in our vector installation, we will remove the lock in the entrypoint every time the container starts.
And, of course, it would be nice to finalize the design for the feature to implement advanced locking capabilities in vector.
dsmith3197
commented
8 months ago I was able to verify that this occurs when removing and then re-adding a sink with a disk buffer and was able to reproduce the behavior through the following sequence:
VECTOR_SELF_NODE_NAME=abc VECTOR_SELF_POD_NAME=def vector --config config.jsondestination/cluster/loki-ebapp to destination/cluster/loki-ebapp-1 in the config.kill -HUP $(pgrep vector).destination/cluster/loki-ebapp-1 to destination/cluster/loki-ebapp in the config.kill -HUP $(pgrep vector).This results in
2023-12-12T16:56:04.063075Z ERROR vector::topology::builder: Configuration error. error=Sink "destination/cluster/loki-ebapp-1": error occurred when building buffer: failed to build individual stage 0: failed to load/create ledger: failed to lock buffer.lock; is another Vector process running and using this buffer?Notably, this behavior occurs when the sink is loki (and also clickhouse) but not when the sink is blackhole.
{
"data_dir": "./vector-data",
"sources": {
"cluster_logging_config/ebapp:ebapp": {
"type": "demo_logs",
"format": "json"
}
},
"sinks": {
"destination/cluster/loki-ebapp": {
"type": "loki",
"inputs": [
"cluster_logging_config/ebapp:ebapp"
],
"healthcheck": {
"enabled": false
},
"buffer": {
"max_size": 268435488,
"type": "disk",
"when_full": "drop_newest"
},
"encoding": {
"only_fields": [
"message"
],
"codec": "text",
"timestamp_format": "rfc3339"
},
"endpoint": "http://loki.d8-monitoring:3100",
"tls": {
"verify_hostname": true,
"verify_certificate": true
},
"labels": {
"container": "{{ container }}",
"host": "{{ host }}",
"image": "{{ image }}",
"namespace": "{{ namespace }}",
"node": "{{ node }}",
"pod": "{{ pod }}",
"pod_ip": "{{ pod_ip }}",
"pod_labels_*": "{{ pod_labels }}",
"pod_owner": "{{ pod_owner }}",
"stream": "{{ stream }}"
},
"remove_label_fields": true,
"out_of_order_action": "rewrite_timestamp"
}
}
}
This is a bug that we should resolve.
dsmith3197
commented
8 months ago We would likely want to make it work somewhat more like a PID file, where any Vector process that wants to take the lock will confirm that, if the lock file already exists, that the PID referenced within it is still running... otherwise, it would delete the lock file and take it over.
Regarding this approach, the fslock crate leverages flock (ref) which automatically releases the lock when all file descriptors are dropped. As such, our implementation already handles non-graceful shutdowns.
nabokihms
commented
7 months ago @dsmith3197 thanks for the info. Do you have any ideas why this problem occurs when the sink is Loki?
jszwedko
commented
7 months ago @dsmith3197 thanks for the info. Do you have any ideas why this problem occurs when the sink is Loki?
What we were able to reproduce is that Vector doesn't correctly release disk buffer locks during reload when components are removed so that if they are re-added it'll fail to lock. Does that match how you ran into this issue? Dropping a component, reloading, readding a component with the same name, reloading?
nabokihms
commented
7 months ago In my case a sidekick application composes a config file for vector and reloads it. A sink with the same id can removed and added multiple times, so yes. I think this is my case. We rely much on reloading.
jszwedko
commented
7 months ago In my case a sidekick application composes a config file for vector and reloads it. A sink with the same id can removed and added multiple times, so yes. I think this is my case. We rely much on reloading.
👍 thanks for confirming! I don't think it is specific to the loki sink then. You'd see the same error for any sinks using disk buffers if they are removed and re-added via vector reload.
A note for the community
Problem
When I try to reload my vector configuration, it says in the log:
It happens before there is a lock file in the directory created three days ago.
No other vector processes are running.
Configuration
Version
v0.32.0
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response