Open belo4ya opened 1 month ago
@elezar, @klueska, @ArangoGutierrez, please take a look at this
Thanks @belo4ya, I have created #899 to add this option and we can continue this discussion there.
@ArangoGutierrez one thing that I noted is that we don't do any cleanup when the NodeFeatureAPI is used. How are labels removed in this case?
Issue description
We use the node-feature-discovery and gpu-feature-discovery features to monitor GPU issues, including cases when the number of available GPUs on a node unexpectedly decreases:
Target Number == nvidia.com/gpu.count == Node Allocatable
.We have noticed that sometimes after restarting gpu-feature-discovery, all the features (labels
nvidia.com/*
) exported by gpu-feature-discovery disappear from the node for a period roughly equal to the nfd-workersleepInterval
(in our case, 1 minute). This causes false positives in our monitoring system.We found that this occurs because gpu-feature-discovery deletes the
features.d/gfd
file before terminating if it is not running in one-shot mode (done using the removeOutputFile function).This behavior is very inconvenient (and undesirable) for us, especially when updating the gpu-feature-discovery version in the cluster.
Feature request
I found that this behavior was added with this commit - https://github.com/NVIDIA/gpu-feature-discovery/commit/bc91c4aec84c2bc3e6da47789d6d0a0326330455. However, I did not find an associated Issue justifying the need for this behavior.
Could you please consider:
--no-cleanup-on-exit
).An argument for 2. could be that node-feature-discovery does not do this. Instead, it uses a prune-job.