kyverno / policy-reporter

Monitoring and Observability Tool for the PolicyReport CRD with an optional UI.
https://kyverno.github.io/policy-reporter/
MIT License
263 stars 78 forks source link

Kyverno Policy Reporter writes extensively on node disk storage in certain ocassions. #443

Open juhis135 opened 1 month ago

juhis135 commented 1 month ago

We have noticed that the Kyverno Policy Reporter writes extensively on Node's disk storage in certain scenarios and we end up getting the below alert The node was low on resource: ephemeral-storage.

We want to identify the steps to debug this scenario of policy reporter writing extensively on node's disk and accordingly decide on the solution ?

fjogeleit commented 1 month ago

Hey, Policy Reporter only writes to disc when it updated its internal SQLite DB, which is file based. It could happen when a large amount of Policy Reports are updated/recreated in a short amount of time. Which leads to reprocessing each PolicyReport and updating the Database.

Do you using it in combination with Kyverno? If so, which Kyverno version do you using? Is it a larger cluster where this iossue happens?

On possible solution you can try right now is to use an alternative Database instead of the internal SQLite DB, supported DBs are Postgres, MySQL and MariaDB.