Closed b-yap closed 4 months ago
@pendulum-chain/product, @ebma this is an assessment from the report in #devops-incidents:
[[FIRING:1] DISK Usage High
**Firing**
Value: B=100, C=1
Labels:
- alertname = DISK Usage High
- grafana_folder = devops_folder
- job = Spacewalk
happened last week, and still occurring until today.
@b-yap can we move this ticket to development as i see you are already working on it?
Nice find @b-yap 👌
:collision: DISK USAGE High :collision:
Possible reason are these files:
These files are created whenever the vault has to build a proof via the archive. SCP: https://github.com/pendulum-chain/spacewalk/blob/b1dc81f9de7bca3408b490587280a10732f7b9dd/clients/vault/src/oracle/collector/proof_builder.rs#L237
TXSET: https://github.com/pendulum-chain/spacewalk/blob/b1dc81f9de7bca3408b490587280a10732f7b9dd/clients/vault/src/oracle/collector/proof_builder.rs#L326
where after downloading and extraction, these files are never removed: https://github.com/pendulum-chain/spacewalk/blob/b1dc81f9de7bca3408b490587280a10732f7b9dd/clients/vault/src/oracle/storage/traits.rs#L103-L104
TODO: remove the file after reading/storing to variable