Accumulated archive xdr files are not removed

[b-yap]

:collision: DISK USAGE High :collision:

• reported constantly in aws, the past few weeks.

Possible reason are these files:

-rw-r--r--   1 root root  7012400 Jan 12 14:59 txs-02f92d3f.xdr
-rw-r--r--   1 root root   113313 Jan 24 12:00 02fbddbf.xdr
-rw-r--r--   1 root root  3377498 Jan 24 12:00 txs-02fbddbf.xdr
-rw-r--r--   1 root root   109698 Feb 21 10:49 0302167f.xdr
-rw-r--r--   1 root root  4221382 Feb 21 10:49 txs-0302167f.xdr
-rw-r--r--   1 root root   106607 Feb 21 11:58 0302197f.xdr
-rw-r--r--   1 root root  4539384 Feb 21 11:58 txs-0302197f.xdr
-rw-r--r--   1 root root   101380 Feb 28 14:42 0303a93f.xdr
-rw-r--r--   1 root root  4702975 Feb 28 14:42 txs-0303a93f.xdr
-rw-r--r--   1 root root    84272 Feb 29 12:18 0303e1ff.xdr
-rw-r--r--   1 root root  8330294 Feb 29 12:19 txs-0303e1ff.xdr
-rw-r--r--   1 root root   115431 Mar  1 00:56 0303feff.xdr
-rw-r--r--   1 root root  1695793 Mar  1 00:56 txs-0303feff.xdr
-rw-r--r--   1 root root   109342 Mar  8 20:33 0305b43f.xdr
-rw-r--r--   1 root root  6481192 Mar  8 20:33 txs-0305b43f.xdr
-rw-r--r--   1 root root    27242 Mar 10 17:19 0305cfff.xdr
-rw-r--r--   1 root root   102574 Mar 13 21:01 0306cc7f.xdr
-rw-r--r--   1 root root  6423701 Mar 13 21:01 txs-0306cc7f.xdr
-rw-r--r--   1 root root   102623 Mar 19 12:01 030802ff.xdr
-rw-r--r--   1 root root  6701406 Mar 19 12:01 txs-030802ff.xdr
-rw-r--r--   1 root root   110281 Mar 19 18:01 030813ff.xdr
-rw-r--r--   1 root root  5563178 Mar 19 18:01 txs-030813ff.xdr

These files are created whenever the vault has to build a proof via the archive. SCP: https://github.com/pendulum-chain/spacewalk/blob/b1dc81f9de7bca3408b490587280a10732f7b9dd/clients/vault/src/oracle/collector/proof_builder.rs#L237

TXSET: https://github.com/pendulum-chain/spacewalk/blob/b1dc81f9de7bca3408b490587280a10732f7b9dd/clients/vault/src/oracle/collector/proof_builder.rs#L326

where after downloading and extraction, these files are never removed: https://github.com/pendulum-chain/spacewalk/blob/b1dc81f9de7bca3408b490587280a10732f7b9dd/clients/vault/src/oracle/storage/traits.rs#L103-L104

TODO: remove the file after reading/storing to variable

[b-yap]

@pendulum-chain/product, @ebma this is an assessment from the report in #devops-incidents:

[[FIRING:1] DISK Usage High
**Firing**
Value: B=100, C=1
Labels:
- alertname = DISK Usage High
- grafana_folder = devops_folder
- job = Spacewalk

happened last week, and still occurring until today.

[prayagd]

@b-yap can we move this ticket to development as i see you are already working on it?

[ebma]

Nice find @b-yap 👌