Investigate growing disk space usage on a long-running node

[LesnyRumcajs]

Issue summary

It turned out we don't have a clear understanding on what should we expect from a long-running node disk usage given that garbage collection should be working fine now. Should it grow indefinitely? If yes, at what rate?

Sample case; given a 100 GiB forest node, how long can it run (assuming no other issues with chain/upgrades/etc) without interventation on:

• mainnet,
• calibnet.

If it cannot run indefinitely - why is that? What takes space in the database that required a restart and a new bootstrap? Can we potentially do better?

Outcomes:

• we understand how Forest behaves on both calibration network and mainnet with regards to disk usage growth, and can communicate it to external stakeholders,
• the numbers are documented, so that any deviation from them can be investigated and marked as potential regression.

Other information and links

[LesnyRumcajs]

Some questions that need to be answered as well, as outlined by @ruseinov

How much space a given object takes up stored in ParityDB (potentially with a comparison of uncompressed/compressed snapshot). How many keys are being deleted on each GC run - I'm working on that as we speak. Whether or not the GC actually manages to run or the node gets restarted somewhere along the way. That should be visible in logs, but it will be definitive with my latest change.