`StatefulJob` should be able to easily `tar` node data

[scirner22]

The StatefulJob docs state this, and I'm attempting to use it for the same function.

Strangelove uses it to compress and upload snapshots of chain data.

I'm having a problem achieving this though. On the Provenance chain our nonpruned nodes contain about 1TB of data. Our PVCs are setup for 1.25TB and to grow when they are at 90% used. With about 25% space overhead, there's not enough space to compress the data and store the tar.gz on the same volume.

The two ways I could think that the StatefulJob could support this would be the following:

• Allow configuration to specify additional PV/PVCs that are created/cleaned up
• Allow setting additional snapshot PV size. Then the PV is restored from the snapshot, it is then edited to increase its size further based on this config

[scirner22]

I can try tackling this if we settle on the solution.

[agouin]

We currently handle this by doing a streamed compress and upload so that storage is not necessary for the compressed file prior to upload.

For resumable uploads though, it would be great to have this feature so that the file is compressed once, and uploads can be retried.

We could add an additional parameter that would allow creating the StatefulJobs PVC with something like twice the size so that additional room was available for these kinds of operations.