Open scirner22 opened 3 months ago
I can try tackling this if we settle on the solution.
We currently handle this by doing a streamed compress and upload so that storage is not necessary for the compressed file prior to upload.
For resumable uploads though, it would be great to have this feature so that the file is compressed once, and uploads can be retried.
We could add an additional parameter that would allow creating the StatefulJobs PVC with something like twice the size so that additional room was available for these kinds of operations.
The StatefulJob docs state this, and I'm attempting to use it for the same function.
I'm having a problem achieving this though. On the Provenance chain our nonpruned nodes contain about 1TB of data. Our PVCs are setup for 1.25TB and to grow when they are at 90% used. With about 25% space overhead, there's not enough space to compress the data and store the
tar.gz
on the same volume.The two ways I could think that the StatefulJob could support this would be the following: