Support for storage size increment / decrement

pigmej commented 9 months ago

There are a few cases when storage size can be changed. https://github.com/spacemeshos/pm/issues/209 added support for PoST size changes on the protocol level. This ticket outlines the requirements to the implementation of PoST size changes.

When implementing PoST size changes the upcoming changes to PoET (phased PoET) should be considered: https://github.com/spacemeshos/pm/issues/215

Manual Resizing

Decrease

The user should be able to decrease their reported storage on request. It should become effective with the publication of the next ATX, i.e. if the user decreases their PoST in epoch N it shall become effective in epoch N+1.

The user should have a way to keep the original files without deleting them
The implementation for decreasing PoST should allow for a "graceful exit" (i.e. decreasing to 0) - see also https://github.com/spacemeshos/go-spacemesh/issues/3775)
The node may be required to restarted to make the change effective, but shall not stay offline for longer than a few minutes.

Increase

PoST increases will go into effect in epoch N+2 (where epoch N is the epoch where the initialization of the additional PoST finished). The ATX published in epoch N will "announce" the size increase while the ATX in epoch N+1 proves it. This means by epoch N+2 a smesher is first able to receive rewards for the increased size. This similar to how in epoch 0 all Smeshers initialize their post and in epoch 2 they are first eligible for rewards.

User should be able to initialize more storage via postcli manually
The risk of adding too much storage is on the user side
Node should see the change in the postdata_metadata.json (or similar) and behave accordingly.
- It is OK to require a node restart for the change to go into effect (e.g. while the node is offline postcli moves the files to the correct location and updates the metadata).
Smapp should have support for that functionality, preferably on the Post edit screen.
- That requires the node to be able to increase post size as well and it should be implemented such that both post and postcli can utilize the functionality.
- It is OK to require a node restart for the change to go into effect
NOTE: PoST increases will usually require the publication of a new VRF Nonce!

Automatic resizing

(mafa: should be its own ticket)

Detecting data corruption

The PoST module and postcli need to be extended to be able to assess the health of PoST data. This that includes, but is not limited to:

detecting files damaged files: skipping them during proof generation and informing the user about which file(s) are corrupted so they can take action (regenerate or delete).
gracefully handling missing files: skipping them during proof generation
handling disk failures when reading data during proof generation gracefully

Manual decrease and increase of PoST should be implemented before data corruption detection, but the implementation for resizing should already consider the impacts of it.

Automatic decrease when not enough time is left with cycle-gap

That requires coordination with POET and plausible https://github.com/spacemeshos/pm/issues/215 It is also possible that the user started the node too late etc. There are two ways:

We could decrease reported storage when the node notices that it cannot generate proof on time
- How to detect?
- How to notify the user?

For simplicity, we should focus on that as the last one.

fasmat commented 9 months ago

Updated description. I think we should split this into 2 (maybe 3) epics:

Allow User to resize PoST (requirements clear)
Resize on data corruption detection (requirements mostly clear)
Optimize published PoST size to proofing Window (requirements tbd.)

robandfender commented 9 months ago

A very anticipated feature, I hope Smapp Ver1.1.0 will launch this feature

spacemeshos / pm