open-chat-labs / open-storage

GNU General Public License v3.0
35 stars 9 forks source link

Store blob data in stable memory #133

Closed hpeebles closed 1 year ago

hpeebles commented 1 year ago

This PR moves all of the blobs into stable memory.

This has 2 main advantages. 1 - This allows us to store far more data within each canister since stable memory is currently 32GB vs 4GB heap memory (and the stable memory limit is set to increase further). 2 - Upgrades will be much quicker and cheaper since we no longer need to copy this data to and from stable memory during upgrades.

hpeebles commented 1 year ago

The implementation generally LGTM.

One high-level question is, have you tested this end-to-end? One critical thing that's missing is making sure your pre_upgrade hook doesn't write to stable memory at arbitrary locations, otherwise that will correct the stable structures. You'll need to use the MemoryManager there as well. See the pre_upgrade hook of the bitcoin canister for reference.

Right ok will do so.

The testing I've done so far is the following -