Message Instruction limit hit when inserting entity records

ByronBecker commented 2 years ago

Issue

When deployed to the IC main network, inserting entity records can hit the message instruction limit, causing the canister to trap and prevents further insertion.

Why is this happening

The when storing small Entity attributes & attribute values associated with keys, the Motoko Garbage Collector (GC) will eventually hit the message instruction limit at various times, on a scale between 250MB-2GB depending on the size of the attributes inserted (from Bytes - 1.5MB per entity) in my experience. The Motoko team is aware of this issue. See related issue - https://github.com/dfinity/motoko-base/issues/390

In-progress work being done to fix this issue

The release of Deterministic time slicing will allow GC to span multiple rounds of consensus, eliminating this issue and the CanDB heapSize limits.
Completed this test, imperative helps but still hits GC (Did a complete re-write of the underlying Red-Black tree structure in an imperative (non-functional) manner)
Eventually the underlying CanDB data structure will be migrated to a BTree, for which a Motoko library is currently being developed by the Motoko team at DFINITY.

Recommended stop-gap measure

When testing locally, this issue does not exist.

If deployed to the IC, it is recommended to set the following ScalingOptions, for sizeLimit using the #heapSize variant with a size of less than 200MB.

// IndexCanister.mo

// in the IndexCanister actor, when creating or upgrading a canister
scalingOptions = {
        autoScalingHook = autoScaleHelloServiceCanister;
        sizeLimit = #heapSize(200_000_000); // Scale out at 200MB
};

jzxchiang1 commented 2 years ago

Interesting! I wasn't aware that the Motoko team is currently developing a BTree (I'm guessing the analogue of the StableBTree written in Rust). I couldn't find mention of it in this thread...

ByronBecker commented 2 years ago

Update: This issue will be resolved upon the release of Deterministic Time slicing. DFINITY has estimated this feature is on the order of weeks away (pending blockers), so hopefully it will be released at some point in October/November. Additionally, the Motoko team has hired a GC specialist to help resolve some of the performance issues involved with the current GC in favor of a more incremental GC.

ByronBecker commented 2 years ago

Update: 2X DTS is currently live. After testing, one can safely insert up to 475 MB before hitting GC limitations. DFINITY is hopeful that 6-10X should fully resolve any issues related to the GC and allow canisters to store close to 4GB in heap memory without issue.

My test results of before vs. with 2X DTS - https://forum.dfinity.org/t/deterministic-time-slicing/10635/55

ByronBecker commented 1 year ago

Update: 4X DTS has been live for some time now.

Canisters can now insert up to 900MB heap memory safely with CanDB.

ORIGYN-SA / CanDB