Motoko Data Inspection

[luc-blaeser]

Motoko Data Inspection

Note: This is still a prototype, not yet ready for merging.

Generic data inspection of Motoko canisters by authorized users.

Your Motoko Program is a Database!

This is only a small first step towards the bigger vision of providing data tooling to Motoko, similar to a database management system. This would support data inspection, data queries, maybe even data modification, data backup/restore, complex data migration, and/or administrative functionality.

Frontend Prototype

A simple frontend is available to test the data inspection, see https://github.com/luc-blaeser/data-inspector (limited access).

Backend Design

The Motoko runtime system is extended to stream the heap state to the frontend canister for displaying the data to authorized users.

Currently, the following design aspects apply to the data inspection in the Motoko runtime system:

• Only a controller can inspect the heap state over this functionality.
• The inspection is based on enhanced orthogonal persistence to profit from the precise object tagging.
• The data inspection returns the set of live objects (stable and flexible) reachable from the main actor.
• The format is mostly a one-to-one binary map of the heap object payload to minimize processing in the backend.
• For simplicity, currently, only full-heap inspection is supported. Larger scalability can be implemented later, see thoughts in .
• Currently, the field names are not yet shown. This can be supported later.

Binary format (EBNF)

Format = Version Root Heap.
Version = `1: usize`.
Root = `object_id: usize`.
Heap = `object_id: usize` `object_tag: usize` `object_payload:

The object payload is organized as follows:

• The regular RTS object payload, with pointers replaced by object ids.
• The payload is always a multiple of the word size.
• For Object, the object size is prepended because the hash blob cannot directly be located in the stream.

object_id are potentially synthetic identifiers of an objects. The ids are skewed, to distinguish them from scalars. Currently, the object_id are heap pointers but this would change with incremental inspection.

usize is 64-bit little endian.

Implementation

• Currently, a separate mark bitmap is used for heap traversal during inspection. This bitmap is independent of the potentially other bitmaps used during incremental GC.
• For arrays, the tag can not be copied one-to-one from the heap object as it may temporarily store slicing information stored during the incremental GC.
• As usual, forwarding pointers of the incremental GC need to be resolved during heap inspection.
• A separate mark stack is needed during heap inspection. This stack additionally stores the array slicing information of the heap inspection, independent of the incremental GC.
• A simple stream buffer is used to serialize the binary result of the heap inspection. The buffer is represented as a linked list of blobs that is finally copied to a combined single blob. This is because the size of the live set is not known in advance.

Future: Incremental Inspection

Incremental inspection can be realized in the future for scalability to larger heaps:

• It enables chunked data downloads in multiple messages without blocking other user messages. This is particularly important because the message response size is limited.
• It establishes a logical session where the client receives incremental heap changes without needing to refetch the full heap.

Possible implementation:

• Synthetic object ids need to be used that are independent of the address of the object. This is because objects are moved by the GC.
• A hash map can be used to map heap pointers to synthetic object ids. This map also serves for marking during heap traversal, such that a mark bitmap would no longer be needed. The pointers in the map are weak pointers that are updated by the GCs but will be removed from the map if the object is collected.
• On chunked data downloads, the object state can only be sent if all their contained pointers have been traversed. Otherwise, their state need to be transmitted on a subsequent download message.
• A pending list record the objects which state is ready to be sent in a next download. The pointers in the pending list are weak.
• Write barriers need to be extended to catch all mutator writes to pointers and scalars during a heap inspection session. The pointers of modified objects are recorded in a hash set, similar to the remembered set of the generational GCs. Again the pointers are treated as weak pointers.
• On incremental inspection, the runtime system resends the state of modified objects of the hash set in addition to a potentially next heap chunk of the pending list. The hash set is eventually cleared and the sent objects are removed from the pending list.
• On the client side, the object graph is updated for each resent object, while new objects are added.

[github-actions[bot]]

Comparing from 247aa056afe9d5499f9f79d78d6d38117b8478ae to 53bcc7514dd2f85154196f2c79aa1d89c7cec3c3: In terms of gas, 3 tests regressed and the mean change is +0.0%. In terms of size, 5 tests regressed and the mean change is +0.4%.

[crusso]

Very cool!

[crusso]

I wonder if leb encodings would compress stuff quite a bit...