Consider: Lazy Collection Projections/Mapping

[ecton]

@colelawrence suggested a use case for a lazily-evaluated collection, which act sort of like a View but with different re-caching behaviors.

Imagine this common use case: you have uncompressed data stored in the database and you want to compress it before sending it over the network. If you were to create a view that stores the compressed representation as values, you have two problems:

• Nebari has a limit to the size that the embedded value can be. (Not enforced until #76)
• When a view is updated, all invalidated entries are updated. There is no way for the view system to know that a particular document that is invalidated won't emit the key being queried, and the only way for it to know it is to process the documents.

For data where it isn't changing often and all data is expected to be requested, the View system is perfectly find for smaller chunks of data.

This request is to consider creating a lazily evaluated collection:

• The primary key is the dependent collection's primary key
• When an entry is requested, a check is done to verify if the original document hasn't changed.
  • If any have changed, the contents will be updated according to the AccessPolicy
  • If the document has changed, the mapping function will be called.
  • The mapping function should be able to decide not to update the record if nothing relevant has changed.
• The collection's contents are returned if found.

While one benefit of this collection is that it can be "individually" lazy, I think the fact a mapping function can optionally decide not to update the stored contents makes it a candidate for building Views atop. I see no reason it can't be made to be performant.

[colelawrence]

RE: The mapping function should be able to decide not to update the record if nothing relevant has changed -> For the majority of cases, I think maybe defining a custom hash function might suffice for checking whether invalidation is necessary. This way, you don't necessarily have to keep the prev value for checking at all, you can just call the custom hashing value and compare that to the hash stored against the document for that view.

Perhaps there are multiple strategies for invalidation, which could have interesting implications, like:

trait InvalidationStrategy<DocumentContent> {
  /// To be stored when cache invalidated
  ///
  /// hmm: Is there an interest in storing a small buffer of say 3 item HashMap<CacheValue, ViewValue> so
  /// you can use previous computations?
  type CacheValue: Serialize + Deserialize;
  /// Lazy? – upon pull of view value?
  fn map_value(content: CollectionDocument<DocumentContent>) -> CacheValue;
  /// Returning true indicates that stored View value is good enough?
  fn is_valid(previous: CacheValue, next: CacheValue) -> bool;
}

[ecton]

I agree that what I wrote it a little too simplistic -- purposefully so. I didn't want to narrow the implementation to a specific approach yet. I think it's hard to know what the ergonomics will be without playing with the feature a little bit.

I agree that a hash-based implementation should be possible. But, other implementations that don't rely on hashes should be possible too -- that's why I didn't want to call out a specific approach to how to solve allowing a mapping function to determine if there are changes or not. The requirement that it must be possible is documented, but the exact approach is still to be determined.