Several design questions have arisen and now seems like a timely point to consider a number of MAP Descriptors design questions that have surfaced during our proof-of-concept efforts to date.
NOTE: This is a design-only activity whose sole deliverables are documentation updates... any code changes will be initiated under separate issues.
In the current (not yet implemented) design, MAP RelationshipDescriptors are bi-directional. The TypeHeader and references to source and target HolonTypes are stored in the RelationshipDescriptor and then details for each direction of the relationship are stored in two RelationshipRole objects.
Under this design, SmartLinks created for this relationship need to encode both the RelationshipDescriptors id as well as the role_name in the LinkTag. Although not strictly necessary, our design includes storing a direction byte as well.
Furthermore, holochain already uses the term role for a very different purpose as part of their capability model.
If Relationships were uni-directional, we would only need to store the relationship descriptor id in SmartLinks. And we avoid the naming conflict with holochain.
Decision
Change the design so that RelationshipDescriptors are uni-directional. Eliminate RelationshipRole. To represent (and support navigation via) both directions, two RelationshipDescriptors will be required. Each such descriptor will include an inverse_of reference to its inverse descriptor.
Consider unifying all descriptors into one EntryType #
The current design has different EntryTypes for HolonDescriptor and PropertyDescriptor and is on a trajectory to add RelationshipDescriptor and AffordanceDescriptor.
We could unify all of these in a single TypeDescriptor EntryType with a DescriptorDetails enum to handle the various sub-type specific parts. Perhaps these details nest (e.g., ValueType).
Decision
After exploring this approach via UML diagrams, I have decided AGAINST this unification. Having a single type makes it more difficult to express type restrictions. For example, properties should ONLY be allowed ValueTypes. But if everything is a TypeDescriptor, how do I express this restriction? I might be able to make this work with some more Rust trickery, but this would just increase the magnitude of the re-factoring effort.
[x] Decision Made (NOT to attempt this unification)
A HolonSpace is a container for holons and maps 1:1 with a DHT. It also proxies inbound and outbound calls to other HolonSpaces.
A HolonSpace is an L0 foundation for AgentSpaces in L1. Specifically, a HolonSpace serves some of the functions of I-Spaces and the outbound and inbound proxies play roles similar to We-Space. The challenge for the prototype is to discover the minimum viable functionality for HolonSpaces in the prototype to demonstrate cross-DHT references, with minimal scope creep.
A few questions (and answers):
What is in-scope for the prototype?
A: Enough to support external holon references -- punting on security/authorization. For now, putting implementation in map-holons repo, but eventually (the We-Space aspects) will need to be moved to separate happ.
Should HolonSpaces (and OutboundSpaceProxy and InboundSpaceProxy) be distinct EntryTypes, or just types of Holons?
A: Gonna try to do this with Holons
4 How important is resilience (e.g., multiple listeners) to inter-space execution
A: might play with this some... design affords multiple listeners, but won't go much into adaptive salience
How much security (if any) should we provide? Inbound validation of requests?
A: Almost none
Should we "hard-code" fetch calls for external references or use Dances?
A: will take a stab at leveraging Dances, but may fall back to hard-coding function calls
Decide on HolonReference design for local and external references #
The need to express references to another object arises in multiple guises within the MAP. For example, in Holon Relationships whose "to- cardinality" is [0..1] or [1], we just want to store the reference to the related holon.
How should we represent such references?
Requirements
Support local references -- i.e., to holons that reside in the SAME DHT as the referring holon.
Support external references -- to holons that reside in a DIFFERENT DHT as the referring holon. Note that the different DHT does NOT necessarily imply different DNA.
Support direct relationship targets -- in these cases the Holon Reference is stored within the Holon's Entry (not in a Link)
Support SmartLink relationship targets (i.e., SmartLinks) -- in these cases the Holon Reference is stored within a SmartLink.
Support typed references -- allow the expected type of the referenced holon to be specified to support (at least, run-time) type checking
Support OpenAPI-style type references -- allow AnyOf, OneOf variants on expected types.
Support versioned references -- where holons support semantic versioning, support the ability to update to new versions of holons per a _versionupdate policy.
Support foreign references (FUTURE) -- to objects that reside in a different systems (i.e., outside the MAP). Note that the method for identifying such objects is specific to the foreign system being referenced.
Design Considerations
ActionHash vs EntryHash as HolonId
EntryHashes are not guaranteed to uniquely identify a single version of an object. So we will use the holon's ActionHash as its unique identifier within its HolonSpace. A type alias (pub type HolonId = ActionHash) is defined to make the intent clear (and to provide some encapsulation of our design choice here).
Decision: Use ActionHash for Local References. EntryHashes are not guaranteed to be unique.
Holochain Resource Locators (HRL)
In a DM with Guillem, he discussed the use of Holon Resource Locators (hrl) to refer to objects in external DHT's. This concept was added to the We app. Hrl's use a syntax similar to URL's: hrl://DNAHASH/ANYDHTHASH. In this design, ANYDHTHASH is the ActionHash or EntryHash of the external object and DNAHASH is the unique identifier for the DHT within which ANYDHTHASH can be resolved. In this design, hrl's are resolved on the client side via a TypeScript service. This assumes the client-side is maintaining a table of bindings to known DHT's -- i.e., the client-side is assuming the same responsibility for resolving hrl's that DNS plays when resolving URL's.
In the MAP, HolonReferences need to be resolvable on the server-side. Thus, the DNS-like service needs to be maintained in the server (i.e., in a Rust Coordinator Zome). Thus, We-like hrl's are not directly applicable to the MAP.
Typed References
Since all independent objects are stored as holons in the MAP, the generic HolonReference type defined above can suffice to refer to any type of object. This is very simple and flexible, but could lead to various kinds of errors. For example, when referring to a shared PropertyDescriptor via HolonReference, what if the type of the referenced holon is not PropertyDescriptor. These problems are similar to the issues encountered with dynamically typed vs. statically type programming languages. The advantage of statically types languages is that they can detect errors of this type at compile time vs. run-time. However, given the dynamic nature of the MAP, compile-time checking is not really an option. Run-time type checking for HolonReferences is the best we can do, but it seems desirable.
This means we need access to the intended HolonType for a HolonReference whenever values are assigned to that reference. A naive approach would be to have each HolonReference be self-describing by including an indication of its intended type. However, this is problematic:
it means every HolonReference, itself, needs to include a reference to the HolonDescriptor for its target type. This begs the question of how to represent the descriptor reference so as not to get caught in an infinite regress.
it costs storage by requiring space for the descriptor-reference in EVERY instance of HolonReference.
We avoid descriptor storage costs for ValueTypes by capturing the intended type of a property in its PropertyDescriptor -- i.e., at the descriptor-level instead of the instance-level. And it is important to note that HolonReferences always exist in some larger context, for example, as a property of a Holon or as a way of representing a single-valued HolonRelationship. In the both cases, we have descriptors that could specify the intended type of the HolonReference.
In short,
**_- Type-checking should be performed as a validation whenever a HolonReference is established
We are assuming we always have access to the appropriate HolonDescriptor when we need it for this type checking WITHOUT having to store a descriptor reference in the HolonReference itself._**
Versioned References
ActionHash is the identity that corresponds to a specific version of an entry. What if new versions have been created? Might we want to update the reference to refer to a new version of this entry? Must this decision always be mediated by human intervention, or are there circumstances where we might want to automatically update the reference to a new version. This could specified via a version update policy. Note, however, that the version update policy shouldn't vary on an instance level, so repeatedly storing the policy doesn't make sense.
It may make sense to store the OriginalActionHash as part of the HolonReference, but (I believe) the OriginalActionHash can always be determined from the Record the ActionHash references, so no need to store that in the reference either.
Proposed Solution
Use ActionHash for Local References. EntryHashes are not guaranteed to be unique.
Type-checking should be performed as a validation whenever a HolonReference is established
Assume we always have access to the appropriate HolonDescriptor when we need it for this type checking WITHOUT having to store a descriptor reference in the HolonReference itself.
Policies for governing version updates should be defined at the descriptor-level, not the instance-level and are likely to be performed as a background task (their design is still pending), so there doesn't seem to be any requirement related to versioning that affects the current structure of HolonReferences.
Use RelationshipTarget as the datatype for representing Holon relationships within a holon.
For any given relationship, the appropriate variant is selected based on the target cardinalities specified by the RelationshipDescriptorbfor that relationship. If the max_target_cardinality is greater than 1, then the Many variant is selected. Otherwise, a HolonReference is used. If the min_target_cardinality = 0, then the ZeroOrOne variant is selected, otherwise the ExactlyOne variant is selected.
The HolonReference enum is used to represent both local holons (i.e., in the same HolonSpace as the source holon) and external holons (i.e., in a different HolonSpace).
pub enum HolonReference {
LocalReference(HolonId),
ExternalReference(ExternalHolonReference),
}
pub struct ExternalHolonReference {
proxy_id: OutboundProxyId, // the ActionHash of the _OutboundSpaceProxy_ in the local space that proxies for the desired external HolonSpace
external_id: HolonId, // the ActionHash of the target holon in the External Space that owns it.
}
Representing HolonReferences in SmartLinks.
The HolonId (i.e., ActionHash) present in either HolonReference variant can be stored in the target_address of the Holochain Link struct. However, the ExternalReference's proxy_id, will need to be encoded in the link tag. There also needs to be an indicator within the LinkTag of whether the HolonReference is Local or External.
Decision
[x] Decision Made
[ ] Diagrams & Wiki updated
Decide on design for Schemas & External Schemas, (containing multiple schemas, but a single DHT) #
What is in-scope for the prototype? What is the design for the in-scope aspects?
I separated the concepts of Schemas from HolonSpaces (whose prototype design decisions are documented above).
Schema as EntryType and as a container and namespace for MAP Descriptors will be added to the prototype.
Decision
See MAP Descriptor Schemas and Type Header
Initial goal is for prototype to include just one schema in just one HolonSpace. However, this HolonSpace will be distinct from the HolonSpace used to store instances of these descriptors. In other words, we will have a Metaspace and a HolonSpace as two distinct apps (with different DHT's). Thus, the reference a holon has to its descriptor will be an external reference. So this is sufficient to prove many aspects of HolonSpaces and external references.
Once the initial milestone has been reached, we can explore multiple schemas in one metaspace and multiple imetaspaces.
I've vacillated between storing Properties in a PropertyMap (BTreeMap) with fast lookup by property name and a Properties relationship (represented via SmartLinks). The latter is structurally simpler but, nominally, means linear lookup of properties by property name. However, I need to define constraints on property names within a HolonType and I've sketched the design of candidate key constraints. My initial reason for doing so was anchor links, but perhaps these can be leverage to provide BTree type lookups.
Decision
It is important not to confuse meta-schema with instance-level concerns.
At the meta-level...PropertyDescriptors are Holons. Therefore, discovering the set of PropertyDescriptors for a given HolonDescriptor, is a matter of navigating the HolonDescriptor'sproperties relationship (and getting back a vector of PropertyDescriptors in return). Under the current design, linear search will be required to locate a desired PropertyDescriptor. As a performance optimization, we could choose to leverage uniqueness constraints defined on the PropertyDescriptor to create Key SmartLinks that could be used to create a BTreeMap for to the PropertyDescriptors. Note however, support for Key SmartLinks is a more general question -- not specific to representing PropertyDescriptor relationships. If supported, in general, in the prototype, we can use them here. If not, we won't. But the PropertyDescriptor use case, by itself, is not compelling enough to warrant support for Key SmartLinks.
At the instance-level...properties are already represented as a PropertyMap, which is a BTreeMap that maps property_names to their values.
Thus, at the instance-level, there is no choice to be made. And at the meta-schema level, the choice is not specific to PropertyDescriptor relationships, so there really is no choice to be made here.
Support for enumerated ValueTypes needs to be part of the prototype. OpenAPI Schema Objects support enum plus a set of discriminators aimed at supporting polymorphism and inheritance. Which of these should be supported in the prototype?
Decision
After briefly re-familiarizing myself with OpenAPI, I've decided to add enum as a ValueType (patterned after the OpenAPI Enum), but not pursue the AnyOf (as a way of denoting SubTypes), AllOf (which I've already covered with composition) or OneOf (as a way of indicating a choice of types in a Response). They add complication and marginal expressiveness. Let's see how far we can get without those constructs.
Enum descriptors are ValueType descriptors whose details are specified via an EnumDescriptor.
pub struct EnumDescriptor {
variant_map: BTreeMap<String,EnumVariant>,
}
pub struct EnumVariant {
variant_label: String, // human-readable text label for this variant
variant_description: Option<String>, \\ human-readable description of this variant
variant_type: Option<ValueType>, \\ variants can be mapped to values of any ValueType
}
will be represented with BTreeMap that maps variant names (represented as Strings) to EnumVariant.
Should Map Descriptors capable of describing MAP Descriptors be a Prototype Goal? #
Use the goal of being able to have rich enough MAP Descriptor capabilities to describe MAP Descriptors as a "completeness" test. Defining a test fixture that creates the MAP Descriptors schema would be a nice test for this goal (and also a nice demonstration of MAP Descriptor capabilities).
To avoid confusion, I've decided to adopt the term Array for ValueTypes and Collection for HolonCollections.
Initially I thought we would need ValueArrays to represent EnumDescriptors. But, at least for now, that does not seem to be the case. Adding ValueArrayDescriptors looks pretty straightforward and Issue #42 has been defined for that enhancement. However, until there is a compelling use case for it, there is no rush to implement that enhancement. It seems that in order to represent the variants within an EnumDescriptor, a ValueArray would be useful.
Decision
Can be added, but only when a compelling use case arises.
Several design questions have arisen and now seems like a timely point to consider a number of MAP Descriptors design questions that have surfaced during our proof-of-concept efforts to date.
NOTE: This is a design-only activity whose sole deliverables are documentation updates... any code changes will be initiated under separate issues.
Design Questions Addressed in this Issue: #
Evaluate Eliminating RelationshipRole #
In the current (not yet implemented) design, MAP RelationshipDescriptors are bi-directional. The
TypeHeaderand references tosourceandtargetHolonTypes are stored in the RelationshipDescriptor and then details for each direction of the relationship are stored in two RelationshipRole objects.Under this design, SmartLinks created for this relationship need to encode both the RelationshipDescriptors
idas well as therole_namein the LinkTag. Although not strictly necessary, our design includes storing adirectionbyte as well.Furthermore, holochain already uses the term role for a very different purpose as part of their capability model.
If Relationships were uni-directional, we would only need to store the relationship descriptor id in SmartLinks. And we avoid the naming conflict with holochain.
Decision
Change the design so that RelationshipDescriptors are uni-directional. Eliminate RelationshipRole. To represent (and support navigation via) both directions, two RelationshipDescriptors will be required. Each such descriptor will include an
inverse_ofreference to its inverse descriptor.Top
Consider unifying all descriptors into one EntryType #
The current design has different EntryTypes for HolonDescriptor and PropertyDescriptor and is on a trajectory to add RelationshipDescriptor and AffordanceDescriptor.
We could unify all of these in a single TypeDescriptor EntryType with a DescriptorDetails enum to handle the various sub-type specific parts. Perhaps these details nest (e.g., ValueType).
Decision
After exploring this approach via UML diagrams, I have decided AGAINST this unification. Having a single type makes it more difficult to express type restrictions. For example, properties should ONLY be allowed ValueTypes. But if everything is a TypeDescriptor, how do I express this restriction? I might be able to make this work with some more Rust trickery, but this would just increase the magnitude of the re-factoring effort.
Top
Decide on design for HolonSpaces #
A HolonSpace is a container for holons and maps 1:1 with a DHT. It also proxies inbound and outbound calls to other HolonSpaces.
A HolonSpace is an L0 foundation for AgentSpaces in L1. Specifically, a HolonSpace serves some of the functions of I-Spaces and the outbound and inbound proxies play roles similar to We-Space. The challenge for the prototype is to discover the minimum viable functionality for HolonSpaces in the prototype to demonstrate cross-DHT references, with minimal scope creep.
A few questions (and answers):
Top
Decide on HolonReference design for local and external references #
The need to express references to another object arises in multiple guises within the MAP. For example, in Holon Relationships whose "to- cardinality" is [0..1] or [1], we just want to store the reference to the related holon.
How should we represent such references?
Requirements
AnyOf,OneOfvariants on expected types.Design Considerations
ActionHash vs EntryHash as HolonId
EntryHashes are not guaranteed to uniquely identify a single version of an object. So we will use the holon's ActionHash as its unique identifier within its HolonSpace. A type alias (
pub type HolonId = ActionHash) is defined to make the intent clear (and to provide some encapsulation of our design choice here). Decision: Use ActionHash for Local References. EntryHashes are not guaranteed to be unique.Holochain Resource Locators (HRL)
In a DM with Guillem, he discussed the use of Holon Resource Locators (hrl) to refer to objects in external DHT's. This concept was added to the We app. Hrl's use a syntax similar to URL's:
hrl://DNAHASH/ANYDHTHASH. In this design, ANYDHTHASH is the ActionHash or EntryHash of the external object and DNAHASH is the unique identifier for the DHT within which ANYDHTHASH can be resolved. In this design, hrl's are resolved on the client side via a TypeScript service. This assumes the client-side is maintaining a table of bindings to known DHT's -- i.e., the client-side is assuming the same responsibility for resolving hrl's that DNS plays when resolving URL's.In the MAP, HolonReferences need to be resolvable on the server-side. Thus, the DNS-like service needs to be maintained in the server (i.e., in a Rust Coordinator Zome). Thus, We-like hrl's are not directly applicable to the MAP.
Typed References
Since all independent objects are stored as holons in the MAP, the generic HolonReference type defined above can suffice to refer to any type of object. This is very simple and flexible, but could lead to various kinds of errors. For example, when referring to a shared PropertyDescriptor via HolonReference, what if the type of the referenced holon is not PropertyDescriptor. These problems are similar to the issues encountered with dynamically typed vs. statically type programming languages. The advantage of statically types languages is that they can detect errors of this type at compile time vs. run-time. However, given the dynamic nature of the MAP, compile-time checking is not really an option. Run-time type checking for HolonReferences is the best we can do, but it seems desirable.
This means we need access to the intended HolonType for a HolonReference whenever values are assigned to that reference. A naive approach would be to have each HolonReference be self-describing by including an indication of its intended type. However, this is problematic:
We avoid descriptor storage costs for ValueTypes by capturing the intended type of a property in its PropertyDescriptor -- i.e., at the descriptor-level instead of the instance-level. And it is important to note that HolonReferences always exist in some larger context, for example, as a property of a Holon or as a way of representing a single-valued HolonRelationship. In the both cases, we have descriptors that could specify the intended type of the HolonReference.
In short,
**_- Type-checking should be performed as a validation whenever a HolonReference is established
Versioned References
ActionHashis the identity that corresponds to a specific version of an entry. What if new versions have been created? Might we want to update the reference to refer to a new version of this entry? Must this decision always be mediated by human intervention, or are there circumstances where we might want to automatically update the reference to a new version. This could specified via a version update policy. Note, however, that the version update policy shouldn't vary on an instance level, so repeatedly storing the policy doesn't make sense.It may make sense to store the
OriginalActionHashas part of theHolonReference, but (I believe) theOriginalActionHashcan always be determined from the Record the ActionHash references, so no need to store that in the reference either.Proposed Solution
Use RelationshipTarget as the datatype for representing Holon relationships within a holon.
For any given relationship, the appropriate variant is selected based on the target cardinalities specified by the RelationshipDescriptorbfor that relationship. If the max_target_cardinality is greater than 1, then the Many variant is selected. Otherwise, a HolonReference is used. If the
min_target_cardinality= 0, then theZeroOrOnevariant is selected, otherwise theExactlyOnevariant is selected.The HolonReference enum is used to represent both local holons (i.e., in the same HolonSpace as the source holon) and external holons (i.e., in a different HolonSpace).
Representing HolonReferences in SmartLinks.
The HolonId (i.e., ActionHash) present in either HolonReference variant can be stored in the target_address of the Holochain Link struct. However, the ExternalReference's proxy_id, will need to be encoded in the link tag. There also needs to be an indicator within the LinkTag of whether the HolonReference is Local or External.
Decision
Decide on design for Schemas & External Schemas, (containing multiple schemas, but a single DHT) #
What is in-scope for the prototype? What is the design for the in-scope aspects? I separated the concepts of Schemas from HolonSpaces (whose prototype design decisions are documented above).
Schema as EntryType and as a container and namespace for MAP Descriptors will be added to the prototype.
Decision
See MAP Descriptor Schemas and Type Header Initial goal is for prototype to include just one schema in just one HolonSpace. However, this HolonSpace will be distinct from the HolonSpace used to store instances of these descriptors. In other words, we will have a Metaspace and a HolonSpace as two distinct apps (with different DHT's). Thus, the reference a holon has to its descriptor will be an external reference. So this is sufficient to prove many aspects of HolonSpaces and external references.
Once the initial milestone has been reached, we can explore multiple schemas in one metaspace and multiple imetaspaces.
Top
PropertyMaps vs SmartLink Vectors #
I've vacillated between storing Properties in a PropertyMap (BTreeMap) with fast lookup by property name and a Properties relationship (represented via SmartLinks). The latter is structurally simpler but, nominally, means linear lookup of properties by property name. However, I need to define constraints on property names within a HolonType and I've sketched the design of candidate key constraints. My initial reason for doing so was anchor links, but perhaps these can be leverage to provide BTree type lookups.
Decision
It is important not to confuse meta-schema with instance-level concerns.
At the meta-level... PropertyDescriptors are Holons. Therefore, discovering the set of PropertyDescriptors for a given HolonDescriptor, is a matter of navigating the HolonDescriptor's
propertiesrelationship (and getting back a vector of PropertyDescriptors in return). Under the current design, linear search will be required to locate a desired PropertyDescriptor. As a performance optimization, we could choose to leverage uniqueness constraints defined on the PropertyDescriptor to create Key SmartLinks that could be used to create a BTreeMap for to the PropertyDescriptors. Note however, support for Key SmartLinks is a more general question -- not specific to representing PropertyDescriptor relationships. If supported, in general, in the prototype, we can use them here. If not, we won't. But the PropertyDescriptor use case, by itself, is not compelling enough to warrant support for Key SmartLinks.At the instance-level...
propertiesare already represented as a PropertyMap, which is a BTreeMap that mapsproperty_namesto theirvalues.Thus, at the instance-level, there is no choice to be made. And at the meta-schema level, the choice is not specific to PropertyDescriptor relationships, so there really is no choice to be made here.
Top
Add Enum ValueType Support, Evaluate OpenAPI Enum and Discriminators (AnyOf, AllOf, OneOf) #
Support for enumerated ValueTypes needs to be part of the prototype. OpenAPI Schema Objects support enum plus a set of discriminators aimed at supporting polymorphism and inheritance. Which of these should be supported in the prototype?
Decision
After briefly re-familiarizing myself with OpenAPI, I've decided to add
enumas aValueType(patterned after the OpenAPI Enum), but not pursue theAnyOf(as a way of denoting SubTypes),AllOf(which I've already covered with composition) orOneOf(as a way of indicating a choice of types in a Response). They add complication and marginal expressiveness. Let's see how far we can get without those constructs.Enum descriptors are ValueType descriptors whose details are specified via an EnumDescriptor.
will be represented with BTreeMap that maps variant names (represented as Strings) to EnumVariant.
Top
Should Map Descriptors capable of describing MAP Descriptors be a Prototype Goal? #
Use the goal of being able to have rich enough MAP Descriptor capabilities to describe MAP Descriptors as a "completeness" test. Defining a test fixture that creates the MAP Descriptors schema would be a nice test for this goal (and also a nice demonstration of MAP Descriptor capabilities).
Decision
YES! This is valuable for a number of reasons.
Top
Decide on Prototype support for ValueArrays #
If such types are needed, what is their design?
To avoid confusion, I've decided to adopt the term Array for ValueTypes and Collection for HolonCollections.
Initially I thought we would need ValueArrays to represent EnumDescriptors. But, at least for now, that does not seem to be the case. Adding ValueArrayDescriptors looks pretty straightforward and Issue #42 has been defined for that enhancement. However, until there is a compelling use case for it, there is no rush to implement that enhancement. It seems that in order to represent the variants within an EnumDescriptor, a ValueArray would be useful.
Decision
Can be added, but only when a compelling use case arises.
Top