Content Directory 2.0

Background

Our current content directory model consists of three modules

Versioned Linked Object Store (VLOS): Models the data store that holds the data model for our content directory.
VLOS Permissions: Models and guards write access to the VLOS.
Working Group: Governance layer which organises the actors that interact with the permissions module and VLOS

The two first modules are a result of wanting to have a metadata layer which hash high integrity guarantee in a multi-writer scenario, and also deal well with changing schemas in-flight, without having to do runtime upgrades. The basic architecture appears to be the best compromise of our goals at present, however there are implementation issues with all three that require attention.

Proposal

VLOS

Drop all events.
Enforce unique class names. <= Why? perhaps we should even drop them?
Enforce unique property names within each class. <= Why? perhaps we should even drop them?
Rename PropertyType::Internal to PropertyType::Reference, and same for PropertyValue::Internal, as we are dropping the notion of external references.
Remove PropertyType::None, was added to have some value for required Default trait, but has no semantic content. Instead just return some other value needed, we should not sacrifice our type safety due to a bad Substrate design decision.
Remove Class::id and Entity::id, as per convention https://github.com/Joystream/joystream/issues/36#issuecomment-527891032.
Raw index types u16 are used for index types, replace with aliases for each use case, e.g. ClassId, EntityId, PropertyId etc. for clarity. Also make public
Add boolean field in ClassSchema which indicates whether a schema is active or not. It is only possible to add schema support to an entity for an active schema. Introduce method for updating this status also.
ClassSchema -> Schema
Change values field of Entity to BTreeMap<u16, PropertyValue> where key is index of property in class, and kill ClassPropertyValue.
Change in_class_schema_indexes in Entity to BTreeSet, and give better name to field, like supported_schemas.
https://github.com/Joystream/substrate-versioned-store-module/issues/13
https://github.com/Joystream/substrate-versioned-store-module/issues/22
Drop from storage these from storage, they need not be tuneable at runtime, just make them configurable through runtime trait. Also remove InputValidationLengthConstraint, as it is no longer needed.

pub PropertyNameConstraint get(property_name_constraint)
    config(): InputValidationLengthConstraint;

pub PropertyDescriptionConstraint get(property_description_constraint)
    config(): InputValidationLengthConstraint;

pub ClassNameConstraint get(class_name_constraint)
    config(): InputValidationLengthConstraint;

pub ClassDescriptionConstraint get(class_description_constraint)
    config(): InputValidationLengthConstraint;

Introduce a new trait for Module, so that other modules depending on VLOS can consume this trait on runtime, which also allows for mocking.

Housekeeping

Write docs for everything, including intro.

VLOS Permissions

Channels now can live in VLOS due to same owner constraint.
Lead is no longer fully trusted, all operations are constrained.
New permissions to users from blowing up tings: like million length playlist.
lockable fields solve the curation problem from beofre
transactions are actually tranasctional!!!
Permissions are much simpler to reason about when everything is combined, and curators can act in groups.

Impl sketch

/// # Content Directory
///
/// There are two basic actors in this module, groups, and actors, where an actor is a member of exactly one group.
/// Further, there are three explicit roles
///
/// - Entity creator: this a group whos members can create entities of a given class. A given class may have multiple such groups at any time.
/// - Entity controller: this is a group, or a specific actor, who can add schema support and update property values. Initially, creator of an entity is set as controller, either directly, or through their group membership. A given entity has only one controller.
/// - Class maintainer: this is a group whos members can update some, but not necessarily all, property values of any entity of a given class. A given class may have multiple such groups at any time.
///
/// Root origin may authenticate as any actor in any group, hence root can impersonate all roles.
/// Moreover, root is also responsible for actually creating classes, permissions, adding classes to schemas, etc.
/// Notice that this means that it is up to the implementor of this code to add critical constraints that prevent
/// the system from being open to DoS attacks. This includes the
/// - number of classes
/// - number of schemas
/// - number of properties in schemas
/// - size of property types in schemas

/// Model of authentication manager.
trait ActorAuthenticator<T: Trait> {

  /// Actor identifier
  type ActorId: /* integer constraint */;

  /// Group identifier
  type GroupId: /* integer constraint */;

  /// Authenticate account as being current authority.
  fn authenticate_authority(account_id: T::AccountId) -> bool;

  /// Authenticate account as being given actor in given group.
  fn authenticate_actor_in_group(account_id: T::AccountId, actor_id: Self::ActorId, group_id: Self::GroupId) -> bool;

}

/// Module configuration trait.
trait Trait: versioned_object_store::Trait + versioned_object_store::Store {

}

/// Identifier for classes.
type ClassId<T: Trait> = <T as versioned_object_store::Trait>::ClassId;

/// Identifier for entities.
type EntityId<T: Trait> = <T as versioned_object_store::Trait>::EntityId;

/// Identifier for properties.
type PropertyId<T: Trait> = T::versioned_object_store::PropertyId;

/// A set of property identifiers.
type PropertyIndexSet<T: Trait> = BTreeSet<PropertyId<T>>;

/// A map of from property identifiers.
type PropertyMapTo<T: Trait, V> = BTreeMap<PropertyId<T>, V>;

/// Identifier for class permissions.
type ClassPermissionId<T> = ClassId<T>;

/// Identifier for entity permissions.
type EntityPermissionId<T> = EntityId<T>;

/// Identifier for a given actor in a given group.
struct ActorInGroupId<T: Trait> {
  pub actor_id: T::ActorId,
  pub group_id: T::GroupId
}

/// Limit for how many entities of a given class may be created.
enum EntityCreationLimit {

  /// Look at per class global variable `ClassPermission::per_controller_entity_creation_limit`.
  ClassLimit,

  /// Individual specified limit.
  Individual(u64)
}

/// A voucher for entity creation
struct EntityCreationVoucher<T: Trait> {

  /// How many are allowed in total
  pub maximum_entity_count: u64,

  /// How many have currently been created
  pub current_entity_limit: Limit,

}

/// Who will be set as the controller for any newly created entity in a given class.
enum InitialControllerPolicy {
  ActorInGroup,
  Group
}

/// Permissions associated with a class.
struct ClassPermissions<T: Trait> {

  /// Whether to prevent everyone from creating an entity.
  ///
  /// This could be useful in order to quickly, and possibly temporarily, block new entity creation, without
  /// having to tear down `can_create_entities`.
  pub entity_creation_blocked: bool,

  /// Policy for how to set the controller of a created entity.
  ///
  /// Examples(s)
  /// - For a group that represents something like all possible publishers, then `InitialControllerPolicy::ActorInGroup` makes sense.
  /// - For a group that represents some stable set of curators, then `InitialControllerPolicy::Group` makes sense.
  pub initial_controller_of_created_entities: InitialControllerPolicy,

  /// Properties for which controller cannot update, and similarly, provide values during creation.
  ///
  /// Example(s):
  /// - Sensitive information, such curation status of the entity.
  pub properties_locked_from_controller: PropertyMapTo<Option<versioned_object_store::PropertyValue>>,

  /// Properties for which maintainers cannot update values.
  ///
  /// Example(s):
  /// - Payment information of a publisher on a video or channel.
  pub properties_locked_from_maintainers: PropertyIndexSet<T>,

  /// Whether to prevent everyone from updating entity properties.
  ///
  /// This could be useful in order to quickly, and probably temporarily, block any editing of entities,
  /// rather than for example having to set, and later clear, `EntityPermission::frozen_for_controller`
  /// for a large number of entities.
  pub all_entity_property_values_locked: bool,

  /// Referential properties that can only reference entities with same controller as entity to which
  /// property value belongs.
  ///
  /// Example(s):
  /// - A video publisher can only place it under a channel they control.
  pub referenced_entity_must_have_same_controller: PropertyIndexSet<T>,

  /// The maximum number of entities which can be created.
  pub maximum_entity_count: u64,

  /// The current number of entities which exist.
  pub current_number_of_entities: u64,

  /// How many entities a given controller may create at most.
  pub per_controller_entity_creation_limit: u64,
}

/// Owner of an entity.
enum EntityController<T: Trait> {
  Group(T::GroupId),
  ActorInGroup(ActorInGroupId<T>)
}

/// Permissions for a given entity.
struct EntityPermission<T: Trait> {

  /// Current controller, which is initially set based on who created entity and
  /// `ClassPermission::initial_controller_of_created_entities` for corresponding class permission instance, but it can later be updated.
  pub controller: EntityController<T>,

  /// Controller is currently unable to mutate any property value.
  /// Can be useful to use in concert with some curation censorship policy
  pub frozen_for_controller: bool,

  /// Prevent from being referenced by any entity (including self-references).
  /// Can be useful to use in concert with some curation censorship policy,
  /// e.g. to block content from being included in some public playlist.
  pub referenceable: bool
}

/*
 * decl_storage
 */

/// Class permission by id.
/// The identifier value for a class permission is identical to the identifier for the underlying class.
class_permission_by_id: linked_map ClassPermissionId<T> => ClassPermissions<T>,

/// Groups who's actors can create entities of class.
can_create_entities_of_class: double_map (ClassId<T>, T::GroupId) -> (),

/// Groups who's actors can act as entity maintainers.
pub entity_maintainers: double_map (ClassId<T>, T::GroupId) -> (),

/// Entity permission by id.
/// The identifier value for an entity permission is identical to the identifier for the underlying entity.
entity_permission_by_id: map EntityPermissionId<T> => EntityPermission<T>

// The voucher associated with entity creation for a given class and controller.
// Is updated whenever an entity is created in a given class by a given controller.
// Constraint is updated by Root, an initial value comes from `ClassPermissions::per_controller_entity_creation_limit`.
entity_creation_vouchers: double_map (ClassId<T>, EntityController<T>) => EntityCreationVoucher<T>

/// Upper limit for how many operations can be included in a single invocation of `atomic_batched_operations`.
maximum_number_of_operations_during_atomic_batching: u64,

/*
 * decl_module
 */

// ======
// Next set of extrinsics can only be invoked by root origin.
// ======

/// Change the controller of an entity.
///

fn update_entity_controller(
  origin,
  entity_id: EntityId<T>,
  new_controller: EntityController<T>
)

/// Make a group into an entity creator for a class.
fn add_entitiy_creator(
  origin,
  class_id: ClassId<T>,
  group_id: T::GroupId,
  limit: EntityCreationLimit
)

/// Remove group
fn remove_entity_creator(...)

/// ...
fn add_entity_maintainer(...)

///
fn remove_entity_maintainer(...)

/// Update entity creation voucher
fn update_entity_creation_voucher(...)

/// ...
fn update_class_permissions(
  origin,
  class_permission_id: ClassPermissionId<T>,
  updated_entity_creation_blocked: Option<bool>,
  updated_initial_controller_of_created_entities: Option<InitialControllerPolicy>,
  ...
)

///
/// Implementation note: changing controller does respect, and impact, voucher of existing or new controller.
fn update_entity_permissions(
  origin,
  entity_perimission_id: EntityPermissionId<T>,
  update_controller: Option<EntityController<T>>,
  update_frozen_for_controller: Option<bool>
)

// ======
// The next set of extrinsics can be invoked by anyone who can properly sign for provided value of `ActorInGroupId<T>`.
// ======

/// Create an entity.
/// If someone is making an entity of this class for first time, then a voucher is also added with the class limit as the default limit value.
/// class limit default value.
/// The `as` parameter must match `can_create_entities_of_class`, and the controller is set based on `initial_controller_of_created_entities` in the class permission.
fn create_entity(
  origin,
  class_id: ClassId<T>,
  as: ActorInGroupId<T>,
  ...
);

/// Update a property with a new value for an entity.
/// The `as` parameter must match either be compatible with the controller of the entity, or the maintainers of the class.
/// Implementaion note:
fn update_entity_property_values(
  origin,
  entity_id: EntityId<T>,
  as: ActorInGroupId<T>,
  ...
);

/// Add schema support to an entity.
/// The `as` parameter must match either be compatible with the controller of the entity, or the maintainers of the class.
fn add_schema_support_to_entity(
  origin,
  entity_id: EntityId<T>,
  as: ActorInGroupId<T>,
  ...
);

/// Atomic batched parametrised operations.
/// Allows you to atomically do multiple operations, where results of earlier operations can be used as inputs into later ones.
///
/// Implementation comment: we must use new atomic updating feature https://github.com/paritytech/substrate/pull/3263 ,
/// otherwise same approach as in current permissions module, and use same `as` value in all operations.
///
/// Input must respect `maximum_number_of_operations_during_atomic_batching`.
fn atomic_batched_parametrised_operations(
  origin,
  as: ActorInGroupId<T>,
  ...
);

/*
 * decl_events
 */

// Add events for every extrinsic

Housekeeping

Write docs for everything, including intro.
Introduce mocking of VLOS in tests? no longer side-effects.

Working Group

Most importantly, it needs to be refactored to follow this approach

https://github.com/Joystream/joystream/issues/163

but this depends on someone building the Bureaucracy module first.

Drop all the channel related stuff, it will go directly in
New authenticatinos scheme
Invoke root origin calls to manage, and keep local constraints.

Version tool

TBD.

Joystream / joystream

Proposal: Content Directory 2.0 #327

Content Directory 2.0

Background

Proposal

VLOS

Housekeeping

VLOS Permissions

Impl sketch

Housekeeping

Working Group

Version tool