Implement API interface for DSL runtime

[pospi]

Proposed toplevel crate API interfaces from conversations between @mhuesch & @pospi -

Compiler

Language parsing and AST introspection

• [ ] fn parse_calculation(dsl_document: AsRef<str>) -> Result<poly_rs::Expr>
  • throws an error if the document doesn't pass type checking
• [ ] fn get_calculation_input_type(calculation: poly_rs::Expr) -> poly_rs::Type
• [ ] fn get_calculation_return_type(calculation: poly_rs::Expr) -> poly_rs::Type
• [ ] poly_rs::Type::TRelated(op, Type, Type) needs to be implemented to support calculated / derived units

Interpreter

Runtime execution of interpreted reputation language expressions against some set of input data

• [ ] fn reduce_calculation<R>(calculation: poly_rs::Expr, input_data: Iter<poly_rs::Value>) -> ReputationCalculationResult

where

  struct ReputationCalculationResult  {
      calculation: poly_rs::Expr,
      type: poly_rs::Type,
      value: poly_rs::Value,
  }

Internal operation:

At each stage in the computation (each unwinding of poly_rs::Expr)—

• poly_rs::Value type is provided by input_data
• expected poly_rs::Type of input is determined by analysing the poly_rs::Expr (possibly, but not necessarily with get_calculation_input_type(calculation)), and validated against poly_rs::Value. Runtime returns an error if incompatible.
• poly_rs::Type of output is determined by get_calculation_return_type(calculation) and carried through as the input type to next step in the VM execution

At every step in evaluation the runtime by its nature should have awareness of poly_rs::Type and poly_rs::Value, which enables it to infer compatibility of arithmetic operations between different poly_rs::Type with compatible underlying poly_rs::Value types. (eg. metres / seconds = valid m/s because both inputs are doubles / decimals)

[mhuesch]

I have made some decent progress on this over on this branch.

the type signatures are a little different:

pub fn parse_calculation(dsl_document: String) -> Result<syntax::Program, String>

pub fn get_calculation_type(program: syntax::Program) -> Result<types::Scheme, String>

pub fn reduce_calculation(
    prog: syntax::Program,
    input_data: &mut dyn Iterator<Item = eval::Value>,
) -> Result<ReputationCalculationOutput, ReputationCalculationError>

pub struct ReputationCalculationOutput {
    pub rcr_calculation: syntax::Expr,
    pub scheme: types::Scheme,
    pub value: eval::Value,
}

pub enum ReputationCalculationError {
    ArityMismatch(usize, usize),
    ProgramTypeInferenceError(TypeError),
    ProgramValuesUnificationError(TypeError),
}

// I opted to create a single function `get_calculation_type` above, onto whose output these functions can be applied.
// it returns a `Scheme`, which is `Type` along with any free `TVar`s in the type. so it's slightly different but these functions can be made to line up with each other.
pub fn type_arguments(ty: &Type) -> Vec<Type>
pub fn type_return(ty: &Type) -> Type

it seems to be going fairly well, and I have mostly filled in the body of reduce_calculation. it has some bugs which I've already identified / left as TODOs as I went along.

I have not seen any significant issues with the approach thus far - seems like it will work, and may be all wrapped up in a few more days of work.

I have not yet tackled poly_rs::Type::TRelated(op, Type, Type).

[pospi]

Looking great!

Clarification- do type_arguments and type_return deal with types::Scheme, rather than Type as indicated?

I have been thinking about ReputationCalculationOutput a bit more. What it really craves is some datum held as an indicator of the source information which yielded the given result.

Maybe a sum would be a good way to carry this information forward in aggregate? So we could include an additional source: eval::Value that can be used in combination with type_arguments to determine what kind of information was passed in, and how many records?

[mhuesch]

Looking great!

Clarification- do type_arguments and type_return deal with types::Scheme, rather than Type as indicated?

Right now they force the user to unpack the Scheme, get out the enclosed Type, and pass that in. I suppose I could make a trait method so that the functions work with both Scheme & Type.

I have been thinking about ReputationCalculationOutput a bit more. What it really craves is some datum held as an indicator of the source information which yielded the given result.

Maybe a sum would be a good way to carry this information forward in aggregate? So we could include an additional source: eval::Value that can be used in combination with type_arguments to determine what kind of information was passed in, and how many records?

Hmm, I think I lack context to understand this & the motivation. Would source be a single element or an iterator/vector of multiple, for the arguments provided to reduce_calculation?

Maybe it would help me to understand the constraints on the “caller” (which is the Vault?) - what it needs and is looking to do.

There’s a lot of data being plumbed around which I could easily imagine being useful “to the outside” and perhaps I can be more systematic in persisting it / making it available.

[mhuesch]

@pospi would it be a good use of time for you to run me through some of the vault so I can understand the constraints? then we can have two brains guiding the structuring of the API.

I see https://github.com/sacredcapital/ReputationVault as empty, maybe code is somewhere else.

[mhuesch]

I think the guts of this are mostly done on the branch. I have not implemented type inference for closures, which means they cannot be provided via the values iterator. I am pretty sure I know how to support them, but figure it is less likely that people will want to send in values?

So, I think we might be able to wire the top level API up to other things (the vault?) now and see what works / breaks.

[pospi]

Noting work to do following today's sync call:

• [ ] Currently all input_data are being evaluated to prog all at once; items from the iterator need to be incrementally evaluated for memory efficiency (like a foldl implementation). So whereas poly_rs is now handling the full fold operation over the complete set of input data, instead the language should be set up to execute individual fold steps of the current datum and return each accumulated result.
  • Probably the method for making this change would be to alter VList to wrap Iterator instead of Vec.
• [ ] Unit / integration testing for key features / aspects of the library so we can prove correctness when making any upgrades going forward
• [ ] Do some investigation into blowing the stack with unbounded iterators and have a think about how we can mitigate non-terminating computation
  • For non-agent-centric apps like Junto 1.0, how would they sandbox this? Would it execute on their servers or in a browser WASM sandbox that fetches data from Junto's web app over the network?
  • Could we potentially implement a total language to address this?

[pospi]

RE total langauges-

• poly_rs doesn't allow recursion yet
• poly_rs has no looping so far; only map & fold

So, unbounded computation is not an issue until we need such features.

[pospi]

• [ ] Another missing feature is needed, implementation still unknown but essentially the input_data needs to provide the Type of each datum as well as yielding each Value in turn. This is so that the Type expected as input to the Program can be validated against the Type found for each piece of data - it cannot be relied upon to match since such types are dynamically provided by Reputation Vault plugins.

[mhuesch]

I am favoring a more "organic" approach to developing this top-level API, starting with https://github.com/sacredcapital/rep_interchange/issues/8 as "first contact between Holochain and rep_lang".

there are good things in this issue to revisit when designing the successor "directed experiments" which will follow https://github.com/sacredcapital/rep_interchange/issues/8.