search

filecoin-project / devgrants

👟 Apply for a Filecoin devgrant. Help build the Filecoin ecosystem!
Other
371 stars 308 forks source link

RFP Application: FVM - High level Rust SDK #562

Closed tchataigner closed 1 year ago

tchataigner commented 2 years ago

RFP Proposal: FVM - High level Rust SDK

Name of Project: FVM - High level Rust SDK

Link to RFP: Please link to the RFP that you are submitting a proposal for.

RFP Category: devtools-libraries

Proposer: @tchataigner

Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT, GPL, and APACHE2 licenses?: Yes

Project Description

References

In the context of the FVM Early Builder program, @BlocksOnAChain suggested that we could handle the creation of a High level Rust SDK to help with the development of Rust-native actors for the FVM.

The following specifications are meant to produce a Rust crate that could be the only fvm_* crate-related import to produce valid code. The different specifications have multiple references:

  1. FVM spec: architecture
  2. @raulk example: Hello world actor example
  3. @jimpick experiment: Hanoi actor
  4. Current low-level sdk

Development Roadmap & Deliverables

Pre-Requirement

Milestone 1 - SDK implementation

This milestone has for objective to release the SDK in its 0.1-beta version.

M1.1 - Extension of current low-level SDK

Technical scope:

Deliverables:

M1.2 - fvm_state proc macro

Technical scope:

Creation of a procedural macro to declare the structure representing the internal state of an actor.

Parameters

Deliverables:

M1.3 - fvm_actor proc macro

Technical scope:

Creation of a procedural macro used to encompass methods available in the actor.

Paremeters

Deliverables:

M1.4 - fvm_export proc macro

Technical scope:

Creation of a procedural macro used to flag exposed functions in the actor.

Parameter

Deliverables:

Milestone 2 - Community feedback integration

This milestone has for objective to iterate with the community (most likely the FVM EBP participants) to update the SDK according to their feedback.

M2.1 - Aggregate and report community feedback

During this milestone, the Polyphene team will not be actively working on the SDK but will still be available to answer messages and questions from testers.

Deliverables:

M2.2 - Feedback integration

Deliverables:

Milestone 3 - Miscellaneous

Deliverables:

Roadmap

The proposed timeline below would allow for the development of the discussed features with the integration of a review and community discussion phase by the end of August 2022. A split with other dates would make it more difficult for the proposed development team to meet this deadline.

Removal of PM

After discussing with Raul it was decided try try to do the project without a PM. The main reason is that we will be developing during summer which should reduce number of feedbacks. Consequences are:

New Roadmap

Phase Dates Workforce
Dev of the milestones M1.1 & M1.2 and the adequate documentation. July 4 - 15 (2 weeks) 2 full-time developers
Feedback integration phase after a demonstration during an FVM EBP check-in. July 18 - 22 (1 week) 1 full-time developer
Dev of the milestones M1.3 & M1.4 and the adequate documentation July 25 - August 5 (2 weeks) 2 full-time developers
Feedback integration phase after a demonstration during an FVM EBP check-in. August 8 - 12 (1 weeks) 1 full-time developer

Note: Feedback will be done through Github issues on the development repository helped by a template defined by Polyphene.

Pricing

Total: ( 2 $1.5k ) 4w 5d + $1.5k 2w * 5d == $75k

To reflect the existing relationship of trust between PL and Polyphene, and the opportunity given in the choice of milestone dates to be set by the development team, Polyphene would agree for this grant to a 15% discount on its rates.

Final pricing: $75k * 85% == $63,75k

Maintenance and Upgrade Plans

We have no plan on maintaining the SDK in this proposal. This should be arranged either at a later time or in another program.

Team

Contact Info

On Filecoin slack:

Team Members

Team Member LinkedIn Profiles

Team Website

https://polyphene.io

Relevant Experience

We already have some experience in handling wasm runtime and development of tooling around it. We spent last year designing and building the Holium project. It allowed us to hone our skills to build rust-based protocols and libraries.

Moreover, we followed the FVM specification and development since its premises. We also participated in its development by helping on the creation of an integration test framework.

Team code repositories

Holium Rust

Holium Rust SDK

tchataigner commented 2 years ago

Hey @BlocksOnAChain, just opened the PR as we discussed. It is still in a draft phase but I need review on the Project description part where the specifications need some iterations!

Hope we can get to a nice result together :)

BlocksOnAChain commented 2 years ago

@tchataigner do we plan details around Deliverables and Funding since this is something we definitely need if we want to go towards the approval stage for the RFP? Other items look ok to me, we just need to improve the draft we started from with more details. cc: @raulk @DeveloperAlly

DeveloperAlly commented 2 years ago

We can also perhaps add the purpose of this sdk - being an abstraction and extension of functionality to the current FVM Rust implementation aiming to provide a better developer experience for those building on FVM and enabling foreign runtime bridge options.

Potential Milestones (we like to only have 2-3) can probably come from your set of ideal behaviours mentioned. Let me know if you want some help with developing the milestones above @tchataigner .

In terms of the funding amount to request - if you don't have a clear idea - @eshon will be able to help. Do you perhaps have a napkin calculation or parameters/formula you use and would be able to share in this regard @eshon ?

Otherwise - let's build this! :P

mishmosh commented 2 years ago

Hello from the grants team. I'll leave the SDK design questions to the FVM team.

For now, we'd like to see more clarity around the following topic:

Once you have more clarity around some of the design questions (eg how serialization calls should work), ping me if you'd like help planning the milestones and deliverables sections.

tchataigner commented 2 years ago

@tchataigner do we plan details around Deliverables and Funding since this is something we definitely need if we want to go towards the approval stage for the RFP? Other items look ok to me, we just need to improve the draft we started from with more details. cc: @raulk @DeveloperAlly

Otherwise - let's build this! :P

Hey everyone thank you for the feedbacks ! I know that the draft is not ready to be approved yet but I would need to clarify some blurry parts raised in the proposed specifications before filling the Deliverables and Funding parts. I guess someone from the FVM team would be best (@raulk would be the best choice I guess).

The parts are the following: Glue code generation and internal dispatch

Extension of low level SDK

What documentation, education, and/or community-building is needed for users to effectively learn & use the SDK? At minimum for an SDK project there should be well-documented code and an excellent README. A short video demo of usage (~5 mins) would be good to have as well.

Thanks for the feedback @mishmosh ! That is interesting! I guess that we should think of documentation for the SDK as a part for the documentation of the FVM (like the Go SDK for IPFS is part of the documentation of IPFS). Is there any plan around that? If none we could use Docusaurus to create a static website to document the SDK.

I feel like having a video is not really part of the documentation as it falls in a overall ramp access to the FVM from my point of view. For the video we could have a presentation in the FVM check-in and have it recorded what do you say.

raulk commented 2 years ago

@tchataigner I am so sorry for the delay here. We were heads down on shipping the M1 development freeze, and then I was OOO for a few days. I'm looking through this now.

raulk commented 2 years ago

Part 1 of feedback

Ideal behaviours

From those sources came out some ideal features to implement:

From current actors

  • Remove the need for the user to code the invoke function

👍 I'd enhance this and say that it should support multiple call conventions, where the method number is one of them. It could be the default so it could be omitted, but I'd stay away from doing so to train developers to think about call conventions explicitly. The snippet at the top of fvm.filecoin.io demostrates how I was thinking about specifying the call convention:

#[fvm_actor(state=ComputeState, dispatch="method_num")]

This attribute won't be enough though. We'll need per-method annotations too to define the binding to the call convention. For example, in that example:

  /// Creates a job with an input DAG, WASM compute logic,
  /// data affinity, geographical bounds, and timeout.
  #[fvm_export(binding=1)]
  pub fn create_job(req: CreateJobReq, st: ComputeState)
    -> CreateJobRes { ... }

  /// Allows a compute node to claim a job by proving it
  /// satisfies the requirements and staking collateral.
  #[fvm_export(binding=2)]
  pub fn claim_job(req: ClaimJobReq, st: ComputeState)
    -> ClaimJobRes { ... }

  /// Proves that a compute node finished running a job,
  /// posts the result, and claims the reward.
  #[fvm_export(binding=3)]
  pub fn prove_done(req: ProveDoneReq, st: ComputeState)
    -> ProveDoneRes { ... }
}

This would automatically generate the invoke entrypoint and the routing table according to the dispatch strategy and the bindings. Maybe the binding should be some kind of "any" type, because values will be specific to the call convention (i.e. dispatch strategy). Imagine for example a Solidity-like call convention (N-truncated bytes from hashed target method signature).

We don't need to implement it now, but users should be able to provide their own dispatch strategy down the line. (Note: eventually we want to phase out message-defined method numbers).

  • Remove the need for the user to handle serde of payloads

Yes! Big +1 to this. To make this future-proof, we'll need to define the IPLD codec that's being used. I think this is best defined at two levels:

That combination would apply DagCbor serde to parameters and return values on all methods except for the one being annotated with DagPb.

  • Ease State structure declaration in actors and auto generate save() and load() functions

Big +1 to this, but there's a little bit more here:

A more complex actor example is the old/outdated ERC20 token actor: https://github.com/filecoin-project/ref-fvm/pull/290. Take a look at the docs, concretely the "Boilterplate" section, as well as the review comments for ideas from @Stebalien (most of which are similar to what I've proposed here, as he and I have refined this thinking together over time).

I think #[fvm_export] should define the transaction semantics for a method and it should be used in combination with the mutability of the state argument of the method. For example:

#[fvm_export(tx=ro)]
fn hey(state: &State) { } // immutable state, can never be upgraded to a transaction

#[fvm_export(tx=rw)]
fn hey(state: &State) { } // immutable state, can be upgraded to a transaction

#[fvm_export(tx=rw)]
fn hey(state: &mut State) { } // mutable state, boilerplate manages the transaction
raulk commented 2 years ago

Part 2 of feedback

  • Some low-level SDK syscall should be available to the user if they are deemed interesting (crypto ...)

I think all low-level SDK syscalls should be available to the user. I'd take a porcelain/plumbing approach there, where the user is expected to use the porcelain (high-level part), but if they need to do something less common, they can fall back to the plumbing (low-level part). These terms are taken from Git nomenclature.

  • Have access to basic types that could be useful for actor development

Yep. Maybe this SDK could depend on the existing low-level SDK and re-export such types?

From other languages

  • Have access to block & tx properties

Sure! Adding these syscalls would be beyond just the SDK -- they'd need to be implemented on the kernel side. Are you good with including that in the scope too? I would be very open. Although because this would imply a protocol change, I'd first start discussions in the FIPs repo: https://github.com/filecoin-project/FIPs/discussions, proposing the concrete syscalls, gathering feedback, and opening issues in ref-fvm to implement if there's consensus.

  • Implement error handling function such as _assert__* for the user to use in their contract

I like this idea. In fact, I wonder if it makes sense to have declarative validation like in https://github.com/Keats/validator. In case of failure, you'd abort with exit code 24 (USR_ASSERTION_FAILED). See https://github.com/filecoin-project/fvm-specs/blob/main/07-errors.md.

  • Have access to the deployed actor information (address ...)

You mean like the receiver address? (This is available through the vm::context() syscall).

From specifications

potentially mapping dynamically-linked libraries (e.g. predefined SDK versions)

  • Means that the SDK core module shall not be within the actor base code but linked to it. Reduce actors’ size.

This would be awesome, although it's a significant endeavour (and a project of its own). We envision some form of a "library registry" actor where users can push non-instantiatable (and only importable) pieces of code. The right way to do this would be through content addressing, i.e. importing a CodeCID from the actor and having the FVM resolve that for you. It's not simple and I would recommend not including it in the scope of this. This is likely FVM M3+ material.

TBC Draft Specifications

Note: These are the proposed implementation choices to be developed. It is still to be discussed before considering a roadmap.

Mandatory elements in an actor

Entry point

  • invoke() : entry point for the actor, contains a map to dispatch call to proper method based on its number.

Note that this is the Filecoin standard call convention (there will be other call conventions going forward).

State

  • Struct that derives Serialize_tuple and Deserialize_tuple
  • Should have a save() and load() implementation from a trait. Proposal for StateObject trait.

I think the state struct should be annotated with an attribute as well that specifies the serialization codec (Cbor) and style (tuple): #[fvm_state(codec = Cbor, style = Tuple)]?

Proposed implementation

  • If we generate invoke() then how can the user and future callers know which function are associated with which index ?

This would be specified in #[fvm_export]?

  • I was thinking of having something that would look like that as an actor:

    ...

But it would mean that all types as parameters and/or returned from functions should implement Serialize_tuple and/or Deserialize_tuple otherwise generated code would not work. Is it alright? Should we do it another way?

Can we do this automatically by adding the attributes to the params and return types? (Might be hard though, I'm also OK with the user having to add attributes to the types, e.g. #[fvm_transferrable] for transferrable types (params and return)?

  • Having glue code generated means heavier actor bytecode on first deployment and more gas spent at runtime. Is it tolerable ?

Could you elaborate? In my head, the glue code would replace code that the user has to write manually today anyway.

raulk commented 2 years ago

@tchataigner One more requirement for the SDK is that it should automatically handle panics by catching them and lowering them to an abort. See https://github.com/filecoin-project/builtin-actors/blob/master/actors/runtime/src/runtime/fvm.rs#L552-L554.

Generally speaking, the built-in actors runtime is decent inspiration for this grant.

raulk commented 2 years ago

@Stebalien probably has some input here.

raulk commented 2 years ago

@karim-agha probably has input here too.

tchataigner commented 2 years ago

Thanks for the feedbacks @raulk !

As discussed on Slack, the following proposition us meant for a first iteration of the SDK. As such we will focus on core implementation will limited flexibility for the user (on codecs for example).

Extension of current SDK

New SDK features

Procedural macros

3 procedural macros will be available for the user: fvm_state, fvm_actor and fvm_export

fvm_state

Description: Procedural macro used to specify implementation for the actor with exported functions available to call

Technical consideration

Parameters

Example

#[fvm_state]
struct ActorState {
  pub counter: u64
}

fvm_actor

Description: Procedural macro used to encompass methods available in the actor

Paremeters

💬 To be discussed @raulk made a proposition to create a system where a procedural macro fvm_export exists. In the macro it could be possible to specify the mutability of the state in the function:

I was wondering if it would be possible to consider the implementation responsible for method exposition as the implementation of the state, like so:

#[fvm_state]
struct ActorState {
  ...
}

#[fvm_actor]
impl ActorState {
  ...
}

This would actually allow to specify state manipulation by using either &mut self, &self or no reference to the state, making it easier for actors developers. The save and load part of the the state could then be generated along with glue code from the SDK.

fvm_export

Description: Procedural macro used to flag exposed functions in the actor

Parameter

Exmaple

impl ActorState {
  #[fvm_export(binding=1)]
  pub fn my_function(&self) -> u64 {
    ...
  }
}

Utilities

Questions

Having glue code generated means heavier actor bytecode on first deployment and more gas spent at runtime. Is it tolerable ?

Could you elaborate? In my head, the glue code would replace code that the user has to write manually today anyway.

It was just to point out that when writting an actor that would have generated glue code the wasm bytecode that would be generated might be heavier than what the developer expect. Thus leading to more data to be sent in the transaction leading to more expensive transaction. No real discussion here I think.

Open subject

FVM Payloads

Parameters and returned values from exposed functions will have to be serialized and deserialized to be received and sent to a caller. An actor developer will have to flag the different types with a given SDK proc macro (e.g.: fvm_payload). I think that for now we could focus on having serialization and deserialization handled with CBOR format. Would that be alright ?

raulk commented 2 years ago

Hey @tchataigner! 👋

  • dispatch: Internal dispatch method used to access exposed functions by the actor. Default and unique value is method-num.

This is good enough for a start, but I think it will fall short quickly as some form of call convention will probably prosper in the next weeks. Maybe we should consider making it pluggable from the get-go? This is hard though, because this component would need to have the full public interface description of the actor (method signatures), so it can select which method to dispatch to.

I was wondering if it would be possible to consider the implementation responsible for method exposition as the implementation of the state, like so:

Mind clarifying this? I'm not sure I followed. If what you meant is that we can do without the explicit transactionality annotation, and just rely on &mut, & or pass-by-value, that would work too at this stage.

I think that for now we could focus on having serialization and deserialization handled with CBOR format. Would that be alright ?

Yes.

Generally speaking, it would be nice to have flexibility to change some approaches on the fly as you guys implement the SDK. This is one of those things that you really need to get a hands-on feel for. On paper, some ideas may look good, but they ultimately may result unergonomic or unintuitive. So let's allow some room to play it by the ear too -- we don't want to lock ourselves up to very concrete solutions at this stage. The development of this SDK should feel more like a conversation with the users (Early Builders), core team, and others, instead of a one-way assignment. You should factor in time for demos and for feedback sessions at Early Builders meetings!

✅ The FVM engineering team signs off on scope of this. Missing deliverables and budget.

BlocksOnAChain commented 2 years ago

@tchataigner now we got tech approval, can we make sure all the details around deliverables and budget are listed in the RFP?

Tom-OriginStorage commented 2 years ago

Some user stories:

  1. As a user, I hope the usage of this high level Rust SDK would be similar to Cosmos's Cosmwasm or Solana's Anchor. It would help users to learn faster.
  2. As a user, I hope to see a standardize parameter format. Explanation below: 
    lotus chain invoke $ADDRESS $METHOD $PARAMS
// $PARAMS should be of a fixed format such as json encode base 64 instead of user's whim
// Even better if it's independent of an IDL, ABI to decode/encode $PARAMS
// Because this will help make Dapp integration & blockchain indexing more friendly and easier to stalk other users
  3. As a user, I hope to see @jimpick like tutorials.
tchataigner commented 2 years ago

All the parts of the grant request have been filled out and are ready for review !

raulk commented 2 years ago

Notes from convo with @tchataigner

tchataigner commented 2 years ago

Proposal updated after convo w/ @raulk & @BlocksOnAChain

raulk commented 2 years ago

@tchataigner Thanks for updating! As always, it's a pleasure to scope and plan out projects with you and Polyphene.

@realChainLife @DeveloperAlly @BlocksOnAChain I'm signing off on the work plan and the numbers.

realChainLife commented 2 years ago

Thanks @raulk! @tchataigner please email grants@filecoin.org as we would like to kick-off next steps now that we've finalized scoping the work proposed for the project.