Closed tchataigner closed 1 year ago
Hey @BlocksOnAChain, just opened the PR as we discussed. It is still in a draft phase but I need review on the Project description part where the specifications need some iterations!
Hope we can get to a nice result together :)
@tchataigner do we plan details around Deliverables and Funding since this is something we definitely need if we want to go towards the approval stage for the RFP? Other items look ok to me, we just need to improve the draft we started from with more details. cc: @raulk @DeveloperAlly
We can also perhaps add the purpose of this sdk - being an abstraction and extension of functionality to the current FVM Rust implementation aiming to provide a better developer experience for those building on FVM and enabling foreign runtime bridge options.
Potential Milestones (we like to only have 2-3) can probably come from your set of ideal behaviours mentioned. Let me know if you want some help with developing the milestones above @tchataigner .
In terms of the funding amount to request - if you don't have a clear idea - @eshon will be able to help. Do you perhaps have a napkin calculation or parameters/formula you use and would be able to share in this regard @eshon ?
Otherwise - let's build this! :P
Hello from the grants team. I'll leave the SDK design questions to the FVM team.
For now, we'd like to see more clarity around the following topic:
Once you have more clarity around some of the design questions (eg how serialization calls should work), ping me if you'd like help planning the milestones and deliverables sections.
@tchataigner do we plan details around Deliverables and Funding since this is something we definitely need if we want to go towards the approval stage for the RFP? Other items look ok to me, we just need to improve the draft we started from with more details. cc: @raulk @DeveloperAlly
Otherwise - let's build this! :P
Hey everyone thank you for the feedbacks ! I know that the draft is not ready to be approved yet but I would need to clarify some blurry parts raised in the proposed specifications before filling the Deliverables and Funding parts. I guess someone from the FVM team would be best (@raulk would be the best choice I guess).
The parts are the following: Glue code generation and internal dispatch
invoke()
then how can the user and future callers know which function are associated with which index ?Serialize_tuple
and Deserialize_tuple
. Would that be alright ?Extension of low level SDK
ref-fvm/fvm_sdk
or ref-fvm/fvm_shared
. Also I do not see an easy way of doing so without adding another syscall so that might have to be discussed and maybe excluded of the scope.What documentation, education, and/or community-building is needed for users to effectively learn & use the SDK? At minimum for an SDK project there should be well-documented code and an excellent README. A short video demo of usage (~5 mins) would be good to have as well.
Thanks for the feedback @mishmosh ! That is interesting! I guess that we should think of documentation for the SDK as a part for the documentation of the FVM (like the Go SDK for IPFS is part of the documentation of IPFS). Is there any plan around that? If none we could use Docusaurus to create a static website to document the SDK.
I feel like having a video is not really part of the documentation as it falls in a overall ramp access to the FVM from my point of view. For the video we could have a presentation in the FVM check-in and have it recorded what do you say.
@tchataigner I am so sorry for the delay here. We were heads down on shipping the M1 development freeze, and then I was OOO for a few days. I'm looking through this now.
Ideal behaviours
From those sources came out some ideal features to implement:
From current actors
- Remove the need for the user to code the invoke function
👍 I'd enhance this and say that it should support multiple call conventions, where the method number is one of them. It could be the default so it could be omitted, but I'd stay away from doing so to train developers to think about call conventions explicitly. The snippet at the top of fvm.filecoin.io demostrates how I was thinking about specifying the call convention:
#[fvm_actor(state=ComputeState, dispatch="method_num")]
This attribute won't be enough though. We'll need per-method annotations too to define the binding to the call convention. For example, in that example:
/// Creates a job with an input DAG, WASM compute logic,
/// data affinity, geographical bounds, and timeout.
#[fvm_export(binding=1)]
pub fn create_job(req: CreateJobReq, st: ComputeState)
-> CreateJobRes { ... }
/// Allows a compute node to claim a job by proving it
/// satisfies the requirements and staking collateral.
#[fvm_export(binding=2)]
pub fn claim_job(req: ClaimJobReq, st: ComputeState)
-> ClaimJobRes { ... }
/// Proves that a compute node finished running a job,
/// posts the result, and claims the reward.
#[fvm_export(binding=3)]
pub fn prove_done(req: ProveDoneReq, st: ComputeState)
-> ProveDoneRes { ... }
}
This would automatically generate the invoke entrypoint and the routing table according to the dispatch strategy and the bindings. Maybe the binding should be some kind of "any" type, because values will be specific to the call convention (i.e. dispatch strategy). Imagine for example a Solidity-like call convention (N-truncated bytes from hashed target method signature).
We don't need to implement it now, but users should be able to provide their own dispatch strategy down the line. (Note: eventually we want to phase out message-defined method numbers).
- Remove the need for the user to handle serde of payloads
Yes! Big +1 to this. To make this future-proof, we'll need to define the IPLD codec that's being used. I think this is best defined at two levels:
#[fvm_actor(codec=DagCbor)]
#[fvm_export(binding=3, codec=DagPb)]
, which can be used to override the top-level codec?That combination would apply DagCbor serde to parameters and return values on all methods except for the one being annotated with DagPb.
- Ease State structure declaration in actors and auto generate save() and load() functions
Big +1 to this, but there's a little bit more here:
A more complex actor example is the old/outdated ERC20 token actor: https://github.com/filecoin-project/ref-fvm/pull/290. Take a look at the docs, concretely the "Boilterplate" section, as well as the review comments for ideas from @Stebalien (most of which are similar to what I've proposed here, as he and I have refined this thinking together over time).
I think #[fvm_export]
should define the transaction semantics for a method and it should be used in combination with the mutability of the state argument of the method. For example:
#[fvm_export(tx=ro)]
fn hey(state: &State) { } // immutable state, can never be upgraded to a transaction
#[fvm_export(tx=rw)]
fn hey(state: &State) { } // immutable state, can be upgraded to a transaction
#[fvm_export(tx=rw)]
fn hey(state: &mut State) { } // mutable state, boilerplate manages the transaction
- Some low-level SDK syscall should be available to the user if they are deemed interesting (crypto ...)
I think all low-level SDK syscalls should be available to the user. I'd take a porcelain/plumbing approach there, where the user is expected to use the porcelain (high-level part), but if they need to do something less common, they can fall back to the plumbing (low-level part). These terms are taken from Git nomenclature.
- Have access to basic types that could be useful for actor development
Yep. Maybe this SDK could depend on the existing low-level SDK and re-export such types?
From other languages
- Have access to block & tx properties
Sure! Adding these syscalls would be beyond just the SDK -- they'd need to be implemented on the kernel side. Are you good with including that in the scope too? I would be very open. Although because this would imply a protocol change, I'd first start discussions in the FIPs repo: https://github.com/filecoin-project/FIPs/discussions, proposing the concrete syscalls, gathering feedback, and opening issues in ref-fvm to implement if there's consensus.
- Implement error handling function such as _assert__* for the user to use in their contract
I like this idea. In fact, I wonder if it makes sense to have declarative validation like in https://github.com/Keats/validator. In case of failure, you'd abort with exit code 24 (USR_ASSERTION_FAILED
). See https://github.com/filecoin-project/fvm-specs/blob/main/07-errors.md.
- Have access to the deployed actor information (address ...)
You mean like the receiver address? (This is available through the vm::context()
syscall).
From specifications
potentially mapping dynamically-linked libraries (e.g. predefined SDK versions)
- Means that the SDK core module shall not be within the actor base code but linked to it. Reduce actors’ size.
This would be awesome, although it's a significant endeavour (and a project of its own). We envision some form of a "library registry" actor where users can push non-instantiatable (and only importable) pieces of code. The right way to do this would be through content addressing, i.e. importing a CodeCID from the actor and having the FVM resolve that for you. It's not simple and I would recommend not including it in the scope of this. This is likely FVM M3+ material.
TBC Draft Specifications
Note: These are the proposed implementation choices to be developed. It is still to be discussed before considering a roadmap.
Mandatory elements in an actor
Entry point
invoke()
: entry point for the actor, contains a map to dispatch call to proper method based on its number.
Note that this is the Filecoin standard call convention (there will be other call conventions going forward).
State
- Struct that derives
Serialize_tuple
andDeserialize_tuple
- Should have a
save()
andload()
implementation from a trait. Proposal forStateObject
trait.
I think the state struct should be annotated with an attribute as well that specifies the serialization codec (Cbor) and style (tuple): #[fvm_state(codec = Cbor, style = Tuple)]
?
Proposed implementation
- If we generate
invoke()
then how can the user and future callers know which function are associated with which index ?
This would be specified in #[fvm_export]
?
I was thinking of having something that would look like that as an actor:
...
But it would mean that all types as parameters and/or returned from functions should implement
Serialize_tuple
and/orDeserialize_tuple
otherwise generated code would not work. Is it alright? Should we do it another way?
Can we do this automatically by adding the attributes to the params and return types? (Might be hard though, I'm also OK with the user having to add attributes to the types, e.g. #[fvm_transferrable]
for transferrable types (params and return)?
- Having glue code generated means heavier actor bytecode on first deployment and more gas spent at runtime. Is it tolerable ?
Could you elaborate? In my head, the glue code would replace code that the user has to write manually today anyway.
@tchataigner One more requirement for the SDK is that it should automatically handle panics by catching them and lowering them to an abort. See https://github.com/filecoin-project/builtin-actors/blob/master/actors/runtime/src/runtime/fvm.rs#L552-L554.
Generally speaking, the built-in actors runtime is decent inspiration for this grant.
@Stebalien probably has some input here.
@karim-agha probably has input here too.
Thanks for the feedbacks @raulk !
As discussed on Slack, the following proposition us meant for a first iteration of the SDK. As such we will focus on core implementation will limited flexibility for the user (on codecs for example).
ref-fvm
.ref-fvm
SDK.3 procedural macros will be available for the user: fvm_state
, fvm_actor
and fvm_export
fvm_state
Description: Procedural macro used to specify implementation for the actor with exported functions available to call
Technical consideration
Serialize_tuple
and Deserialize_tuple
Copy
, Clone
, Eq
, PartialEq
, Ord
, PartialOrd
, Hash
, Debug
, Display
, Default
) as per Rust API GuidelinesParameters
codec
: IPLD codec used to store the state of the actor. Default and unique value is DAG-CBOR.
Note: This could be extended in the future to support more codecsdispatch
: Internal dispatch method used to access exposed functions by the actor. Default and unique value is method-num
.
Note: This will have to be extended depending on the conclusion of the discussion about the FVM native calling convention proposalExample
#[fvm_state]
struct ActorState {
pub counter: u64
}
fvm_actor
Description: Procedural macro used to encompass methods available in the actor
Paremeters
state
: Structure that represent the state of an actordiscpatch
: Internal dispatch type used in the actor. Default and unique value is method-num
💬 To be discussed
@raulk made a proposition to create a system where a procedural macro fvm_export
exists. In the macro it could be possible to specify the mutability of the state in the function:
I was wondering if it would be possible to consider the implementation responsible for method exposition as the implementation of the state, like so:
#[fvm_state]
struct ActorState {
...
}
#[fvm_actor]
impl ActorState {
...
}
This would actually allow to specify state manipulation by using either &mut self
, &self
or no reference to the state, making it easier for actors developers. The save
and load
part of the the state could then be generated along with glue code from the SDK.
fvm_export
Description: Procedural macro used to flag exposed functions in the actor
Parameter
binding
; Entry point used in the dispatch method for the actor. Numerical value for now as method-num
is the default and unique valueExmaple
impl ActorState {
#[fvm_export(binding=1)]
pub fn my_function(&self) -> u64 {
...
}
}
assert_*
tests to the user as to help him test values in the actor. Current ref-fvm
SDK could be extended to access those macros.Having glue code generated means heavier actor bytecode on first deployment and more gas spent at runtime. Is it tolerable ?
Could you elaborate? In my head, the glue code would replace code that the user has to write manually today anyway.
It was just to point out that when writting an actor that would have generated glue code the wasm bytecode that would be generated might be heavier than what the developer expect. Thus leading to more data to be sent in the transaction leading to more expensive transaction. No real discussion here I think.
Parameters and returned values from exposed functions will have to be serialized and deserialized to be received and sent to a caller. An actor developer will have to flag the different types with a given SDK proc macro (e.g.: fvm_payload
). I think that for now we could focus on having serialization and deserialization handled with CBOR format. Would that be alright ?
Hey @tchataigner! 👋
dispatch
: Internal dispatch method used to access exposed functions by the actor. Default and unique value ismethod-num
.
This is good enough for a start, but I think it will fall short quickly as some form of call convention will probably prosper in the next weeks. Maybe we should consider making it pluggable from the get-go? This is hard though, because this component would need to have the full public interface description of the actor (method signatures), so it can select which method to dispatch to.
I was wondering if it would be possible to consider the implementation responsible for method exposition as the implementation of the state, like so:
Mind clarifying this? I'm not sure I followed. If what you meant is that we can do without the explicit transactionality annotation, and just rely on &mut
, &
or pass-by-value, that would work too at this stage.
I think that for now we could focus on having serialization and deserialization handled with CBOR format. Would that be alright ?
Yes.
Generally speaking, it would be nice to have flexibility to change some approaches on the fly as you guys implement the SDK. This is one of those things that you really need to get a hands-on feel for. On paper, some ideas may look good, but they ultimately may result unergonomic or unintuitive. So let's allow some room to play it by the ear too -- we don't want to lock ourselves up to very concrete solutions at this stage. The development of this SDK should feel more like a conversation with the users (Early Builders), core team, and others, instead of a one-way assignment. You should factor in time for demos and for feedback sessions at Early Builders meetings!
✅ The FVM engineering team signs off on scope of this. Missing deliverables and budget.
@tchataigner now we got tech approval, can we make sure all the details around deliverables and budget are listed in the RFP?
Some user stories:
lotus chain invoke $ADDRESS $METHOD $PARAMS
// $PARAMS should be of a fixed format such as json encode base 64 instead of user's whim
// Even better if it's independent of an IDL, ABI to decode/encode $PARAMS
// Because this will help make Dapp integration & blockchain indexing more friendly and easier to stalk other users
All the parts of the grant request have been filled out and are ready for review !
Proposal updated after convo w/ @raulk & @BlocksOnAChain
@tchataigner Thanks for updating! As always, it's a pleasure to scope and plan out projects with you and Polyphene.
@realChainLife @DeveloperAlly @BlocksOnAChain I'm signing off on the work plan and the numbers.
Thanks @raulk! @tchataigner please email grants@filecoin.org as we would like to kick-off next steps now that we've finalized scoping the work proposed for the project.
RFP Proposal: FVM - High level Rust SDK
Name of Project: FVM - High level Rust SDK
Link to RFP: Please link to the RFP that you are submitting a proposal for.
RFP Category:
devtools-libraries
Proposer: @tchataigner
Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT, GPL, and APACHE2 licenses?: Yes
Project Description
References
In the context of the FVM Early Builder program, @BlocksOnAChain suggested that we could handle the creation of a High level Rust SDK to help with the development of Rust-native actors for the FVM.
The following specifications are meant to produce a Rust crate that could be the only
fvm_*
crate-related import to produce valid code. The different specifications have multiple references:Development Roadmap & Deliverables
Pre-Requirement
Milestone 1 - SDK implementation
This milestone has for objective to release the SDK in its
0.1-beta
version.M1.1 - Extension of current low-level SDK
Technical scope:
ref-fvm
.ref-fvm
SDK.assert_*
macro available for testing in the actors.Deliverables:
0.0.1
M1.2 -
fvm_state
proc macroTechnical scope:
Creation of a procedural macro to declare the structure representing the internal state of an actor.
save()
andload()
implementation for the structure representing the state, allowing for state transition and retrieval.Serialize_tuple
andDeserialize_tuple
Copy
,Clone
,Eq
,PartialEq
,Ord
,PartialOrd
,Hash
,Debug
,Display
,Default
) as per Rust API GuidelinesParameters
codec
: IPLD codec used to store the state of the actor. Default and unique value is DAG-CBOR. Note: This could be extended in the future to support more codecsdispatch
: Internal dispatch method used to access exposed functions by the actor. Default and unique value ismethod-num
. Note: This will have to be extended depending on the conclusion of the discussion about the FVM native calling convention proposalDeliverables:
0.0.2
M1.3 -
fvm_actor
proc macroTechnical scope:
Creation of a procedural macro used to encompass methods available in the actor.
invoke()
function, entry point of basic actors and internal dispatcher to execute the right method.Paremeters
state
: Structure that represent the state of an actor. Default value is the structure being implemented.dispatch
: Internal dispatch type used in the actor. Default and unique value ismethod-num
Deliverables:
0.0.3
M1.4 -
fvm_export
proc macroTechnical scope:
Creation of a procedural macro used to flag exposed functions in the actor.
Parameter
binding
; Entry point used in the dispatch method for the actor. Numerical value for now asmethod-num
is the default and unique valueDeliverables:
0.1.0-alpha
README.md
allowing for external developers to handle the SDK and try to experiment with it.Milestone 2 - Community feedback integration
This milestone has for objective to iterate with the community (most likely the FVM EBP participants) to update the SDK according to their feedback.
M2.1 - Aggregate and report community feedback
During this milestone, the Polyphene team will not be actively working on the SDK but will still be available to answer messages and questions from testers.
Deliverables:
M2.2 - Feedback integration
Deliverables:
1.0
Milestone 3 - Miscellaneous
Deliverables:
Roadmap
The proposed timeline below would allow for the development of the discussed features with the integration of a review and community discussion phase by the end of August 2022. A split with other dates would make it more difficult for the proposed development team to meet this deadline.
Removal of PM
After discussing with Raul it was decided try try to do the project without a PM. The main reason is that we will be developing during summer which should reduce number of feedbacks. Consequences are:
New Roadmap
Note: Feedback will be done through Github issues on the development repository helped by a template defined by Polyphene.
Pricing
Total: ( 2 $1.5k ) 4w 5d + $1.5k 2w * 5d == $75k
To reflect the existing relationship of trust between PL and Polyphene, and the opportunity given in the choice of milestone dates to be set by the development team, Polyphene would agree for this grant to a 15% discount on its rates.
Final pricing: $75k * 85% == $63,75k
Maintenance and Upgrade Plans
We have no plan on maintaining the SDK in this proposal. This should be arranged either at a later time or in another program.
Team
Contact Info
On Filecoin slack:
Team Members
Team Member LinkedIn Profiles
Team Website
https://polyphene.io
Relevant Experience
We already have some experience in handling wasm runtime and development of tooling around it. We spent last year designing and building the Holium project. It allowed us to hone our skills to build rust-based protocols and libraries.
Moreover, we followed the FVM specification and development since its premises. We also participated in its development by helping on the creation of an integration test framework.
Team code repositories
Holium Rust
Holium Rust SDK