search

ArweaveTeam / Bounties

Arweave sponsored Gitcoin bounties.
145 stars 47 forks source link

Clarity language support for SmartWeave #34

Closed cedriking closed 3 years ago

cedriking commented 4 years ago

Clarity language support for SmartWeave

SmartWeave is a new smart contracting system, built on top of the Arweave network. SmartWeave allows developers to deploy contracts that require significant amounts of computation without additional gas fees, by pushing the burden of contract execution to the users of the contract, rather than the network. Since its launch two months ago, numerous applications have started to use SmartWeave -- here are just a few of them.

Clarity is a smart contract programming language, built by Blockstack and Algorand, to provide a secure, predictable base for building in decentralised environments. One of the core issues facing SmartWeave developers is that while the use of JavaScript as the base programming language is flexible, it does not have many built in security features relevant for smart contract development.

This bounty is for building a mechanism of writing contracts for SmartWeave using the Clarity language. This should be achieved by essentially providing a 'compiler' that takes a Clarity program, and outputs a JavaScript WASM bundle that executes the smart contract code when called. A WASM version of the Clarity VM already exists, so the task here is mainly to integrate and provide the 'glue' that links the projects together.

Application requirements

The application shall facilitate the means of compiling and bundling a smart contract written in Clarity, in such a way that it can be deployed as a SmartWeave contract.

It must fulfill the following criteria:

  1. The application should take a program written in Clarity and compile it to WASM
  2. Function calls made in the program via contract calls must be appropriately routed in the resulting SmartWeave-compatible output code
  3. The resulting output must be a valid SmartWeave contract, deployable to the Arweave network, and successfully be interacted with
  4. Errors in Clarity must propagate in SmartWeave compatible terms

Rules

Successful submissions will meet the following criteria:

  1. Submissions must be your own original work.
  2. Submissions must be open source, with the full source code available on Github or another open code hosting repository. Feel free to use whatever OSS license you prefer.
  3. All submissions must include a brief description of the application and functionality in the GitHub repo

Resources

Need to learn more about Arweave technology? Use the resources below, or reach out to the support team on the discord channel:

SmartWeave

Arweave Dev Discord

Get an Arweave Wallet & Free Tokens

Arweave Web Extension Wallet

Developer Documentation & Getting Started Guides

Submission Deadline

The hackathon will take place from September 8th to October 7th. The last day for the selected applicant to submit their work is October 7th @ 23:59 PM PST.

gitcoinbot commented 4 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 7500.0 USDC (7500.0 USD @ $1.0/USDC) attached to it as part of the arweaveteam fund.

luciotato commented 4 years ago

I have experience writing transpilers. I can deliver a transpiler that parses Clarity and generates a SmartWeave-compatible WASM Javascript package, or AssemblyScript if you prefer: https://www.assemblyscript.org/

I can manually code a PEG parser which ensures better extensibility and useful and accurate compiler errors.

We can also explore the option of generating WebAssembly (WAT) text files which, even if they are low level, are based on s-expressions like Clarity. https://developer.mozilla.org/en-US/docs/WebAssembly/Text_format_to_wasm

Here's a repo with my own transpiler for a python-like language of my design. It can output javascript or C code from the same source: https://github.com/luciotato/LiteScript

I also have a transpiler that I recently wrote in Typecript specifically for smart contracts. This transpiler converts smart contract code from rust (NEAR) to AssemblyScript. See: https://github.com/near/core-contracts-as/issues/2#issuecomment-684090990

The code is self-documenting, easy to understand and to extend.

This is an example of the rust parser; how to parse a fn declaration:

//`FunctionDeclaration: 'fn [IDENTIFIER] ["(" [VariableDecl,]* ")"] [ -> TypeAnnotation ] Body`

export class FunctionDeclaration extends ASTBase {

    paramsDeclarations: FunctionParameters

    // ---------------------------
    parse() {

        //manage special keywords like 'pub'
        this.optPub()

        this.req('fn') //require the token 'fn'
        this.lock() 
        this.name = this.reqToken(TokenCode.WORD)

        //get parameters declarations
        this.paramsDeclarations = this.opt(FunctionParameters) as FunctionParameters

        // get the return-type (optional)
        if (this.opt('->')) {
            this.typeAnnotation = this.reqClass(TypeAnnotation)
        }

        //now parse the body
        if (this.owner.lexer.token.value == ";") {
            //just a fn signature declaration (no body)
            return
        }
        else {
            this.req("{")
            Body.parseIntoChildren(this)
        }
    }
}
// end class FunctionDeclaration

The parser generates an AST, and then you can connect a "producer" to each AST node class, to output transpiled code, This is the producer to output TypeScript from a parsed rust's function declaration AST node

class FunctionDeclarationWriter extends Grammar.FunctionDeclaration {
    produceTS() {
        const o = this.owner.codeWriter
        if (this.isPublic) o.write("export ")
        o.write("function ")
        o.write(this.name)

        //param decl
        o.write("(")
        let inx = 0
        for (const paramDecl of this.paramsDeclarations.children) {
            if (paramDecl.name != 'self') { //rust 'self' is implicit 'this' in ts/assemblyscript
                if (inx > 0) o.write(", ")
                paramDecl.produce()
                inx++
            }
        }
        o.write(")")
        //end param decl

        //Type Annotation
        this.typeAnnotation?.produce()

        //Body
        if (this.children.length) {
            o.write(' {')
            BodyWriter.produce()
            o.writeLine('}')
        }
    }
}
vird commented 4 years ago

I will upgrade my existing solution https://github.com/vird/solidity2assemblyscript with clarity parser (I'm interested is there ready-to-use tool for getting clarity->AST, if can't find, then I will craft my own parser with https://github.com/hu2prod/lang_gen (parser constructor) or will use pegjs)

I'm author of related projects transpiler solidity to Ligo (tezos) https://github.com/madfish-solutions/sol2ligo/ transpiler solidity to smartpy (tezos) (old online version is here https://vird.github.io/eth2tez_online/smartpy.html) https://github.com/vird/solidity2near_rust (online versions https://vird.github.io/solidity2assemblyscript_online/)

Also I was active bug reporter for assemblyscript https://github.com/AssemblyScript/assemblyscript/issues?q=is%3Aissue+author%3Avird Also I have some experience with translation to multiple different programming lamguages https://github.com/hu2prod/ast2cpp https://github.com/hu2prod/ast2coffee https://github.com/hu2prod/ast2rust https://github.com/hu2prod/ast2java

Also I have experience with WASM https://github.com/hu2prod/wasm (hotreload for wasm modules) https://github.com/hu2prod/wasm_runtime (own array, hashmap, memory allocator implementation for wasm)

I think I can try implement something good

UPD: seems clarity doesn't require sophisticated parser, so I can write it even without extra libraries (for a purpose: best error messages possible). Also it's probably possible to produce WAST (https://developer.mozilla.org/ru/docs/WebAssembly/Understanding_the_text_format) directly, without intermediate assemblyscript

UPD2: Specs for clarity is relatively clear. Where can I find WASM examples or specs for SmartWeave (expecially external interactions documentation and WASM corresponding examples)?

vird commented 4 years ago

As far as I understand. You want that some 3rdparty developer will develop transpilerfor you and you will not provide explicitly any spec for WASM smart-contract-specific API endpoints. Ok I can even guess how it should work and select some type sizes and just wrap existing stuff

But there is no single standatd about async calls from WASM https://github.com/ArweaveTeam/SmartWeave/blob/master/examples/test-api-things.js#L16

I found at least 3 different proposals, which are not standard, and are not very suitable for this task

  1. https://stackoverflow.com/questions/50731650/is-it-possible-to-run-webassembly-code-async (wasm threads are not standard)
  2. https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html (will work, but it requires a lot more effort if I will generate WAST as low-level as possible, otherwise I need translate clarity to C/assemblyscript and then to WASM, also this will introduce a lot more wrapper code for that, and I preffer/vote not to do this way even if it works)
  3. https://github.com/hyperdivision/async-wasm (this is only suitable for call wasm from js in async manner)
gitcoinbot commented 4 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 3 weeks, 1 day from now. Please review their action plans below:

1) luciotato has applied to start work _(Funders only: approve worker | reject worker)_.

I have experience transpiling code. I can deliver a transpiler parsing Clarity and outputting a Javascript WASM bundle or AssemblyScript https://www.assemblyscript.org/ if you prefer. Here's a repo with my own transpiler converting an phyton-like language of my design into javascript or C https://github.com/luciotato/LiteScript I have also a transpiler I recently write in Typescript specifically for smartcontracts. This transpiler converts rust smart contract code (NEAR) into AssemblyScript. See: https://github.com/near/core-contracts-as/issues/2#issuecomment-684090990 2) evgeniuz has applied to start work _(Funders only: approve worker | reject worker)_.

This issue is really interesting, I'm interested in diving deeper into SmartWeave. So far my plan is to "extend" Clarity VM with "glue" to allow interacting with the network (for example reading/making contract calls etc. from Clarity) and then implement toolchain to compile Clarity for this extended environment. I also think it's possible to leverage Arweave here and store this VM on Arweave globally, so that Clarity contracts only need to reference specific version they would like to use instead of deploying whole VM with each contract.

I previously worked on adding Vyper support to Truffle, which is kind of similar task, i. e. using another compiler to work within existing ecosystem, so I think this might help a bit here. 3) vird has applied to start work _(Funders only: approve worker | reject worker)_.

I will upgrade my existing solution https://github.com/vird/solidity2assemblyscript with clarity parser (I'm interested is there ready-to-use tool for getting clarity->AST, if can't find, then I will craft my own parser with https://github.com/hu2prod/lang_gen (parser constructor) or will use pegjs)

I'm author of related projects transpiler solidity to Ligo (tezos) https://github.com/madfish-solutions/sol2ligo/ transpiler solidity to smartpy (tezos) (old online version is here https://vird.github.io/eth2tez_online/smartpy.html) https://github.com/vird/solidity2near_rust (online versions https://vird.github.io/solidity2assemblyscript_online/) Also I was active bug reporter for assemblyscript https://github.com/AssemblyScript/assemblyscript/issues?q=is%3Aissue+author%3Avird

Also I have some experience with translation to multiple different programming lamguages https://github.com/hu2prod/ast2cpp https://github.com/hu2prod/ast2coffee https://github.com/hu2prod/ast2rust https://github.com/hu2prod/ast2java

Also I have experience with WASM https://github.com/hu2prod/wasm (hotreload for wasm modules) https://github.com/hu2prod/wasm_runtime (own array, hashmap, memory allocator implementation for wasm)

I think I can try implement something good 4) vporton has applied to start work _(Funders only: approve worker | reject worker)_.

I will write the JS/WASM bundler in D Programming Language (the first option, because it is high performance, easy to develop, and reliable), or if it fails for some reason, then in Python.

I already had experience with Clarity.

I also already know SmartWeave.

For simplicity, I will create (in D) a relaxed parser of Clarity not following all the syntax rules. 5) artob has been approved to start work.

I have created a prototype called Clarc (for "Clarity compiler") that is able to already compile simple contracts such as the Clarity example project counter.clar to WebAssembly as well as directly to SmartWeave-compatible JavaScript:

https://asciinema.org/a/2G3MG4c6CrSGPDpyQO4iNR8tn

The SmartWeave JS output is immediately usable without the need to integrate any WebAssembly runtime into SmartWeave:

https://asciinema.org/a/pWvCS7VZW8MvrFhRIpKbgeyUN

Clarc will be delivered as a standalone, dependency-free native executable for each desktop platform (Windows, Mac, and Linux) and does not need any third-party tooling whatsoever (in particular, no AssemblyScript installation needed).

Clarc itself can also be transpiled into JavaScript, meaning the compiler can be easily embedded in web IDEs for SmartWeave.

Clarc is written in OCaml, uses the standard WebAssembly reference implementation (which, happily, is also written in OCaml) for WebAssembly code generation, and uses the world-class ReScript (formerly known as BuckleScript) toolchain libraries for JavaScript code generation.

Learn more on the Gitcoin Issue Details page.

gitcoinbot commented 4 years ago

@artob Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

artob commented 4 years ago

After discussion with the Arweave team, my compiler prototype was renamed to Sworn. It is now published at https://github.com/weavery/sworn and under active development. For updates, please follow the #smartweave channel on the Arweave Discord.

artob commented 4 years ago

I've just released Sworn 0.2.0, with some good progress. With 34 commits since 0.1.0, this is the changelog summary:

Note that my current development focus is on the JavaScript code generation, so the WebAssembly target doesn't have support for all these features as yet.

I expect that Sworn 0.3.0, later this week, will be the first alpha-quality release that it makes sense for people to play with.

I've also begun working on Clarity.js, the runtime support library that contracts making use of certain Clarity features (128-bit arithmetic, hash functions such as SHA-256, etc) will need to import. Sworn 0.3.0 will support (and indeed, require) this library.

@cedriking I've posted important considerations for SmartWeave API/ABI stability to #smartweave on Discord, please read when you have a chance.

gitcoinbot commented 4 years ago

@artob Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

artob commented 4 years ago

I've just released Clarity.js 0.1.0, which provides runtime support for Clarity contracts on SmartWeave.

Clarity.js is now feature complete for the core language constructs, with the next step being to actually integrate it into SmartWeave to finish the implementation of all blockchain-related Clarity standard library functions: block-height, tx-sender, etc, can't be implemented in the abstract.

Sworn 0.3.0, to be released soon, will require Clarity.js for the generated JavaScript contracts.

artob commented 4 years ago

I've just released Sworn 0.3.0. Sworn has now reached an alpha stage, about 80% of the way to the 1.0 release. With 29 commits since 0.2.0, this is the changelog summary:

I expect the JavaScript code generation to reach stability during this week. After that, I will continue work on the WebAssembly target as well. (Though as discussed with @samcamwilliams, the JavaScript target may ultimately be more valuable to Arweave since it enables the inspection and facilitates the audit of contract code. The Stacks Foundation interprets Clarity code instead of compiling it, for the same reason.)

Beyond iterating on the JS target and wrapping up odds and ends for the full language support, my main focus for this week is the Clarity.js runtime integration with SmartWeave, for which I'll need @cedriking's support. I'm prepping the first pull requests now.

artob commented 4 years ago

I've submitted the pull requests https://github.com/ArweaveTeam/SmartWeave/pull/36 and https://github.com/ArweaveTeam/SmartWeave/pull/37 to integrate the Clarity.js runtime support into SmartWeave. Once merged, it will be possible to run SmartWeave contracts generated by Sworn.

artob commented 4 years ago

@cedriking Note also the https://github.com/ArweaveTeam/SmartWeave/pull/38 pull request, which needs to be merged to complete the SmartWeave integration for Clarity.js.

cedriking commented 4 years ago

Amazing @artob , thanks a lot! Amazing work 👍

artob commented 4 years ago

I've just released Sworn 0.4.0. Sworn has now reached a beta stage, with the JavaScript target substantially completed.

The remaining bits still to do overall for full Clarity support in SmartWeave are blocked on discussion with and decisions by the Arweave team, as well as required further developments in SmartWeave and Clarity.js. The Sworn compiler itself supports the full language. (Modulo my understanding of the language, since there is no upstream specification for Clarity at this moment.)

With 29 commits since 0.3.0, this is the changelog summary:

Note that much of the work over the past week has taken place outside the Sworn repository per se, in the two spin-off projects Clarity.ml and Clarity.js as well as in SmartWeave itself.

Clarity.ml is the Clarity parser and abstract syntax tree (AST) representation for OCaml. I've spun it out as a standalone OCaml library so that anyone familiar with OCaml can quickly and easily develop more best-of-class tooling for Clarity contracts.

Clarity.js, mentioned here previously, is the JavaScript runtime for Clarity contracts. It implements Clarity's standard library using TypeScript, compiled into ES6 for Node and web projects to pull in.

SmartWeave now integrates Clarity.js and is thus able to run Clarity.js contracts, as produced by Sworn, out of the box. (It would probably be a good idea to tag and release a SmartWeave 0.4 version incorporating all that, @cedriking?)

My focus now is on further developing the Sworn, Clarity.ml, and Clarity.js test suites, as well as on making sure Clarity contracts seen in the wild work on SmartWeave (modulo any needed changes for porting the contract from Blockstack to Arweave). Additionally, I'll work on building Windows, macOS, and Linux binaries of Sworn in preparation for the Sworn 1.0 release.

artob commented 4 years ago

To summarize some of the questions we discussed on a call with @jespern earlier today, in preparation for the Sworn 1.0 release:

  1. Clarity's Blockstack-specific stx-burn?, stx-get-balance, and stx-transfer? functions will, unsurprisingly, not be supported.

  2. It's not at all clear how we could map Clarity's trait system (contract-of, define-trait, impl-trait, use-trait) to SmartWeave, so that won't be supported.

  3. Though Clarity's at-block and get-block-info? could be supported in principle, we probably don't want to support them since that would require the SmartWeave executor having access to block metadata for all Arweave blocks. That in turn requires either network access for querying a full node, or else at least having and maintaining a block metadata index. Neither option is palatable at the moment, so these functions won't be supported for now.

  4. An open question is Clarity's built-in fungible/non-fungible token support (the ft- and nft- functions): they obviously can't map to actual AR transfers since SmartWeave contracts can't issue new transactions. It might be possible to map the ft- functions to PSTs, though. Needs further discussion with @cedriking and @samcamwilliams as well.

  5. In terms of future directions, we should define pst- and psc- functions for Clarity contracts that vastly simplify implementing PSTs and PSCs along the lines of Clarity's fungible/non-fungible token support. With these kinds of primitives, issuing a new PST could be a really simple Clarity contract in the common case.

  6. In terms of future directions, we should afford Clarity contracts access to the global SmartWeave object in some way, so that Clarity contracts will be on an equal footing with SmartWeave contracts written directly in JavaScript.

  7. The WebAssembly target is a very low priority, given that the JavaScript target produces nicely-formatted, human-readable contracts that are possible to understand and audit.

gitcoinbot commented 4 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 7500.0 USDC (7500.0 USD @ $1.0/USDC) has been submitted by:

artob commented 3 years ago

@cedriking When you're back, would you please release the funds for the first milestone on Gitcoin, as okayed by @jespern and @samcamwilliams.

gitcoinbot commented 3 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of 7500.0 USDC (7500.0 USD @ $1.0/USDC) attached to this issue has been approved & issued to @artob.