[idea] web3js compatible tooling for solidity devs on substrate

[extraymond]

Info

description: This is a idea shared within my organization, some background here. We're a team working on substrate that decided to use pallet-contracts as our smart contract layer for the diverse support of programming languages, especially solang for solidity. The above might not apply for most solang users on substrate, but it's something that might benefit most of us to have some shared tooling that have a familiar abi for eth devs coming after solang.

target audience: Two types of users having similar problem that don't want to give up web3js knowledges. I'll emphasise more on type 1 which is for the case for my org.

1. want to use web3.js on all chain operations for eth-rpc compatible chain
2. want to use web3.js on all contract related operations.

web3.js for eth-rpc compatible substrate chain

While we have some success creating solang based contracts just like the integration test, the devs are facing great difficulty when dealing with most of the dev journey other than compiling, such as testing, deploying in batch and so on, they are familiar with web3.js and ether.js and to some extend truffle and such. This also apples to the frontend team that are more familiar with web3.js and ether.js. While polkadot.js is really powerful, it's something exotic and alien to solidity devs, the semantics around the api is less intuitive for those coming from ethereum.

As a response, our team decides to use substrate's self_contained_call feature and allows converting ethereum transaction payload into substrate's pallet-contract call format. On this foundation, we've currently built a partially working ethereum rpc compatible node that can trigger pallet-contracts calls underneath, which ultimately powered by solang for solidity based programs.

While it's now possible to trigger call/create transaction using ethereum rpc now(web3.js/ethere.js), since the ABI format is different between solc and solang(pallet-contracts metadata), we cannot use most of the modules available on web3.js and ether.js

current drawbacks: the pseudo process on using the rpc-compatible-relayer and pallet-contract based contract is as follows:

1. use web3.js to connect to the rpc.
2. use polkadot.js to load the contract metadata, and use it to encode the arguments into raw input bytes.
3. manually create eth transaction body with the call input derived on step2.
4. sign the transaction using the signer(metamask).
5. send the signed transaction to the client for it to trigger the proper response.

graph LR
A[solang] -->|compiled| C[polkadot.js]
C -->|encode selector| D[web3.js]
D -->|manual payload| E[unsigned payload]

For this particular setup, we can recognize that the use of polkadot.js is very minimal and only used to load and prepare the selector. But having to learn all this for the node is unfortunately a overhead for devs not familiar with substrate.

possible solution:

ABI loading and input preparing: A possible solution with tooling is using an adapter for polkadot.js, so chain can provide the correct mapping to tx-payload to basically do the following.

graph LR
A[solang] -->|compile artifact| B[adapter] --> C[web3.eth.Contract]

The first level of support for this adapter is creating the web3/etherjs compatible object that loads pallet-contracts metadata underneath. It should also make the underlying read/write method mapped to the corresponding rpc's.

import { ContractAdapter } from "@solang/utils";
import { ContractPromise } from '@polkadot/api-contract';

// this is typical procedure for ink users.
const contract = new ContractPromise(api, metadata, address);

// this is typical solidity users on ethereum
new web3.eth.Contract(jsonInterface[, address][, options])

// the proposed workflow, which shares the same interface to web3.js
const contract: web3.eth.Contract = new ContractAdapter(jsonInterface[, address][, options])

rpc adapter: with the above web3 compatible object available, for most substrate chains we should be able to use substrate rpc to submit and wait for events, but the high level calling interface can still be web3.js compatible.

// this is typical workflow for polkadot.js/contracts-api users
const contract = new ContractPromise(api, metadata, address);
await contract.tx
  .inc({ storageDepositLimit, gasLimit }, incValue)
  .signAndSend(alicePair, result => {
    if (result.status.isInBlock) {
      console.log('in a block');
    } else if (result.status.isFinalized) {
      console.log('finalized');
    }
  });

// this is the proposed workflow which has similar calling convention to web3.js
const contract: web3.eth.Contract = new ContractAdapter(jsonInterface[, address][, options])

// pallet-contract-instantiate
contract.deploy({
    data,,
    arguments
})
.send({
    from,
    gas,
    gasPrice
}, function(error, transactionHash){ ... })

// pallet-contract-call
contract.methods.myMethod(123).send({from: 'addr'})
.then(function(receipt){
    // receipt can also be a new contract instance, when coming from a "contract.deploy({...}).send()"
});

// pallet-contract-query
contract.methods.myMethod(123).call({from: 'addr'}, function(error, result){
    ...
});

The default backend of the above operation will use polkadot.js underneath to call the actual api. But the adapter may take a additional argument to assume the rpc is eth-rpc compatible(such as our org), so it'll use the actual web3.rpc for various operation, this argument might also be able to take an additional adapter if the target chain has different sets of rpc.

For example if for some reason using a chain that implement other mechanism for transaction fee and will omit various inputs related to gas limit.

import { RpcAdapter } from @solang/utils;
import { EthAdapter } from @eth-rpc-compatible/chain;

// default being pallet-contracts-api's generic implementation.
const contract: web3.eth.Contract = new ContractAdapter(jsonInterface[, address])
// eth-rpc compatible project
const contract: web3.eth.Contract = new ContractAdapter(jsonInterface[, address], {adapter: EthAdapter })

// custom rpc
let customAdapter = new RpcAdapter(url,  {
     on_call: callback,
     on_deploy: callback,
     on_events: callback,
     ... 
})
const contract: web3.eth.Contract = new ContractAdapter(jsonInterface[, address], {adapter: customAdapter })

summary

so the proposed web3.js compatible tooling should make the current workflow using polkadot.js into:

graph LR
A[solang] -->|compiles| B[artifact]
B -->|ContractAdapter| C[web3.eth.Contract]
C -->|default adapter| D[polkadot.js]
C -->|eth-rpc adapter| E[web3.js]

For solidity devs, they can now:

1. only use polkadot.js API for connection and signing. And interact with the contract body like they are in eth environemnt.
2. for chain that actually provide full eth-rpc compatible, they can directly use most of the tools develop with web3.js in mind.

[xermicus]

Thanks for sharing this with us! I like the overall concept of your Idea and think having such a "compatibility layer" would benefit solang. I think this could have a place under solang CC @seanyoung ?

@extraymond are you already planning on implementing this idea?

[extraymond]

Unfortunately I'm not that productive with typescript, I'll check with the typescript guy in our team and see if he got the bandwidth to carry this. I'm thinking maybe Ink users might benefit from this too, at least the dapp dev will have easier time porting their frontend with pallet-contracts underneath.

[seanyoung]

Thanks for sharing this with us! I like the overall concept of your Idea and think having such a "compatibility layer" would benefit solang. I think this could have a place under solang CC @seanyoung ?

If this works, of course. It can exist in the solang repo or another repo on hyperledger.

So one problem is that address on Ethereum is 20 bytes and usually 32 bytes on Substrate. This means that addresses will have stored in bytes32 type fields.

[extraymond]

It seems web3.js only stored the address as string, so not limited by the address length. https://github.com/web3/web3.js/blob/a7b5dea0e98c0b7742c2477c511cb16acc1f27c2/packages/web3-eth-contract/types/index.d.ts#L35

[joshuacheong]

Hey this looks very promising, I strongly recommend pushing this for a grant with the Web3 Foundation to continue this work.