Vyper contract flattener [APE-872]

fubuloubu commented 1 year ago

Overview

Would be nice to have a contract flattener for vyper, since currently right now Etherscan doesn't allow verification using vyper with multiple files

Specification

First, add a function like from ape_vyper.flatten import flattener that would take a path (and an optional base path) and flatten that file then output the flattened file as a string

Second, add ape vyper CLI extension where ape vyper flatten PATH flattens a particular file from a user's contracts/ folder for them, and prints out the results

Dependencies

n/a

antazoey commented 8 months ago

The CLI may be exactly the same regardless of vyper or solidity, makes me wonder if there is a better place for it.

Also we never added the CLI part for ape-solidity yet, we should do that as well

mikeshultz commented 7 months ago

To be clear, "flattening" is the act of collapsing the contracts and all the dependencies into a single source file, right?

fubuloubu commented 7 months ago

To be clear, "flattening" is the act of collapsing the contracts and all the dependencies into a single source file, right?

Correct, only until stdlib imports remain.

It could be a little tricky since Vyper does allow importing JSON ABI interfaces

mikeshultz commented 7 months ago

Just documenting some research and learnings from my testing. Seems this is definitely non-trivial. Not as straight forward as Solidiy whre you can just mash contract files together into one.

We cannot have multiple contracts per file (pretty sure that used to be a thing).
Any imported Vyper contracts must be converted into an interface because of the above
stdlib imports should be left alone
Any imported JSON interface files must be converted into a Vyper interface definition
Interfaces can not be flat/non-indented, and must be declared (e.g. interface Iface:). So some code formatting will need to take place.
Imports and version pragamas will need to be stripped, cleaned, normalized, and re-inserted at the top (or at a minimum, flattened imports must be removed)

Here's an ideal output from one of our unit test contracts:

# @version ^0.3.3
from vyper.interfaces import ERC20

# File: @exampledep/Dependency
interface Dep:
    def read_stuff_2() -> uint256: view

# File: interfaces/IFace.vy
interface IFace:
    def read_stuff() -> uint256: view

# File: interfaces/IFace2.vy
interface IFace2:
    def read_stuff_3() -> uint256: view

# File: use_iface.vy
@external
@view
def read_contract(some_address: address) -> uint256:
    myContract: IFace = IFace(some_address)
    return myContract.read_stuff()

It seems that we may have ABI specs available via ContractTypes and LocalDependency structures, which we may then be able to construct Vyper interfaces. This may be ideal as a separate package.

There's a relevant tool (abi-to-sol) for Solidity, but I don't think there's one for Vyper, as most people can just import JSON interfaces directly. If anyone knows of any libs or tools that might eliminate some of this work, please share. I haven't found much.

So, I think step 1 is to create something that will convert ABI specs into Vyper interface source code. The definitions seem fairly straight forward, so it's probably not too difficult.

mikeshultz commented 7 months ago

I put together a little module for doing the source code generation for an ABI spec. Not sure it warrants its own package (unless we wanted to break out its own CLI or something). I'll probably just add it to this package.

https://gist.github.com/mikeshultz/e06c7cfe2e45cba0571b127db671d56d

mikeshultz commented 7 months ago

Can't compile imported interfaces as if they were a contract because they're basically an invalid contract syntax (mainly because pass body has no return type). Can't just mash them into the flattened source because they aren't the same as an inline interface, either.

Imported:

@view
@external
def read_stuff() -> uint256:
    pass

Inline:

interface Example:
    def read_stuff() -> uint256: view

So, we need something that can parse the imported interface. Perhaps something in the vyper repo (AST utils maybe) could help in this regard. Will need to do some research.

fubuloubu commented 7 months ago

A few additional notes:

Vyper v0.4.0 will add full module support, so code flattening will be harder as there needs to be a way to rectify @charles-cooper's "unique" state initialization mechanism, which won't make sense for a flattened contract to keep
Vyper has an "interface builder" mode where you can export properly-formatted vyper code for an ABI, I believe
There was at least one attempt to build a black-like Vyper linter/fixer, and in general since the module is python you can just "extend" vyper's ast parsing (although in practice this is hard with the requirement of handling many different vyper versions)

charles-cooper commented 7 months ago

There was at least one attempt to build a black-like Vyper linter/fixer, and in general since the module is python you can just "extend" vyper's ast parsing (although in practice this is hard with the requirement of handling many different vyper versions)

We should probably just maintain a prettifier in the compiler at this point

Vyper v0.4.0 will add full module support, so code flattening will be harder as there needs to be a way to rectify @charles-cooper's "unique" state initialization mechanism, which won't make sense for a flattened contract to keep

Probably the best way to do this (short of etherscan just supporting multi file verification) is via name mangling for internal functions and types.

mikeshultz commented 7 months ago

Is 0.4 features really something I should be considering at this point? I'm not entirely convinced there's a reasonable solution for it at the current version.

@charles-cooper We could probably do this fine if we just had a way to read in the imported interface to something usable (ABI would be ideal). Any tooling you know that might be useful in that regard? Like something in the Vyper library that handles these imports in the compiler?

fubuloubu commented 7 months ago

Is 0.4 features really something I should be considering at this point? I'm not entirely convinced there's a reasonable solution for it at the current version.

@charles-cooper We could probably do this fine if we just had a way to read in the imported interface to something usable (ABI would be ideal). Any tooling you know that might be useful in that regard? Like something in the Vyper library that handles these imports in the compiler?

Would say v0.4 is within 3 months of becoming necessary to work with, so not important right now but soon will be

mikeshultz commented 7 months ago

I was able to put together a quick proof of concept script that will generate a Vyper inline interface from a Vyper source file. Basically it uses the vyper Python package's AST tools to convert a source into AST, then iterate through the external function defs to create an Ape-compatible ABI. Then from there it's trivial to turn into an inline interface.

https://gist.github.com/mikeshultz/1c3743587bdbd031ba7caae9529edb4b

It's probably not exhaustive, nor tested on everything Vyper, but I was able to get it working with some basic contracts and interfaces. Will need to work on various different complex input and output types like structs, and multi-type tuples if Vyper supports those.

A nice bonus is that this method could also be used for regular contracts as well (and maybe the modules hinted at) so we don't have to get the compiler involved to generate the interface.

mikeshultz commented 6 months ago

Doesn't appear we can import structs from other Vyper contracts (though it's on the way). And the compiler doesn't produce sane interfaces with struct returns, either. So maybe structs aren't a concern right now. Kind of curious how Vyper would interact with Solidity contracts with complex return types. Maybe it doesn't.

Multi-type tuples appear to be an upcoming feature so also probably not an immediate concern.

Should be able to move forward by moving from my proof of concept into a functional flattener though.

antazoey commented 6 months ago

I think we could use named tuples everywhere.

mikeshultz commented 6 months ago

I think we could use named tuples everywhere.

No tuples in Vyper yet, and I'm not talking about representing anything in Python. Was just looking into how Vyper interfaces worked so I knew what needs handling when building an inline interface from Vyper AST.

mikeshultz commented 6 months ago

Made some progress on the flattener. So far it works on use_iface.vy and outputs this:

# @version ^0.3.3

from vyper.interfaces import ERC20

interface Dependency:
    def read_stuff_2() -> uint256: view

interface IFace:
    def read_stuff() -> uint256: view

interface IFace2:
    def read_stuff_3() -> uint256: view

@external
@view
def read_contract(some_address: address) -> uint256:
    myContract: IFace = IFace(some_address)
    return myContract.read_stuff()

Still some open questions like what to do with commentary (it's stripped out in this case), import alias support, how to properly do code formatting, and I bet there's some edge cases I haven't seen yet. But making steady progress. Will continue testing on different contracts and probably have a WIP PR up sometime on Monday.

mikeshultz commented 6 months ago

Second, add ape vyper CLI extension where ape vyper flatten PATH flattens a particular file from a user's contracts/ folder for them, and prints out the results

I added this as specced, but considering flatten_contract() is a part of CompilerAPI, would it be better to add this CLI command to Ape?

antazoey commented 6 months ago

would it be better to add this CLI command to Ape?

If you haven't realized yet, adding a CLI to ape is more a challenge than normal click programs. You basically have to create an entire plugin (whether it be in core or wherever).

that being said, we could potentially utilize the pm core plugin:

ape pm flatten <file.whatever>

mikeshultz commented 6 months ago

would it be better to add this CLI command to Ape?

If you haven't realized yet, adding a CLI to ape is more a challenge than normal click programs. You basically have to create an entire plugin (whether it be in core or wherever).

that being said, we could potentially utilize the pm core plugin:
ape pm flatten <file.whatever>

Did a quick test and added a new @click.command func in ape_compile and register it like "ape_flatten=ape_compile._cli:flatten",. It's filtered out of the help message (and possibly some other side-effects) but it does seem to actually work.

https://github.com/ApeWorX/ape/blob/791a76f9565607240c9d71d7b0211ebb9146a549/src/ape/_cli.py#L60-L61

Could maybe get this to work with a little entry point key format change (e.g. ape_compile:flatten=ape_compile._cli:flatten). Though maybe I don't want to get into refactoring this CLI loading right now for this issue... I'm probably missing some key context anyway.

I'll just get the PR ready for the original spec and we can iterate later if it makes sense.

mikeshultz commented 6 months ago

Feature released as experimental in v0.7.1

ApeWorX / ape-vyper