Spec for new deploy API

[pipermerriam]

New Deployment API

This document provides a spec for the new deployment API. This document is primarily targeted at the command line interface for deployment, but should be 1:1 mappable to all python deployment APIS.

Deployment Basics

In order for populus to deploy a contract it needs the following information.

• bytecode
• Mapping of placeholder => link_value

Where:

• bytecode is the deployment bytecode for the contract.
• placeholder is an offset into the bytecode which defines a location that needs linking.
• link_value is the value to be used for linking.

In addition to this, we need a way to reference previous deployments in order to refer to previous deploy assets. To accomplish this, we'll package every deployment in an ERC190 package. To accomplish this we need a name for this package.

Solution

We will lay out a solution that covers the following use cases.

• A contract defined in a source file within the project.
• A contract from a previous deployment which may no longer exist in the source files for the project.
• A contract from a source file from a project dependency.
• A contract from the compiled assets from a project dependency.

Deployment Command

The deployment command will be as follows.

$ populus deploy --save <package-name> <contract-identifier> --link <placeholder-identifier>:<link-value-identifier>

Alternatively, if you wish to not record this deployment you can use --no-save, or if you wish to have populus auto-generate a package name for you you can use --auto-save.

The --link portion of this command is optional and can be included multiple times to specify multiple link values.

Linking

To link, we need information about where the placeholders are in the source code and what value should be linked into each placeholder.

All link resolved values for linking must be validated as:

• Referencing either existing locations or named placeholders in the list of available link references for the bytecode.
• The proper length for the placeholder.

Definitions

Contract Identifier

The <contract-identifier> is interpreted as follows.

Contract Name

$ populus deploy MyContract

If the value conforms to the ERC190 definition of a Contract Name populus will check for a contract from the project's local contracts which exactly matches the provided name.

• If a single match is found, populus will use that contract.
• If no matches or multiple matches are found populus will raise an exception.

Source Path & Contract Name

$ populus deploy contracts/MyContract.sol#MyContract

If the value conforms to the format <path-to-project-source-file>#<contract-name> where <path-to-project-source-file> is a source file local to the project which contains a contract which exactly matches <contract-name> which is an ERC190 Contract Name, populus will use that contract.

Source Path URI & Contract Name

$ populus deploy file:///contracts/MyContract.sol#MyContract
$ populus deploy file://project/contracts/MyContract.sol#MyContract

If the value is a valid URI perform the following validation.

• Check that the scheme is file
• Check that the netloc is either empty or .
• Check that the path is a source file which exists under the defined source file locations for the project.
• Check that the fragment is an ERC190 Contract Name

Once these checks have passed, populus will use the defined contract from the compiled source file, or raise an exception if no contract by that name is produced from that source file during compilation.

Asset from a top level installed package

$ populus deploy crowdsale-v1:MyCrowdsale

It should be noted that this is simply a special case of the generic form for referencing assets from installed dependencies.

If the value conforms to the format <package-name>:<contract-name> where <package-name> is a valid ERC190 package name and <contract-name> is a valid ERC190 contract name then populus will use the precompiled assets for the specified contract from the specified contract. If no package by the given name is found or no contract by the given name is found within the package, populus will raise an exception.

Asset from an installed package

$ populus deploy crowdsale-v1.zeppelin-crowdsale:Crowdsale

IF the value conforms to the format <package-name>(.<package-name>)+:<contract-name> where the <package-name> portion can be repeated many time populus will perform the following validation which is loosely equivalent to how link references are specified in ERC190.

Let p_1, p_2, ..., p_n be the sequence of package names extracted from p_1.p_2.p_n

• All package names must be valide ERC190 package names.
• For each p_n, p_n must be present in the build_dependencies for p_n-1.

Populus will validate that <contract-name> is a valid ERC190 contract name and that it is present in the contract types of p_n and that it contains a bytecode entry. If all of the validation steps pass, populus will use this contract. Otherwise it will raise an exception.

URI for installed package asset.

$ populus deploy package://crowdsale-v1.zeppelin-crowdsale#Crowdsale

If the value is a URI populus will check the following.

• The scheme is package
• The netloc is non-empty and when split on ., all values are valid ERC190 package names.
• The fragment is a valid ERC190 contract name.

The package names extracted from the netloc should be validated in the same manner as the previous case.

Populus will use the precompiled asset for the specified contract from the package in the dependency tree or raise an exception if it is not present or any other validation failes.

Asset from an installed package source file

$ populus deploy crowdsale-v1.zeppelin-crowdsale:contracts/Crowdsale.sol#Crowdsale

If the value is in the format <dot-separated-package-path>:<source-file-path>#<contract-name> where <dot-separated-package-path> is a dot separated list of valid ERC190 package names and <source-file-path> exists as a source file in the specified package and <contract-name> is a valid ERC190 contract name, populus will recompile this source file using the local compilation settings for the project and use the resulting compiled asset from the specified <contract-name>. If no such compiled asset is produced, populus will throw an error.

URI for installed package source file

populus deploy file://crowdsale-v1.zeppelin-crowdsale/contracts/Crowdsale.sol#Crowdsale

Note this is merely a long form URI version of the previous case.

When the value is a URI the following validation will checked.

• The scheme is file
• The netloc is a dot separated list of valid ERC190 package names.
• The path is a valid source file path from the specified package.
• The fragment is a valid ERC190 contract name.

Populus will then compile the specified source file using the local compiler settings and use the specifed contract name as extracted from the URL fragment. If no contract by that name is produced during compilation and exception will be raised.

Placeholder Identifiers

The <placeholder-identifier> can take one of two forms.

• Exact bytecode location.
• Placeholder name

Exact placeholder location

$ populus deploy MyContract --link 120:<link-value-identifier>

An exact placeholder location can be specified using the numeric index into the bytes representation of the bytecode.

Named placeholders

$ populus deploy MyContract --link MyLibrary:<link-value-identifier>

The name may be used to link all placeholders of a given name with the provided value.

Link Value Identifiers

The <link-value-identifier> can take one of the following forms.

• Literal value
• Reference to a deployed instance address from one of the installed packages.

Link value literals

$ populus deploy MyContract --link MyLibrary:0xdf6c53df56f3992fc44195518a2d8b16306af9ff

Use of a link value literal is done by providing the hex encoded value (optionally 0x prefixed)

Reference to deployed instance

$ populus deploy MyContract --link MyLibrary:crowdsale-v1.multisig-v1#MyMultiSig

If the value is in the format <dot-separated-package-path>#<deployed-instance-name> then the <dot-separated-package-path> will be validated as a valid path to an installed dependency. The available blockchain URIs in this package will then be filtered down to the ones which validate against the current chain that deployment is being executed on. If there are no matches or more than one match, populus will raise an exception. If <deployed-instance-name> is found in the deployments for the matching chain, the address from that instance will be used.

URI Reference to deployed instance

$ populus deploy MyContract --link link://MyLibrary@crowdsale-v1.multisig-v1#MyMultiSig

Note that this is simply an alternate format for the previous case.

When the value is a URI populus will perform the following validation.

• The scheme is link
• The username is a valid ERC190 contract name.
• The netloc is a valid dot-separated package path
• The fragment is a valid ERC190 contract name.

Populus will perform the same validation as the previous case and use the resulting contract address if all validation passes.

[pipermerriam]

Note that this hinges on the yet-unspecified API for how we record deployments using ERC190 as a serialization format.

[fubuloubu]

Browsed through this, not sure if I missed something. Let's say I wanted to have my contracts in more hierarchical structure like:

contracts:
    folder1:
         ContractA...
          Others...
    folder2:
          ContractB...
Etc...

Is there a standard means to do that, under the current or upcoming API?

Additionally, is there a means to store the Deployment Parameters for your set of smart contracts in this contracts/ folder so it can be available to anyone who has access to the it?

One of my project teams I'm working on is making the contracts/ folder publicly available, but tests, deployments, etc. will not be and I was hoping that there was a workaround for the monolithic structure of Populus.

[pipermerriam]

@fubuloubu very open to working with you to figure out where the pain points are for your use case and how we can make those easier.

Here is a long winded reply that hopefully answers this question. The roadmap for the coming year involves these four high level things.

step 1 is going to be an under-the-hood improvement in how populus serializes compiled assets and deployed assets. The new serialization will be the same format that smart contract packages use which will functionally make every one of these assets a package.

step 2 is this issue. Taking advantage of the new serialization format to fix some long standing deficiencies as well as generally improve the contract APIs. This includes the python APIs such as Providers and Registrars, as well as command line APIs like this issue describes.

step 3 will be full support for packaging. This will include functionality for deploying your own package registrar, creating and publishing a package from your project's contracts, and installing other packages for use in your project.

step 4 is where I think your problems get addressed. Command line deployment is fine for relatively simple deployments. Python code based deployments are fine for when people want full control. There's a middle road between these two which is some level of automation but not requiring you to do everything. It's not clear at this point exactly what this will look like, but my best guess is that it will look something like django migrations and management commands. The ability for you to write scripts that are easy to run against your project and deployed assets as well as batteries included automation for things like deployments.

---

With respect to how you organize your source files:

Populus currently doesn't mind the folder structure you proposed and should do fine finding all of your contracts. This however doesn't really help when a project just has a ton of contracts. The package management features are the best solution I know of for this. It will allow you to split parts of your project out into stand-alone packages and simply install them in your project. All of the source files, deployed assets, etc will be available for use, but your project won't be cluttered with extra source files and what-not.

[fubuloubu]

Thanks for the response @pipermerriam.

I come from a non-web background, specifically aerospace, so (for better or worse) I'm used to a way different deployment pipeline, so about half of what you said I'll have to revisit when I have more time to working through the project timeline.

It's good to hear that I can set my folder heirarchy however I want, I wasn't sure about that. I think a good way of explaining my second issue is being able to specify where the project.json is stored. I am looking to set up all my subfolders as separate repos in order to support a more modular development process, but having the project file separate sort of ruins that.

The last thing is the deployment parameters, which I thought you had something similar to for in project.json (e.g. custom parameters). I wanted to have it so custom parameters could be added for each contract that get used as the defaults unless they are overriden (say I want to test my 28 day ICO using significantly less blocks, I'd override the duration).

I have some other ideas as I started down the path of my own IDE-ish kind of thing that leveraged python more. I tried to use Populus but the pyethereum>2.0.0 issue conflicted with a lot of my text fixtures. Definitely looking to collobarate on Populus now that I'm getting over my need to reinvent the wheel lol

[pipermerriam]

I tried to use Populus but the pyethereum>2.0.0 issue conflicted with a lot of my text fixtures

This should be fixed within 2-3 weeks. It's top on my priority list but it has 1-2 things blocking it.

[fubuloubu]

Anyway I can help?

[pipermerriam]

Anyway I can help?

Most everything that needs to be done requires prior knowledge of one or more codebases.

The only part of it that might be doable would be implementing a PyEthereum2.0 backend in the ethereum-tester codebase.

[fubuloubu]

Cool. I've been working a lot with pyethereum-tester. My problems with Populus befote was mostly that my tests used a lot of fixtures based on the tester module, which was only in the 2+ release. I develop for Viper with David and Jacques, Viper and any project could really benefit from a standard set of pytest fixtures. I'll see what I can whip up when I get a better look at the codebase after Thanksgiving

[pipermerriam]

@fubuloubu so I already mentioned it but, I think it's worth pointing out that the ethereum-tester library is probably a good replacement for what I suspect you're doing with ethereum tester.

Is it possible for you to provide me with a link to the actual code where you have those fixtures setup so that I can see how you're using it? PyEthereum is on it's way out, replaced by PyEVM soon and I want to be sure that people don't get left in a lurch when that transition really picks up.

[fubuloubu]

A lot of what I've been doing is using the helpers from here. I've got some extra solidity helpers for compiling I can show you, but I'm still disentangling it from my production stuff. You're probably right, mostly stuff that module can provide or be upgraded to provide. I'm just looking for the easiest set of pytest fixtures to use in testing, abstractimg away a lot of the complexities of the EVM implementation so I can just call methods and make sure they're showing my contract functionality works as is, eventually having a methos to get relevant coverage metrics ideally. After that, easy ways of managing deployments to different networks (which Populus provides) is what I am working on figuring out currently.

Question: you state Pyethereum is on its way out, but also that ethereum-tester needs a Pyethereum 2+ backend. Is this just a temporary measure until PyEVM gets up to speed? It's good to know some of the changes happening in the ecosystem.

[pipermerriam]

Is this just a temporary measure until PyEVM gets up to speed?

Yes, there will be a period where PyEVM and PyEthereum co-exist for a while as we validate the new implementation and provide upgrade paths for people who have infrastructure that depends on PyEthereum.

[pipermerriam]

Looking at the code you linked I'm confident that transitioning to ethereum-tester is likely to be a big improvement. Are you using web3.py for anything? Is python anywhere else in your stack besides development (like being used in backend servers or something?)

[fubuloubu]

Yes, there will be a period where PyEVM and PyEthereum co-exist for a while as we validate the new implementation and provide upgrade paths for people who have infrastructure that depends on PyEthereum.

If Populus and/or Ethereum-Tester can implement the correct tools/fixtures to use to replace the ethereum.tester module usages a lot of people use, then we can transition the backend from PyEthereum to PyEVM when it's ready. I think this is what you're trying to tell me, just saying it out loud to make sure I understand the flow of all these projects.

[pipermerriam]

we can transition the backend from PyEthereum to PyEVM when it's ready

Exactly, and the cool part is that if you go ahead and switch over to ethereum-tester (when it has PyEthereum2.0 support) then making the switch from PyEthereum to PyEVM will be a single line configuration change. You can even run your tests against both if you want that level of confidence.

[pipermerriam]

I'm also thinking that many of the fixtures that populus provides will probably make sense to be relocated into the ethereum-tester library since people shouldn't have to adopt populus just to have nice testing tools.

[fubuloubu]

I'm definitely going to see if ethereum-tester meets my needs for functional testing. I was playing around with web3.py in order to figure out the deployments due to aforementioned issues using Populus.

I'm not a web guy, both my projects have separate people I'm working with to do the websites. A majority of one if not both of the projects will involve Oracles and other behind-the-scenes work that I am writing in Python.

[fubuloubu]

I sent you a PM on GItter, I think we've turned this into a conversation thread lol