Revamp synchronized management of Rust- and non-Rust dependencies for contracts

[ryscheng]

Ekiden, as a library, doesn't fit neatly into Cargo, because it needs to provide some non-Rust resources to dependent contracts, and Cargo does not naturally manage non-Rust resources. These non-Rust dependencies include, for example Makefile.toml files, EDL files from the Rust SGX SDK, and miscellaneous enclave signing configuration files.

We have currently been delivering these non-Rust dependencies through a Git submodule. However, this solution of using both Cargo and Git submodules has some undesirable properties, compared to a hypothetical ideal build system (HIBS):

• Initializing Git submodules and running Cargo make take up two steps in the process of getting started with contract development. HIBS would obtain both Cargo dependencies and Git submodule dependencies in a single step, which is fewer steps than two.
• There are two locations where dependency versions are canonically stored: Rust dependencies' in Cargo.toml/Cargo.lock and non-Rust dependencies' in a submodule commit ID. HIBS would allow you to manage dependency versions in a single location.
• Some dependencies, namely the Rust parts of the submodule, can be specified as Cargo dependencies in more than one way: (i) as a path in the submodule or (ii) from a repository (a Git repo or a Cargo repo). Some mixtures of these ways to specify the dependencies are incorrect, and importantly, the incorrect mixing may not immediately cause problems. In HIBS, it would be impossible to configure the dependencies incorrectly in this way.
• Git submodules are wasteful when you are working on multiple contracts. You would have multiple checkouts of Ekiden and have to build them all separately, even if they are the same version. HIBS would re-use sources and compiled dependencies.

This issue identifies several ways our build system falls short of HIBS and asks that we bring our build system closer to HIBS. We have collected some proposals for achieving this, which for example involve getting rid of the submodules and further extending our Makefile.toml's and adding additional scripts to be run by Cargo.

(previous title and description below)

---

Remove need for git submodule from contract repos

Currently there's a possibility that the ekiden git submodule in the contract repo is out of sync with Cargo.toml.

The only reason we need this is to extend the Makefile, proposal is to replace the git submodule with a script that fetches the correct Makefile based on the ekiden version in cargo.toml

I acknowledge that our build system can pose issues to beginners and that it would be nice to minimize possible complications in onboarding and that doing so may involve changes to our build system. As we move forward with this revamp, I want us to keep in mind some things that our build system does well and that an ideal build system would do well too.

• It is easy to make changes to Ekiden while working on a contract and test changes together. This is important when, in the course of developing a contract, you find a bug in Ekiden and want to fix it and verify that the fix resolves the problem.
• It is possible to have CI build a contract with a modified version of Ekiden without having that modified version be approved and merged into master.
• Within the build process, the point where reliance on network resources ends is composed of landmarks that are understood by the community outside of Ekiden: after cloning the repo and submodules and after Cargo downloads dependencies. This may come in handy in future work on offline builds and deterministic builds.
• The build process is satisfactorily decoupled from the conceptually separate systems of version control and code repository hosting. It is reasonably easy to make a fork of Ekiden, even if that fork were never to be distributed publicly or never to be managed with Git or to be distributed as a source archive.

---

If everything could be provided in crates, I think Cargo can handle these use cases because it supports dependency overrides.

It is easy to make changes to Ekiden while working on a contract and test changes together. This is important when, in the course of developing a contract, you find a bug in Ekiden and want to fix it and verify that the fix resolves the problem.

The canonical way to handle these is the [patch] section. Also see documentation on overriding dependencies which specifically mentions this scenario (you find a bug in an upstream create or you even need to add a feature to the upstream crate).

It is possible to have CI build a contract with a modified version of Ekiden without having that modified version be approved and merged into master.

As long as the CI can fetch the modified version of Ekiden, this can also be achieved with the patch method above. You just need to specify the repository containing your patched version of Ekiden and it will be used.

The build process is satisfactorily decoupled from the conceptually separate systems of version control and code repository hosting. It is reasonably easy to make a fork of Ekiden, even if that fork were never to be distributed publicly or never to be managed with Git or to be distributed as a source archive.

The same applies to the patch method as you can specify local paths in case you are not using Git.

Another question is how to distribute binaries like the compute and consensus nodes? Should we just cargo install them?

[kostko]

So as of #77 we've simplified the way we do external contracts as we basically use our contract builder (part of ekiden-tools) for building contracts, so there is no need to fetch any makefiles or use any Git submodules.

The only thing that currently still needs to be fetched separately is the sgx-enter.sh script (and the hw-mode counterpart). This could be further improved as described in #80.

[ryscheng]

Great. Let's close this for now