I'm wondering if there's utility in shipping a CLI for installing packages to disk as well as going ahead and defining the specification for the layout of packages on disk so that 3rd party integration is well supported. This would lower barrier to using EthPM based features for both Vyper and Web3.py use case and should reduce some barriers to adding support to some of those other 3rd party frameworks.
Here's the general layout as I remember it from the Populus implementation.
ROUGH Filesystem Installation Spec
The below is written using UNIX environment variable syntax for simplicity sake.
let $CWD be the current working directory.
let $PACKAGES_DIR be the directory that packages are installed to. This defaults to the relative path ./ethpm_packages
If the path is relative it is resolved with respect to $CWD
To install a package a user must provide the following information.
PACKAGE_IDENTIFIER: specifies the package to install
ALIAS: an optional alias name to install the package under
This is needed to allow installation of multiple versions of the same package under the same the same namespace.
Subject to the same name validity rules as package_name from the EthPM spec
The PACKAGE_IDENTIFIER concept should be fully fleshed out, potentially using a URI scheme, however for this spec, I'll define two identifier types which can generally be mapped to URI schemes.
NAME: The name of the package (as it exists in the registry)
REGISTRY_URI: the package registry to install the package from.
VERSION: The version identifier to use to resolve the package.
Things like =1.2.3, >=1.2.3,<1.8, etc
Installation of packages is a recursive operation, with the top level namespace being comprised of user specified packages and deeper name spacing being the dependencies of the user specified packages. Here is what a single package namespace looks like.
|--- CWD
|---PACKAGES_DIR
|--- ethpm.lock
|--- <package-name>
| |--- package.json
| |--- SRC/<package-source-tree>
| |--- PACKAGES_DIR (same directory format containing installed dependencies for this package.
| |--- <...>
|--- <...>
When a user installs a package, the directory structure above is lazily created. Each top level package is installed to it's own directory as follows.
$PACKAGES_DIR/<package-name>/
Within that directory, the following files and directories are created.
package.json: The package manifest for the package
A directory named ./src which contains the fully resolved package source tree.
another $PACKAGES_DIR which contains all of the installed dependencies for the package.
This could probably be omitted for packages without dependencies.
Each $PACKAGES_DIR contains a top level file named ethpm.lock. The purpose of this file is track what packages are installed and what mechanism they were installed with. This is necessary to preserve the information of where packages came from in for packages installed via mechanisms like registry URIs. The file structure is out of scope for this spec but it would contain thing like the following.
What was wrong?
It's still kind of hard for people to work with ethpm packages.
But other than the first two which are snake charmer projects, it's likely that the 3rd party projects end up implementing support in their own
How can it be fixed?
This has me thinking about the concept of filesystem installs for ethpm packages. I implemented this once in populus a long time ago:
https://github.com/ethereum/populus/blob/4e9cc1efcea2333f48ceec9081ef5392d87608e8/populus/packages/installation.py
I'm wondering if there's utility in shipping a CLI for installing packages to disk as well as going ahead and defining the specification for the layout of packages on disk so that 3rd party integration is well supported. This would lower barrier to using EthPM based features for both Vyper and Web3.py use case and should reduce some barriers to adding support to some of those other 3rd party frameworks.
Here's the general layout as I remember it from the Populus implementation.
ROUGH Filesystem Installation Spec
The below is written using UNIX environment variable syntax for simplicity sake.
$CWDbe the current working directory.$PACKAGES_DIRbe the directory that packages are installed to. This defaults to the relative path./ethpm_packages$CWDTo install a package a user must provide the following information.
PACKAGE_IDENTIFIER: specifies the package to installALIAS: an optional alias name to install the package underpackage_namefrom the EthPM specThe
PACKAGE_IDENTIFIERconcept should be fully fleshed out, potentially using a URI scheme, however for this spec, I'll define two identifier types which can generally be mapped to URI schemes.NAME: The name of the package (as it exists in the registry)REGISTRY_URI: the package registry to install the package from.VERSION: The version identifier to use to resolve the package.=1.2.3,>=1.2.3,<1.8, etcInstallation of packages is a recursive operation, with the top level namespace being comprised of user specified packages and deeper name spacing being the dependencies of the user specified packages. Here is what a single package namespace looks like.
When a user installs a package, the directory structure above is lazily created. Each top level package is installed to it's own directory as follows.
$PACKAGES_DIR/<package-name>/Within that directory, the following files and directories are created.
package.json: The package manifest for the package./srcwhich contains the fully resolved package source tree.$PACKAGES_DIRwhich contains all of the installed dependencies for the package.Each
$PACKAGES_DIRcontains a top level file namedethpm.lock. The purpose of this file is track what packages are installed and what mechanism they were installed with. This is necessary to preserve the information of where packages came from in for packages installed via mechanisms like registry URIs. The file structure is out of scope for this spec but it would contain thing like the following.