Add dependency version pinning and hash verification

[ghost]

From https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode:

Since version 8.0, pip can check downloaded package archives against local hashes to protect against remote tampering.

Since btcpy is likely to be used in wallets and handle real money, it is imperative that it use dependency version pinning and hash verification to protect against the possibility of the pypi repository (or package maintainers) being compromised.

Hashes can be determined manually by downloading their respective wheel packages from pypi and running them through sha256sum, or using the procedure described here.

Once hashes are specified, pip will refuse to install any dependency whose sha256 hash doesn't match the one specified in the requirements file.

[SimoneBronzini]

This is a very important point indeed! Will look into it in the next few days.

[lorenzogiust]

Hey @vedran6, we are looking for a method to enforce this behavior also for who installs this library directly from pip, but this would require adding the hashes to setup.py. Otherwise this only works by manually cloning the lib from github and doing pip install -r requirements.txt.

We could not find a way to add the hashes to setup.py, we would appreciate any help about that.

[ghost]

To my knowledge, this may be hard to do. Besides, if a user is simply installing from pypi without knowing which version of btcpy they're downloading and what its hash should be, they are still trusting that pypi hasn't been compromised and serving them a bad copy of btcpy.

One way to deal with this issue is to specify a list of btcpy versions and their corresponding hashes in a visible place independent from pypi.org (e.g. in the README file here). We should also make it known to users to only use the following installation idiom:

pip install chainside-btcpy==0.2 --require-hashes

Using this, pip will print out the remote hash, and the user will have an opportunity to verify that the remote hash matches the one in the README file, and will then have to specify it manually in order to proceed.

This is a trust issue that manifests quite often with pgp keys, where people choose to trust key directories without verifying key identities independently and through a secondary channel. Ultimately this type of security is hard to outsource and make "transparent" for end users, and eventually education is the only way to make sure that good security practices are enforced.

[SimoneBronzini]

This is an important point, but I think the question meant something else. At the moment, we pinned dependency versions in requirements.txt. This means that when a user clones this repo and runs pip3 install -r requirements.txt the hash check is enforced. However, users that download chainside-btcpy from pip will implicitly run setup.py to download dependencies, where there is no version pinning. Unfortunately we don't know of any way to enforce version hash pinning in setup.py dependencies as well.

[ghost]

Understood. It doesn't seem to be possible at the moment: https://github.com/pypa/pip/pull/3137

One workaround might be removing remote dependencies and implementing ecdsa and base58 locally. Neither is too hard to implement.

[ghost]

Another possible solution, a bit hacky albeit:

1. Maintain a list of pinned dependencies only in requirements.txt. Do not list any dependencies in setup.py.
2. Make setup.py parse requirements.txt and extract any requirements from there (save the hashes in a local variable).
3. Let setup.py install the dependencies as usual (hash agnostic)
4. Add a post-installation hook that uses pip download and pip hash on each dependency. Compare the hashes with with those in requirements.txt. Fail if mismatch.
5. Clean up the files created by pip download.

[SimoneBronzini]

That might be a good solution. Unfortunately I have never played with setup.py and I do not have much time in my hands but I'd be glad to merge a PR about that.