bitcoin-core / HWI

Bitcoin Hardware Wallet Interface
MIT License
500 stars 198 forks source link

Reproducibility issues for historical releases #303

Open achow101 opened 4 years ago

achow101 commented 4 years ago

It seems that I am unable to reproduce the results of existing releases.

As noted by https://github.com/spesmilo/electrum/issues/5859#issuecomment-577121504, part of this is related to Cython which is a build requirement for hidapi. Changing the version of this does result in different builds. However that is not the only thing causing issues. Even when forcing cython to a particular version (by adding poetry run pip install cython==0.29.13 to contrib/build_bin.sh), the result is different from the published builds. So there must be something else that is not version locked that is causing these issues.

achow101 commented 4 years ago

It looks like there are 2 other things making builds non-deterministic: poetry itself, and system library versions.

Poetry itself recently changed to use a deterministic but random looking path for the virtualenvs. This causes issues with building old releases because the full path is embedded in some files and this path is different from the path produced by older poetry versions. I don't think this should be a problem in the future, but perhaps we should lock the poetry version as well.

Pyinstaller embeds some system libraries that are necessary. But these libraries can change when the docker container updates its packages. This introduces non-determinism when they change. So we need to lock those versions as well. In particular, it looks like the problematic libraries are libcrypto.so.1.1, libssl.so.1.1, and libudev.so.1.