search

cryptoadvance / specter-desktop

A desktop GUI for Bitcoin Core optimised to work with hardware wallets
MIT License
802 stars 238 forks source link

Instructions and environment for reproducible builds of releases #1386

Open achow101 opened 3 years ago

achow101 commented 3 years ago

It would be useful to have instructions on how to reproduce the binaries published for releases without needing to setup an entire CI environment. There do not appear to be any instructions on how someone can do a reproducible build of the release binaries.

eriknylund commented 3 years ago

I too would appreciate instructions for how to build the binaries. I think this would be an important tool to have in situations like the false positive trojan windows binary incident in the beginning of July 2021.

k9ert commented 3 years ago

Sorry for the late response. Fully agree, this is necessary however, what about this: https://github.com/cryptoadvance/specter-desktop/blob/master/docs/build-instructions.md

Any feedback highly appreciated if it doesn't fit your needs.

eriknylund commented 3 years ago

Sorry for the late response. Fully agree, this is necessary however, what about this: https://github.com/cryptoadvance/specter-desktop/blob/master/docs/build-instructions.md

Any feedback highly appreciated if it doesn't fit your needs.

Thanks for the link. I'll give it a go and let you know!

eriknylund commented 3 years ago

@k9ert

Following the instructions, I was able to complete the first part using a Python:3.8 docker container. There seem to be a step missing to install the requirements, by adding that it ran without problems.

docker run -it -v $PWD:/tmp --workdir /tmp/specter-desktop python:3.8 bash
pip list
# Missing step in https://github.com/cryptoadvance/specter-desktop/blob/master/docs/build-instructions.md
pip3 install -r requirements.txt
python3 setup.py sdist bdist_wheel

For the second part, building specterd I ran into an issue with a missing version.txt. A workaround was to create it with touch version.txt and then the binary is built. Are you able to say if I'm missing a step that generates it or what the version.txt should look like?

cd pyinstaller
# prerequisites
pip3 install -r requirements.txt --require-hashes
pip3 install -e ..
# cleaning up
rm -rf build/ dist/ release/ electron/release/ electron/dist release-linux/ release-win/
pyinstaller specterd.spec
35 INFO: PyInstaller: 4.0
35 INFO: Python: 3.8.12
37 INFO: Platform: Linux-5.10.47-linuxkit-x86_64-with-glibc2.2.5
39 INFO: UPX is not available.
63 INFO: Extending PYTHONPATH with paths
['/tmp/specter-desktop/pyinstaller']
70 INFO: checking Analysis
71 INFO: Building Analysis because Analysis-00.toc is non existent
71 INFO: Initializing module dependency graph...
74 INFO: Caching module graph hooks...
85 INFO: Analyzing base_library.zip ...
2475 INFO: Processing pre-find module path hook distutils from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
2476 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.8'
4070 INFO: Caching module dependency graph...
4425 INFO: running Analysis Analysis-00.toc
4447 INFO: Analyzing specterd.py
8828 INFO: Processing pre-safe import module hook urllib3.packages.six.moves from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-urllib3.packages.six.moves.py'.
12768 INFO: Processing pre-safe import module hook six.moves from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
13775 INFO: Analyzing hidden import 'pkg_resources.py2_warn'
13776 ERROR: Hidden import 'pkg_resources.py2_warn' not found
13776 INFO: Processing module hooks...
13776 INFO: Loading module hook 'hook-hwilib.devices.py' from '/tmp/specter-desktop/pyinstaller/hooks'...
14052 INFO: Loading module hook 'hook-certifi.py' from '/usr/local/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
14054 INFO: Loading module hook 'hook-jinja2.py' from '/usr/local/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
14055 INFO: Loading module hook 'hook-cryptography.py' from '/usr/local/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
14177 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
14178 INFO: Loading module hook 'hook-pkg_resources.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
14755 INFO: Processing pre-safe import module hook win32com from '/usr/local/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
14757 WARNING: Hidden import "pkg_resources.py2_warn" not found!
14757 WARNING: Hidden import "pkg_resources.markers" not found!
14759 INFO: Excluding import '__main__'
14762 INFO:   Removing import of __main__ from module pkg_resources
14762 INFO: Loading module hook 'hook-importlib_metadata.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
14765 INFO: Loading module hook 'hook-lib2to3.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
14777 INFO: Loading module hook 'hook-pytz.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
14835 INFO: Loading module hook 'hook-encodings.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
14883 INFO: Loading module hook 'hook-xml.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
14939 INFO: Loading module hook 'hook-sqlite3.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
14974 INFO: Loading module hook 'hook-babel.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
15028 INFO: Loading module hook 'hook-distutils.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
15031 INFO: Loading module hook 'hook-sysconfig.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
15086 INFO: Looking for ctypes DLLs
15559 WARNING: library setupapi required via ctypes not found
15634 WARNING: library Advapi32 required via ctypes not found
15653 INFO: Analyzing run-time hooks ...
15666 INFO: Including run-time hook '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
15669 INFO: Including run-time hook '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgres.py'
15670 INFO: Including run-time hook '/usr/local/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/rthooks/pyi_rth_certifi.py'
15691 INFO: Looking for dynamic libraries
16498 INFO: Looking for eggs
16498 INFO: Using Python library /usr/local/lib/libpython3.8.so.1.0
16514 INFO: Warnings written to /tmp/specter-desktop/pyinstaller/build/specterd/warn-specterd.txt
16912 INFO: Graph cross-reference written to /tmp/specter-desktop/pyinstaller/build/specterd/xref-specterd.html
17023 INFO: Appending 'datas' from .spec
Unable to find "version.txt" when adding binary and data files.
# workaround to create the missing file
touch version.txt
pyinstaller specterd.spec
33 INFO: PyInstaller: 4.0
33 INFO: Python: 3.8.12
35 INFO: Platform: Linux-5.10.47-linuxkit-x86_64-with-glibc2.2.5
37 INFO: UPX is not available.
53 INFO: Extending PYTHONPATH with paths
['/tmp/specter-desktop/pyinstaller']
61 INFO: checking Analysis
250 INFO: Appending 'datas' from .spec
275 INFO: checking Tree
276 INFO: Building Tree because Tree-00.toc is non existent
276 INFO: Building Tree Tree-00.toc
295 INFO: checking PYZ
297 INFO: Building PYZ because PYZ-00.toc is non existent
297 INFO: Building PYZ (ZlibArchive) /tmp/specter-desktop/pyinstaller/build/specterd/PYZ-00.pyz
2798 INFO: Building PYZ (ZlibArchive) /tmp/specter-desktop/pyinstaller/build/specterd/PYZ-00.pyz completed successfully.
2851 INFO: checking PKG
2852 INFO: Building PKG because PKG-00.toc is non existent
2852 INFO: Building PKG (CArchive) PKG-00.pkg
34465 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
34567 INFO: Bootloader /usr/local/lib/python3.8/site-packages/PyInstaller/bootloader/Linux-64bit/run
34567 INFO: checking EXE
34568 INFO: Building EXE because EXE-00.toc is non existent
34569 INFO: Building EXE from EXE-00.toc
34573 INFO: Appending archive to ELF section in EXE /tmp/specter-desktop/pyinstaller/dist/specterd
36285 INFO: Building EXE from EXE-00.toc completed successfully.

The third part - building the Electron app - works without problems.

eriknylund commented 3 years ago

I did a bit more digging and found that the proper way to build is probably through ./build-unix.sh (https://github.com/cryptoadvance/specter-desktop/blob/master/pyinstaller/build-unix.sh) since this creates the version.txt and wraps the second and third steps in the instructions building both specterd and the AppImage.

k9ert commented 3 years ago

Yes, sorry for not mentioning it in the first place. The instructions should definitely mention it. I'm not sure why i wrote these instructions the way i did. I did it as part of https://github.com/bitcoin-dot-org/Bitcoin.org/pull/3626 and for some reason the build-scripts were not enough. Can't remember exactly.

eriknylund commented 3 years ago

No worries, I appreciate your input! If you think it would help I can provide a PR with updated instructions on how to build it from start to end. I've now focused on the Linux build but I can give macOS and Windows a go too and see if I get it all together.

k9ert commented 3 years ago

Oh man, again too much time for answering! Yes, definitely PRs would be highly appreciated! The challenge here is to make it easy for both usecases: Manual builds AND automatic building/releasing with CI-systems.

eriknylund commented 3 years ago

Oh man, again too much time for answering! Yes, definitely PRs would be highly appreciated! The challenge here is to make it easy for both usecases: Manual builds AND automatic building/releasing with CI-systems.

I think I can improve on the manual build instructions and let's see if that brings us closer on the CI-side as well. At least the first step looks reproducible in my tests. Are the releases built in a CI pipeline now, or is it a manual process?

eriknylund commented 3 years ago

Reading more into the docs I guess it's somewhat automated if this https://github.com/cryptoadvance/specter-desktop/blob/master/docs/continuous-integration.md#releasing is still up to date.

k9ert commented 3 years ago

Yes, i think it's mostly up2date. Only the MacOS binaries are not fully automated.