Mingw Build

[josephbisch]

@droark

It is probably easiest to just click on the "Mingwbuild: Initial commit" commit to see my actual changes rather than look at this whole PR, since it also merges ffreeze.

All my changes are in the mingwbuild directory. This PR is currently a way to build Armory as a standalone exe for Windows from a Linux computer. It is also a step towards a fully automatic build. Right now the process is not fully automatic. Once this is fully automated, it can work with Gitian to produce a reproducible Windows build.

I will try my best to give complete step-by-step directions to building the exe. Note that I haven't done an independent, entire build all the way start to finish while following these exact directions, so there may be a mistake along the way. Rather, I am coming up with streamlined directions based on my experience building the exe.

Note that there is currently an issue with ssl somewhere along the way (whether that be with the Python build itself or the freezing process I don't know) that should prevent Armory from connecting to Bitcoin Core. So as it stands, this build is probably useless on an online computer. Before performing the build you will need to comment out smtplib, bitcoinrpc, and urllib stuff (including imports) from the Python files to be able to run the resulting exe.

Note that for this to run, you will need to merge in the python3 branch.

Note that it is easiest to test on Linux using Wine, because of the built in debugger (winedbg).

Instructions

1. Download the sources and uncompress them into mingwbuild/src. Refer to the filenames in the appropriate scripts for the names of the directories and version numbers for Python, Qt, and PyQt. Also download openssl (I used 1.0.1m), psutil (used 2.2.1), sip (4.16.7), Twisted (15.2.1), zlib (1.2.8), zope.interface (4.1.2).
2. Make sure you have mingw-w64 install. It is the mingw-w64 package in Debian-based distros.
3. I don't remember exactly what needs openssl and zlib, so just get those over with. Make sure you set any static flags available to the configure scripts and specify x86_64-w64-mingw32 as the host or export x86_64-w64-mingw32-gcc as CC, x86_64-w64-mingw32-g++ as CXX, etc. Ask me if you need help. In the future there should be a script that handles this.
4. Follow the instructions in build_py3_instructions.txt to build Python 3.
5. Run ./build_qt5.sh.
6. Build sip. Ask me if you need help. In the future there should be a script that handles this.
7. Run ./build_pyqt5.sh.
8. Build psutil, sip, zope.interface, and Twisted with the Python we build for Windows. Ask me if you need help. In the future there should be a script that handles this.
9. Install pyqtdeploy. Run pyqtdeploy Armory.pdy in the mingwbuild directory. In the GUI replace all paths that begin with /home/joseph with the appropriate path for your system. In the future there should be a variable used, so that this works just by setting an environment variable.
10. Go to the build tab in pyqtdeploy. Modify the resource file number so it is higher than one. I use fifteen. If the number is too low, it uses too much memory and mingw-w64 chokes. Unfortunately the resource file number is not saved when you save the Armory.pdy file, so this has to be set every time you redo this. Click the build button. You should end up with green text reading "Code generation succeeded".
11. Exit pyqtdeploy. Create CppBlockUtils.py. We don't need _CppBlockUtils.pyd, because we built the _CppBlockUtils module into Python. But if it is easier, just run the Makefile to go through the entire build process for Armory.
12. Go the mingwbuild/build. Modify Armory.pro so that win32 has INCLUDEPATH as /path/to/armory/mingwbuild/src/Python-3.4.3/Modules/_ctypes/libffi instead of /path/to/armory/mingwbuild/src/Python-3.4.3/Modules/_ctypes/libffi_msvc. Delete the SOURCES line with ffi.c that is also under win32.
13. Go to the next win32 section and find the DEFINES. Make it so it is DEFINES += MS_WINDOWS _WIN32_WINNT=0x0600 NTDDI_VERSION=0x060000 WINVER=0x0600. 0x0600 represents Windows Vista as the minimum version. I don't know what Armory really supports, but 0x0600 has worked for me.
14. Search for mmap. There should be a single line found. Delete the entire line.
15. Close the file after saving. Run ../qt-5.4.1/bin/qmake Armory.pro. Edit Makefile.Release.
16. Add -I'../include' to INCPATH. Add -static -static-libgcc -static-libstdc++ to LFLAGS. Add -L/path/to/armory/mingwbuild/src/Python-3.4.3/Modules/_ctypes/libffi/x86_64-w64-mingw32/.libs/ -lffi to LIBS. Close the file after saving.
17. Go into the resources directory. Run find . -name "pyqtdeploy*.qrc" -print | xargs sed -i 's/BitcoinArmory\///g' Where BitcoinArmory is the name of your Armory source directory (just the directory itself, not the full path). Run mv BitcoinArmory/* .. Run rm -r BitcoinArmory. Again BitcoinArmory is the name of your Armory source directory. This step allows the imports to work correctly. In the future it should be scripted.
18. Move up a directory, back to build. Run make and be patient.
19. The result is in release. It is named Armory.exe. You need to take libpython3.4.dll from mingwbuild/bin and put it next to Armory.exe otherwise Armory.exe won't run.

[josephbisch]

By the way, the patches in python3-arch come from the Arch Linux Mingw Python3 package.

[droark]

Thanks, Joseph. Sounds like you went through quite the convoluted process to get to where you are! It's really good work. More is needed, obviously, but this is a fantastic start.

I do have a few questions and comments before I dive into the code.

• I believe both Python and Qt require, or can at least use, OpenSSL and zlib. I don't know if you have to do anything special in order to compile either one. (I say this mainly because of the SSL issues you're having.)
• Regarding pyqtdeploy, are you familiar with pyqtdeploycli? Do you think this will be helpful? Obviously, the long-term solution can't use the GUI. If there are steps that are just unavoidable, somebody should probably get in touch with the team and see if there's a way to get the necessary work done to make the CLI option feasible.
• Vista is the lowest version we support. I know XP support was dropped a few months ago for reasons I can't remember.
• Placing the DLL next to the binary should be enough. Plenty of Windows programs need to be able to access DLLs at specific locations, sometimes within the same folder or a subfolder.

I'll dive into the code now. It may take a little while to cover everything but I'll do it as quickly as I can.

Thanks again.

[droark]

Oops. I meant to have my comments stick to the code in this PR. Sorry. :) Anyway, it's a bummer that Armory.pdy is so large. A 26 MB file in the codebase isn't the most ideal option. I'm guessing there's no reasonable alternative, unfortunately.

[josephbisch]

In response to @droark 's questions/comments:

I did see pyqtdeploycli and it looks like it has a build mode that can replace the build functionality of pyqtdeploy. I just don't know if it can do the configuration that the GUI can, so I was using the GUI for now. Once the configuration is set to be path setup agnostic so that it works on any system, then we can just use pyqtdeploycli.

Placing _CppBlockUtils.pyd or _CppBlockUtils.dll next to Armory.exe didn't seem to be working for me, but the documentation does say it should work. I guess it would make the build easier, so I can always revisit that. And then we don't have to list all the files in cppForSwig in the build_py3.sh script.

[droark]

Hello. Quick follow-up.

Regarding step #9, you're right, that will need to use an environment variable at some point. I'm not sure what the current best practice is when doing something like that. The variable needs to be something that has very little chance of colliding with anything on the build system. (Then again, in Gitian, I suppose this really wouldn't be a concern.)

Regarding step #10, I assume pyqtdeploycli covers this via the --resources option. So, I'm pretty sure that automated building via pyqtdeploycli is reasonable.

I do have another question about pyqtdeploy. What are the cases when the .pdy file will need to be updated via the GUI? Would we need to update it, say, when adding new .py files to Armory?

I think that covers everything for now. I'll leave this open for a little while before merging it in. If possible, try to figure out why things like bitcoinrpc have to be commented out. That's obviously a non-starter. Also, if you have time, the SSL issue also needs to be covered.

Thanks.

[droark]

Hello. I noticed a couple more things.

• The size of the .pdy file led me to believe it's a binary file. It's not. It's a text/XML file. One thing worth considering is uploading a .tar.gz (or perhaps .tar.bz2 or .tar.xz) version so that space can be saved in the repo. 26 MB is a bitter pill to swallow but a zipped version should be okay. It can be decompressed on the fly, and I can't imagine developers wanting to search the contents under normal circumstances.
• For troubleshooting purposes (and possibly implementing workarounds), you can edit the PDY file directly. For example, I see a " ssl="1" " line in there. Maybe that's one of the issues you're having? (To be fair, I don't know what the flag does.) It's also possible to search for all instances of any strings you wish to replace, although this isn't recommended due to length of time it'd take to execute.

Thanks.

[josephbisch]

@droark

Regarding step 9, the environment variable can be something like ARMORY_MINGWBUILD_SYSROOT, because the pyqtdeploy tutorial uses SYSROOT and adding ARMORY_MINGWBUILD should prevent any potential conflicts.

Regarding step 10, it does look like the --resources option handles it.

The .pdy files needs to be updated when adding new .py files that need to be part of the Windows build, when changing the Python or Qt version, when changing to/from PyQt 4 or 5 (but we should just stick with PyQt 5 for Windows, so I don't see that changing), when modifying the PyQt modules that Armory uses, when adding or removing parts of the Python standard library from what we freeze (we don't include every single Python lib for file size reasons and because we might not be able to successfully build all of them), and when adding new Python modules to site-packages (for example, we already have Twisted there, but if we ever needed a new 3rd party library, it would have to be ticked in the pyqtdeploy GUI). AFAIK, all those cases require use of the pyqtdeploy GUI or some way of modifying the pdy file. The pyqtdeploycli interface doesn't seem to support modifying the pdy file.

I'll replace the .pdy file with a .tar.gz version.

I think that "ssl="1"" line corresponds to the "Enable optional SSL support" checkbox in the pyqtdeploy GUI. I enabled it to try to fix the SSL issue I was having, but it didn't seem to fix anything. But that is a good idea to look at the pdy file directly.

Also, I think there is a little misunderstanding. The things like bitcoinrpc need to be commented out for now, because of the SSL issue. All those things seem to be networking related and so importing them causes an error related to (if I remember correctly) the ssl.pyo file included with the exe.

[droark]

Hello.

• ARMORY_MINGWBUILD_SYSROOT sounds good.
• Thanks for clarifying when the .pdy file has to be updated. You're right, the GUI version seems to be mandatory for putting everything together. Maybe this will change in the future. For now, I suppose we'll just treat the file like the MSVS build files currently used for the Windows build (and maybe still maintained in the future as an alternative).
• Instead of .tar.gz, I'd prefer .tar.xz if possible, or .tar.bz2 as an alternative. I'm more concerned about size than anything else, although the decompression tools will obviously need to be built into the correct system. (If .tar.gz is all that's available, it'll be fine.)
• Sorry about the SSL flub. Would you happen to know the exact ssl.pyo error? I think that's a Python 3.4 file. I'm guessing all this has something to do with how OpenSSL is being built or connected to Python.

Thanks.

[josephbisch]

Hi,

• It actually looks like I can just use relative paths and not need the environment variable, so I'll just go with that.
• I'll go with .tar.xz then.
• The SSL error was something about a "slot offset". I just looked at the Modules/Setup file and it looks like I have to uncomment some stuff to build the _ssl socket module helper (which links to OpenSSL) and then the error shouldn't occur.

[droark]

That all sounds good to me.

[josephbisch]

Whoops, I still used .tar.gz even though I said I would use .tar.xz. I'll fix that another time.

[droark]

Hello. No worries about the .tar.xz thing. :) Anyway, I did open the original .pdy file in pyqtdeploy and used the pyqtdeploy tutorial to guide me. The file looks pretty good overall. I do have a few questions.

• Just checking, is the page about static packages and pyqtdeploy relevant?
• Why are asserts disabled? I'm pretty sure we need them enabled.
• The files & directories chosen under "Application Package Directory" need to be altered. The things that I know can be left out are the cppForSwig/gtest, po and pytest directories, and any README text files. I'm pretty sure there's more, especially in cppForSwig, but I haven't done a thorough search. (An exhaustive sweep isn't necessary for now.)

Thanks.

[droark]

Well then, that was simple. I got an email just now explaining the current dependencies. :) Here's the relevant stuff. This is taken from Windows_Build_Instructions.rtf and should be up to date. Note that it doesn't mention anything related to Autotools or Gitian, which is expected. It also doesn't mention libpng or Webkit, both of which are currently used by the OS X build, along with setuptools or pip (both related to compiling the OS X build.)

---

Windows build instructions shows the following python dependencies:

*Python 2.7.8
*Python psutil 2.1.3
*Python twisted 14.0.2
*Zope interface 4.1.1
*Python-Qt4
*SIP (required by Qt4)
*pywin32 2.19
py2exe
SWIG 3.0.2

The starred dependencies are the ones that are actually code that is leveraged in the Armory project. The non-stars are tools that are used to help build Armory or its executables/distributables.

The C++ dependences:

*LMDB
*Crypto++  (not all source is included... only most of it)
MSVS 2013
gtest
SWIG
py2exe
NSIS

Again, anything without a star is just a tool for helping make the code work, primarily as a build/release process, not actual code execution when Armory is running.

[droark]

Hello again. One more thing. You can leave LICENSE out of the .pdy file alongside the directories I mentioned. However, that file needs to be copied into the same directory as the final binary so that users can read it.

Thanks.

[josephbisch]

• The page about static packages and pyqtdeploy isn't entirely applicable in our case. For example, looking at the configuration code and patches for Python, it doesn't look like it would be suitable for Mingw cross-compilation from Linux. I also didn't use pyqtdeploycli configure for PyQt or Qt, because I was able to use the script and config file from the OS X build as a basis. I did use pyqtdeploy configure for sip as outlined on the static package page.
• Asserts were disabled by default and I didn't look at what it did. I'll enable them.
• I think the entire cppForSwig directory can be left out, because we don't need it anymore once _CppBlockUtils is built.
• Regarding the dependencies, it looks like I am building most of the Python dependencies (but using Python 3/Qt 5). The exceptions are pywin32, py2exe, and SWIG. Py2exe is of course unnecessary since we are using pyqtdeploy instead. For now I am using the version of SWIG from my package manager, but we probably want to use the native_swig descriptor when moving over to Gitian. Pywin32 looks like it can potentially be a Windows-specific psutil replacement, so it might be a good idea to look into cross-compiling it. Just because it is Windows-centric, I doubt they put much effort into making it cross-compilable. I'm not sure how Armory currently actually depends on pywin32, because I never ran into issues with it not being installed when running this Mingw Windows build. I guess maybe the VS project adds some special Windows-specific Python code?
• Regarding the C++ dependencies, LMDB and Crypto++ are included in the source tree. MSVS, py2exe, and NSIS all seem to not be applicable to the Mingw build. SWIG, like above, can be installed via a package manager for now. And gtest can probably also be installed via a package manager for now. We can see about adding gtest to the depends system if we want to run tests as part of the Gitian build process.

[droark]

Hello.

• Thanks for clarifying re: the static package page. I had a feeling. Just checking.
• Just to clarify, everything checked under APD gets packaged in the resultant binary, correct? The tutorial's a little vague on this point, and I can't seem to find any other references out there.
• AFAIK, your assumptions about the C++ dependencies are correct. Crypto++ and LMDB are included in the codebase, along with the Google C++ Testing Framework (gtest). The unfortunate problem with compiling gtest is that it does add quite a bit of bloat, and it only tests Armory C++ functionality. It actually creates its own executable binary. We don't want it to be part of the Armory binary. I'm not even sure how one could execute it, honestly, unless it happened to be built alongside the Armory binary and had direct access to the compiled library.

Thanks.

[josephbisch]

• Yes, everything checked under APD does get packaged in the resultant binary. Plus whatever pyqtdeploy magic that allows the whole thing to work.

[droark]

Cool.

Also, regarding pywin32, what I'm told is that ctypes relies on it. (Most ctypes references are in subprocess_win.py. So, I'm not sure what's going on there.

[josephbisch]

I am just putting some notes here for future reference.

I got the _ssl module to build, but importing ssl still causes the same slot error with _ssl. Not quite sure what I should do. I tried running almost all of the Openssl test exes and they ran fine, so I don't think there is any issue with the Openssl build itself.

And now _ctypes fails. Here is the relevant output for _ctypes:

build/temp.mingw-3.4/home/joseph/git/BitcoinArmoryJosephBisch/mingwbuild/src/Python-3.4.3/Modules/_ctypes/callproc.o:callproc.c:(.rdata$.refptr.IID_ISupportErrorInfo[.refptr.IID_ISupportErrorInfo]+0x0): undefined reference to `IID_ISupportErrorInfo'

I am linking wsock32, ws2_32, ole32, and oleaut32 which is what allows me to only get that single undefined reference. I'm not sure which Windows dll needs to be linked to get rid of that last error.

[droark]

Hello. IID_ISupportErrorInfo is related to the Interface ID (IID) concept from Microsoft's Component Object Model (COM). It looks like you need to include the UUID library. Google it and you should find more info. Here's an example.

Not sure about the SSL module. I did some Google searching and couldn't find any info. You may need to ask around.

[josephbisch]

Thanks, adding -luuid allows _ctypes to build.

What is interesting with _ssl is that if I enable it in Modules/Setup it causes _ssl-cpython-34m.dll to stop building. Definitely not what I expected. I'll ask around. I uninstalled the libssl-dev package on my computer, so I know there is no conflict with the wrong ssl library being picked up.

[droark]

Hello. I spent a few minutes looking to the slot offset issue. (I assume it's "invalid slot offset"?) I might have found the problem. Unfortunately, the guy didn't show exactly how he fixed it. I'm guessing the basic idea is correct, though. There's probably a bad link happening somewhere.

[josephbisch]

I saw that and tried removing all libssl.{a,so} instances from my computer (other than the version I built in mingwbuild). But I still get the following error on import of ssl after rebuilding:

import _ssl             # if we can't import it, let the error propagate
RuntimeError: invalid slot offset

Maybe the guy means I have to use some other version of Openssl than 1.0.1m? It's weird though, because I am using the latest 1.0.1 version of Openssl and the latest Python 3 release, which you would think would be compatible.

I had sent an email to the python-help mailing list earlier today, but I haven't received a response yet. It looked like the most appropriate list for my issue. Unfortunately it isn't public, but I'll let you know if I receive a response from one of the "Python 'Wizards'" that monitor the list. Otherwise, I don't know where to turn, because I don't know for sure that I have a bug here and I am using a bunch of unsupported patches, so it doesn't seem appropriate to file a bug against Python.

[droark]

Thanks for the update. I guess we'll see what unfolds. In the meantime, this needs to be wrapped up so that the importing of the Gitian/Autotools work into 0.94 can begin, as discussed earlier this week. I'll leave this PR open for now in case there are any updates.

[droark]

Hello. Just thought of something. I'm pretty sure that, for now at least, we don't care if Qt uses OpenSSL. We should probably disable it for now. There's info here. If that leads to problems, you may want to try the static linking option.

Thanks.

[josephbisch]

Okay, I'll disable OpenSSL for Qt. If OpenSSL isn't necessary for Qt, then maybe the ssl module as a whole is unnecessary for our static Python to run Armory? I did a quick grep for "https" on the Python files and it looks like the only place it is used directly is for announcement fetching, but those are signed anyway, so it doesn't seem to matter from a data integrity standpoint whether HTTPS or HTTP is used. I also did greps for "ssl" and "SSL" but didn't find anything. So it might be easier to remove the ssl module from the Windows build rather than try to get it to build successfully.

What do you think? I am trying to get creative here with ideas to avoid needing to build the _ssl module, because, no matter what, it just doesn't seem to be building without the "invalid slot offset". And I don't think we can even rule out for certain an issue with the OpenSSL build rather than with the Python build, so I may be "barking up the wrong tree" by trying to fix some ssl-related issue with the Python build. Though I did try running some of the tests and played around with openssl.exe and both seemed to run fine, so it seems like the OpenSSL build was successful.

[droark]

Hello. I have mixed feelings about leaving out SSL altogether. I understand your point regarding switching from HTTPS to HTTP. I've seen others make that argument too. I think it's appropriate in some instances. Here, I'm not sure. For example, when using the Secure Downloader, Armory reaches out to an external server via HTTPS and retrieves not only a list of appropriate downloads but a set of SHA-256 (I think) hashes. Using HTTP, a malicious actor could easily modify the link to make Evil Armory available to a user. Adding hard-coded hashes doesn't really work if we have to push out an emergency patch or something similar. So, I'm not sure that we can drop HTTPS, unfortunately.

Here's my suspicion. I think Qt uses OpenSSL for networking and for some hash functionality. AFAIK, Armory doesn't care about either one in the context of Qt. So, I think it's safe to disable OpenSSL for Qt but maybe not for Python. If nothing else, disabling OpenSSL for Qt would assist with troubleshooting and possibly reduce code bloat.

Another option would be to go back to an older version of OpenSSL. 1.0.1m, perhaps? I'd say it's worth trying at least once.

Thanks.

[josephbisch]

Well, by looking at the download link, we can see that all the data, including hashes, is signed by some presumably hard-coded Bitcoin address. So it is not the hashes themselves that have to be hardcoded. Rather it is the public key/address signing the data, which appears to be how Armory already does it. I don't know exactly how Armory handles bad signatures, but it should handle it somehow other than present the downloads to the user. So I don't think there is any risk of a user getting Evil Armory once they already have a version of Armory so that they can use the Secure Downloader, even if the data is served using HTTP.

However, I understand if we should still try to get the ssl module working.

I originally tried OpenSSL 1.0.1m, because I had the source to that from the depends system lying around. I also tried 1.0.2c, which is the newest 1.0.2 version. I found out that the official Windows bundle for Python 3.4.3 uses 1.0.1j, but I haven't tried that yet. So it might be worth trying 1.0.1j, but I don't see any reason why 1.0.1j would work but not 1.0.1m, based on the changelogs between 1.0.1j and 1.0.1m.

[droark]

Just as a point of reference, BC Core is going to use Qt 5.5. When we get back to this one day, it might be interesting to see if Cory has changed anything. I didn't really notice anything at a glance that would be of much interest to us, like something to address the SSL bug we're seeing (uggh), but who knows.

[josephbisch]

Thanks, I hadn't yet seen the latest BC Core PR for updating Qt. I don't think it would have any effect on the SSL bug we're seeing, because that is just related to the fact that the SSL module for Python isn't being built successfully. But in any case it is good to keep up with updates to dependencies.

[josephbisch]

I got Python to build with Mingw such that the ssl module successfully imports. These are some notes for future reference.

I used OpenSSL 1.0.1p instead of 1.0.1m like I had been using. I don't know if that made any difference.

In Modules/timemodule.c in the Python source tree, add the following lines:

#if defined(MS_WINDOWS)
#include <winsock2.h>
#endif

In crypto/cryptlib.c in the OpenSSL source tree, change line 724 (shortly before the DllMain definition) to be:

#if ((defined(_WIN32) || defined(__CYGWIN__)) && !defined(__MINGW32__)) && defined(_WINDLL)

Notice how some parentheses and the check for __MINGW32__ were added, so that DllMain isn't defined if we are using Mingw (__MINGW32__ is defined whether we are using 32-bit or 64-bit Mingw). Without the check for Mingw, the DllMain definition was clashing with the DllMain definition in the Python DLL. I'm not entirely sure if checking for the __MINGW32__ definition is the correct way to handle the issue.

[droark]

Closing out per our discussion. The work will be archived for future usage. Thanks for figuring out the final bug! That was unexpected and greatly appreciated. We can narrow down the exact issue later.