QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
536 stars 48 forks source link

Make Qubes Windows Tools build with MinGW #5065

Closed marmarek closed 1 year ago

marmarek commented 5 years ago

Inspired by https://github.com/tabit-pro/qwt-crossbuild by @easydozen it seems possible to build QWT with MinGW. Lets add qubes-builder plugin for doing this, which would enable automatic builds whenever signed tag is pushed. This would be a major improvement over current manual process involving copying files into Windows VM and back.

brendanhoar commented 5 years ago

For those who just want to build it in a fedora 29 VM today, I scripted a solution you can use in a purpose-configured VM. Thanks to @easydozen / @tabit-pro for all the hard work!

#!/bin/bash

# the following commands worked to build qwt-crossbuild under a new fedora-29 VM, using the trick
# from the mirage firewall build script for QubesOS where you link /var/lib/docker
# into /home/user/var_lib_docker to prevent the system volume from filling up.

# I adjusted the private volume from 2GB to 20GB and the build filled RW to 24%. 
# Possibly could squeak by with 5GB, but I'd go to at least 10GB to be sure.

# my fedora-29 has git installed already, but I keep it here just in case that
# isn't the default

# go home
cd

# wipe (if necessary) and (re)create the ISO RPM output directory
sudo rm -rf ~/qwtisorpm
mkdir -p ~/qwtisorpm

# wipe (if necessary) and (re)create the build directory
sudo rm -rf ~/qwtbuild
mkdir ~/qwtbuild

# prevents weirdness
sudo qvm-sync-clock

# tear down docker if already installed
sudo systemctl stop docker
sudo rm -rf /var/run/docker* # removes dir, .sock and .pid if still there
sudo dnf -y remove docker

# warning: these will delete all docker container instances!!!
sudo rm -rf /home/user/var_lib_docker
sudo rm -rf /var/lib/docker 

# recover some thin provisioned space
sudo fstrim -av

# set up docker but ensure that /rw is used for the docker instance
sudo mkdir /home/user/var_lib_docker
sudo ln -s /home/user/var_lib_docker /var/lib/docker
sudo dnf -y install docker git # git may already be installed under fedora-29
sudo systemctl start rhel-push-plugin # needed in fedora-29 if docker is uninstalled and reinstalled, not necessary if first install
sudo systemctl start docker

# copy source to the build directory - the following worked on 2019-05-31 - the docker build and/or run commands may change over time
# so please see https://github.com/tabit-pro/qwt-crossbuild site to check invocation if the below fails.
cd ~/qwtbuild
git clone https://github.com/tabit-pro/qwt-crossbuild .

# prepare docker environment with WINE in Windows XP emulation mode
sudo docker build -t tabit/qwt .

# build the ISO RPM
sudo docker run -v $(pwd):/src -v ~/qwtisorpm:/build/noarch -it tabit/qwt sh -c "sh prep.sh && make x86 && make x64 && rpmbuild -bb --define '_sourcedir /build' --define '_rpmdir /build' *.spec && cd noarch && createrepo ./"

# show the ISO RPM
cd ~/qwtisorpm
ls -al

# done
easydozen commented 5 years ago

I'm really appreciated by your attention.

It seems the ongoing topic is a well-known task and you are totally aware about caveats. So I'll briefly point out the current build pitfalls:

All the rest are working as expected, afaik.

marmarek commented 5 years ago
  • create/disable-device.exe - could be built, not thoroughly tested, was replaced with devcon.exe from MS driver samples;

Any specific reason for that?

  • prepare-volume.exe - the same, was replaced with conventional diskpart script;

AFAIR the reason for custom binary was detecting which disk should be initialized. Going with "use second disk" was very unreliable, because in some cases Windows swapped the order. prepare-volume.exe looks at Xen disk ID (see GetPrivateImgDriveNumber function).

  • catalog creation - introduced pkihelper (generate self signed certificate, create cat file, install cert to the trusted storage, delete private key);

One day we'll have proper signing cert... But until that day, this solution looks fine.

  • wix package - manifest was reworked to avoid modules.

Indeed with merge modules, wix under wine crashes (SEGV) somewhere in mergemod.dll. Have you tried alternative methods of running wix under Linux (https://github.com/wixtoolset/issues/issues/4381) before abandoning modules? This feature fits nicely into components separation...

easydozen commented 5 years ago

To be honest, all replacements common reason is to simplify not actively supported project, so the code could be manageable with our own efforts.

I understood your worries about devcon, but initial phase is the place where BSODs are located so far. Moreover, device helpers functionality will not be enough in some cases: there is an issue with win7 and vgasave driver; I believe that helpers will not manage device properly in win10 distributions. However, I'll give it a second glance for sure if you are not willing to accept corresponding PR.

Xen disk id (51728) hardcoded at the installation phase. I'm not sure is there a big difference. If any, something similar could be easily scripted too (e.g. wmic diskdrive get SerialNumber).

I've tried to get rid of wine, but with no luck. WiX 3.x with native mono runtime generates corrupted install package. I assume it worth another chance with WiX 4.x manifests adoption.

easydozen commented 5 years ago

I've uploaded several patches to correctly implement MinGW support. Short description is available in the spec file.

Feel free to give me some advice if I could do something to resolve the issue on my side.

marmarek commented 5 years ago

I've done minimal set of changes directly in relevant components, I'll push what I have today (as separate branches + pull requests). This should be good base for adjusting code and adding actual improvements.

I'm struggling with qvideo driver signing. List of problems:

If you have any hints, don't hesitate to share :)

marmarek commented 5 years ago
* I'm trying with separate CA + code signing cert now

It didn't help. Properties of Qubes Video Driver says Windows cannot verify the digital signature for the drivers required for this device. .... But in "Driver" tab I see Digital Signer: QUBES-AUTOGENERATED-CERT. "Driver Details" shows that none of qvgdi.dll nor qvmini.sys is signed (Digital Signer: Not digitally signed) although if I go to those files properties I see correct signature (by "Qubes Test Cert" - the one generated during the build). Not sure how driver signature (signed cat file) is related to signatures on individual files, though. Could it be about missing timestamp? All the guides I've found (and our Windows based build too) include using timestamping service during driver signing.

Anyway, I'll leave debugging this problem for another time and focus now on cleaning up and pushing what I have.

easydozen commented 5 years ago

ossligncode is not deterministic. I'm going to drop it.

pkihelper works most of the time. The following should be enough to build it: /usr/bin/x86_64-w64-mingw32-gcc pkihelper.c -I include/ -I qubes-windows-utils-*/include/ -L qubes-windows-tools-4.0/ -lwindows-utils -o pkihelper.exe

Actually, I've just made a commit to use only pkihelper. I didn't spend much time on it, but it works for me.

marmarek commented 5 years ago

Actually, I've just made a commit to use only pkihelper.

I see you generate separate certificate for each file. Looks like unnecessary pollution of trusted certs store...

easydozen commented 5 years ago

I see you generate separate certificate for each file. Looks like unnecessary pollution of trusted certs store...

Did this to double check if signature list in file properties is showing cert of the signed file and not the cat file's CN. Fixed.

marmarek commented 5 years ago

Not necessary fixed, now you still generate separate cert for each file, but keep only the last one in trusted store. Which means only the last signature will be considered ok by Windows... I think you need to move RemoveCertFromStore + CreateSelfSignedCert + AddCertToStore out of SelfSignFile

easydozen commented 5 years ago

At installation stage both drivers and cat file should be signed. It's enough to have only Catalog's certificate installed in the trusted storage. Then drivers signature will be checked while loading, but IMO there is no support for trusted storage in winkernel. So, while we are not talking about WHQL it's not necessary to sign all files with one cert.

marmarek commented 5 years ago

Everything besides installer-qubes-os-windows-tools pushed and relevant PRs created - all linked to this issue. As for the last component, I need to review/cleanup combined qubes-tools.wxs. Most of the changes are minimal to get thing compile. One place where I've done more significant reorganization is pkihelper (in gui-agent-windows repo), to use proper includes and normal dynamic linking, instead of doing everything manually.

historicbruno commented 5 years ago

Nice work! Do you plan to keep the MSYS(2)-based build as an option? It seems this is rather a merge than a replacement.

I ask mostly because I have been fixing up the MSYS-based build scripts these last few days, and after testing, they will be ready to review. So now, it's possible to run get-be.ps1, copy the modified builder.conf file into place, and the rest is basically automatic as described here: https://github.com/QubesOS/qubes-builder-windows/

It should make a nice little improvement for those building the tools on a Windows system.

marmarek commented 5 years ago

Yes, Windows build will stay there. As you can see, cross-compilation have a number of limitations, so a fallback option is always useful. I haven't decided yet to switch release builds to Linux.

marmarek commented 5 years ago

Ok, all code is pushed and PRs linked here. I've ended up writing qubes-tools-combined.wxs myself, by copying things from other wxs files. I'd like to have any functional changes in separate commits, with proper commit messages - some of the differences wasn't clear to me.

For anyone wanting to give it a try, here is builder.conf you can use:

GIT_BASEURL ?= https://github.com
GIT_PREFIX ?= marmarek/qubes-

BACKEND_VMM = xen
DIST_DOM0 = fc25
DISTS_VM = win7x64-cross

BRANCH = mingw
BRANCH_builder = master
BRANCH_builder_rpm = master

BUILDER_PLUGINS = builder-windows-cross

COMPONENTS = builder-windows-cross \
             builder-rpm \
             vmm-xen-windows-pvdrivers \
             core-vchan-xen \
             windows-utils \
             core-qubesdb \
             core-agent-windows \
             gui-common \
             gui-agent-windows \
             installer-qubes-os-windows-tools

# Uncomment to show build messages on the console
#VERBOSE = 2

# Uncomment for debug build (development only)
#WIN_BUILD_TYPE = chk

# until merged, skip signature check (unsafe!)
NO_CHECK = 1
easydozen commented 5 years ago

What do you think about nsis as a replacement of wix? It looks like electron-builder from the issue above already did this. I haven't tried to make a PoC yet, but in conjunction with devcon (to install drivers) and pkihelper (to manage certs) it could be a better choice - plain imperative install script, available in repositories, no wine and docker. Would it be suitable for Qubes?

marmarek commented 5 years ago

I don't have any experience with nsis, I've just looked at its documentation. My general impression is that it's much more primitive tool, and really "just" a scripting environment you could use to write your own installer. Instead of tool to create installer from a description of desired state. Declarative vs imperative programming. A lot of examples (like this one or this one) contains fragile parts like manually synchronizing state of different components, or manually building control elements - something that in Wix is needed only in custom actions (extra entries in "InstallExecuteSequence") which is rather exception than a main method of implementing installer.

Anyway, that's just my impression from a brief look at documentation, I haven't tried to use nsis before. If you believe nsis can be a tool to create clean and easy to maintain installer, I'll definitely consider it.

For the record: I believe we should use right tools for given tasks. If building Windows software require Windows environment, so be it. Being able to cross-compile from Linux is definitely convenient for us, as most of Qubes OS is developed under Linux. But if doing so would require significant degradation of maintainability and/or readability of the code, that may not be worth it. The need to drop wix modules and use a single wxs source is already on the border (maybe include directive could be used instead of merge modules?). Using open source toolchain for building software ultimately for a closed source platform isn't a requirement, but only a "nice to have". Reproducibility is a nice goal too, but I believe it isn't the only way to achieve it.

easydozen commented 5 years ago

I see your point. Looks reasonable to me. So, I'll better take my time to bring back manifests modularity.

easydozen commented 5 years ago

Modular solution based on wix Fragment element is ready to review. The following are the main points:

@marmarek, please, take a look if it fits your need.

marmarek commented 5 years ago

Looks good! This way both build methods could use the same packaging files. And manual construction of MsiDriverPackages table is not needed anymore.

Todo list:

The acceptance criteria here is working both builds (native and cross) with as little distinct parts as possible. I think Makefile vs vcproj files is unavoidable, but hopefully not much more.

I'll slowly go through the above list, but if you could take care of any of the above (for example the second point), that would help a lot. Please separate changes in separate commits - for example don't mix formatting changes with other changes, as that makes review much harder.

easydozen commented 5 years ago

I hope I've done PRs the right way.

Another one thing that makes the difference in cross vs native builds is ready. All code is borrowed from llvm's compiler-rt builtins.

Makefile vs vcproj

Don't you think that cmake could be usefull here?

I'm going to switch to UX improvements, but if you need any help I'll be around.

marmarek commented 5 years ago

I've pushed a bunch of fixes, un-breaking native build of mingw branch (note force-push in some of them). I've verified the native build produce working package (including qvideo driver for Windows 7). But the mingw build from the same sources produce not working installation. At least qvideo driver doesn't work. Shortly after system startup I also loose ability to connect with RDP (it works only for about 30s, the I get connection timeout, looking at tcpdump, windows very slowly responds). Note that this build still doesn't use wix fragments.

Does the above sounds familiar, is it something fixed by any of remaining changes?

But at least native build works again, so I can proceed with further steps (wix fragments). I'd like to keep native build (and later also mingw build) working between individual changes, so when anything got broken, there is less changes to check.

marmarek commented 5 years ago

As for cmake, I'm not sure, maybe? When @omeg maintained those tools, having visual studio build files was a great help. But from what I see, VS2017 do support cmake, so in theory it should be fine. What is not obvious for me, is drivers build support with cmake. I've found this: https://github.com/SergiusTheBest/FindWDK

Anyway, as said in previous comment, lets make it work first and only then consider subsequent changes.

easydozen commented 5 years ago

Does the above sounds familiar, is it something fixed by any of remaining changes?

I think it caused by mixed flags take a look at -mwindows.

marmarek commented 5 years ago

I've cleaned it up a little, but it still doesn't work. Does qvideo driver work in your build?

easydozen commented 5 years ago

I can't check it right now, but I believe you did the right thing. Driver from your build complained about missing or corrupt driver (error code 39 in device manager). Now it shouldn't. Another one issue is not properly signed driver. I thought that you are unsure about my recent additions to pkihelper and have your own workaround. At the moment your version of pkihelper only creates cat file, but sys and dll should also be signed.

marmarek commented 5 years ago

Recent force-push fix that and when I open cat file, it does have reference to sys and dll files. Also, pkihelper output mention them. Without that, installation complained about not signed driver, but now it install nicely again. But still doesn't work. And since RDP dies quite quickly (which may be unrelated), I can't even get to the error message. qvm-run -p ... cmd.exe works, but I don't know how to access windows logs, or any other useful information from there.

easydozen commented 5 years ago

I'm not using RDP. If debug mode is turned on then vgasave (win7) fallback should be available even with broken qvideo drivers. I'm setting up debugger environment to solve tough issues as described here.

QWT from your sources and with combined wix manifest works for me. So, It's driver setup issue. create-device.exe causes suspicion. I guess, sooner or later we'll have to drop device management utilities and switch to something more conventional. That was the place where I chose devcon. As was promised I could try to investigate utilities futher if you think that we are not ready to go with devcon. What would you prefer?

marmarek commented 5 years ago

I've pushed mingw-fragments branches on all relevant repositories. This include your wix fragments PRs on top of mingw branches. Plus fixes for that conversion. I'm testing if native build still works and I have some mixed results... I think the only real difference is pkihelper usage. I'm hitting #5073 often, which makes debugging inconvenient, so I'll implement some fix for it first.

marmarek commented 5 years ago

Some more info about problems with native build: qvideo driver finishes installation only after reboot. This means qga service initially fails with "No 'Qubes Video Driver' found." error (qgawatchdog restart it multiple times). When it finally loads, then it fails with "EnumDisplaySettings() didn't return expected mode". After another reboot it works fine.

I have no idea how that could be caused by pkihelper... Maybe it's something else? create-device.exe indeed may be responsible, but this is the native build and it worked before.

marmarek commented 5 years ago

Exactly the same sources (mingw-fragments branch) built with mingw produce the same result. Fails exactly the same after first reboot, but works after second! One curious thing is that menu entries, features etc (generally advertise-tools.exe) is done only at this second reboot with mingw build.

easydozen commented 5 years ago

In my experiments result was the same. Windows device management created properly working device on the third boot. I didn't find a quick solution and made a replacement with devcon.

jevank commented 3 years ago

Hi @marmarek , inspired by your post, I`ve made separate repo with our latest stable release and source verifying adapted to qubes-builder. I placed "as is" with our patch set since it had some test experience.

We`ve made some simplifying in component:

I think I`ve missed smth, but hope it helps to move on.

jevank commented 3 years ago

Builds on centos require config_opts['isolation'] = 'simple' to use wine in mock, fedora looks like don`t need it.

brendanhoar commented 3 years ago

@fepitre tagging you as I saw you were doing some work to bring this into qubes builder (I think).

fepitre commented 3 years ago

@brendanhoar, yes I'm aware of this but currently I'm running out of time and especially, some prior tasks are assigned to me.

brendanhoar commented 3 years ago

Ok just wanted to be sure double work was avoided. :)

brendanhoar commented 2 years ago

Submitted a PR to the tabit-pro repo above to allow diskpart.ps1 to work correctly under Qubes R4.0. This enables templating Windows templates. For both R4.0 and R4.1, the update also verifies that target volume is essentially empty before formatting it (reducing the danger of unexpected data loss). This is done by checking the first 32KB of data on the volume for zeros.

Tested ok with Windows 7.

https://github.com/tabit-pro/qubes-windows-tools-cross/pull/6

Brendan

jevank commented 2 years ago

Hooray!