Cross-compiling (ungoogled) Chromium for Windows from Linux

[iskunk]

...without touching a Windows PC.

This should make possible a u-c build that is...

• More easily reproducible and CI-friendly, since no manual "install Visual Studio on your PC" step is involved;

• More hermetic, in that we don't have to worry about said PC having been surreptitiously compromised;

• Faster, in line with the experience of others;

• Less costly to run, given how GitHub double-counts each minute of GitHub Actions that runs on Windows as compared to Linux.

Google has posted official instructions on how to cross-compile Chromium for Windows, but that documentation makes some CI-averse assumptions (packaging up a manual Visual Studio installation) and glosses over several important details (e.g. SetEnv files and the rc executable).

Even more unfortunately, I could not find that anyone has distilled those instructions into a working setup that others can use as a starting point. So, given that I've already built this beast (for Linux) more times than I care to count, I figured I'd take a stab at it.

To begin with, I'd like to draw your attention to a nifty little project called msvc-wine. They have put together a Python script (vsdownload.py) that can download any of Microsoft's current SDK offerings, without needing to run some setup/installer/wizard thing. This eliminates the requirement of a Windows PC altogether.

For compiling Windows resource files, Google has a re-implementation of rc that they use in the cross build. The source is available here. (That may seem like a random location, but it is directly referenced in the Chromium tree. Nico appears to be a Googler.)

Many guides for performing cross builds have some variation of a "put the Windows SDK on a VFAT-formatted filesystem" step, to recreate the case-insensitive lossage of the target OS. (You have winsock2.h vs. WinSock2.h, advapi32.lib vs. AdvAPI32.Lib etc., and files are often referenced via both forms.) This was more hassle than I wanted to deal with, so I wrote up a script (case-fold.sh) that creates a bunch of symlinks to get the same effect.

And with all that, I've put together tooling to build a Docker image that provides a suitable Linux environment (based on Debian unstable or Ubuntu mantic) for performing the cross build. I've already run through building 119.0.6045.159 with it several times.

Edited 2024-04-09: Please see current information and usage instructions here. This work has come a long way since the initial post.

Original, now-obsolete verbiage

Unfortunately, the build is not fully working yet---with `-k 0`, it gets a bit past the halfway mark before stopping. While all the tooling issues I've come across have been addressed, there are a number of funky C++ errors that I can't make heads or tails of... things like implicit `StringPiece` conversions not working correctly. I'm using Clang 17, so I don't think it's an issue of the compiler not being new enough. I'm hoping that folks who have more experience with this codebase, and with the Windows side of things in general, will be able to make sense of what's going on. [Here](https://github.com/ungoogled-software/ungoogled-chromium-windows/files/13496245/win-cross-compile-errors.txt) is a log file produced by a second invocation of `ninja ... -k 0 chrome` that shows all the error messages, strung together. If you are successful in reproducing my build, then this is what you should see. Steps to get this working: 1. `git clone` the relevant [source location](https://github.com/iskunk/ungoogled-chromium-windows/tree/feature/win_cross) onto a system with a Docker host; 2. Edit the top of the `Dockerfile` to choose Debian or Ubuntu as your base (it should not take much work to adapt the image build to some other distro, like Arch, but this is what I can provide), and edit `APT_MIRROR` to point to a faster/local APT host if you have one; 3. Edit the `setup.sh` script to uncomment the `accept_license` variable, by which you accept the Microsoft EULA mumblemumble; 4. Build the Docker image with `make build-image`; 5. If that is successful, then you can start a container with `make run`; 6. Unpack a Chromium source tarball inside the container; 7. In the top-level directory of the Chromium source tree, run the `~/prepare-source.sh` script. This will fix a few minor source issues, put some necessary files into place, and run `gn gen` with a configuration that is known to work; 8. Run e.g. `ninja -C out/Release -k 0 chrome`, and see how far it gets; 9. To delete the image, exit the container and run `make clean`. Of course, you will have to customize the container workflow to suit your needs (add mounts, change the user ID, etc.). Some points to keep in mind: * All the Microsoft SDK stuff is installed under `/opt/microsoft/`, and the Chromium-relevant environment variables are set accordingly; * Google's `rc` is available in source form under `/usr/local/src/`, and binary under `/usr/local/bin/`; * All the necessary compiler and library (distro) packages that I'm aware of are installed. More may be needed by later stages of the build; * Additional case-variation symlinks under `/opt/microsoft/` may be needed by later stages of the build. These will have to be added to the `case-fold.sh` script; * If you build the image more than once, then I suggest tarring up the files under `~/cache/`, copying that out, and then uncommenting the `ADD ... cache.tar ...` directive in the `Dockerfile`. This will allow you to run through the MSVC "install" process again without re-downloading the ~2.5 GB lot from Microsoft; * The scripts are reasonably commented to explain what's going on, so please feel free to read through them beforehand; * I am more than happy to reproduce and sort out tooling issues as you encounter them. What I need help with is the Chromium codebase itself, as the C++-fu in use is beyond my ken; * As you can see, I haven't even bothered to bring in the u-c patches or tooling yet. Gotta walk (i.e. build plain Chromium) before you can run, I say! But that is naturally the end goal. Some additional notes: Google has a [CI instance](https://ci.chromium.org/ui/p/chromium/builders/ci/linux-win_cross-rel/) covering this cross build, but it has been broken for months. The most recent successful build that I could find (without spending too much time looking for it, given the poor UI) dates from [the end of May](https://ci.chromium.org/ui/p/chromium/builders/ci/linux-win_cross-rel/29086/overview). Here is the GN config they use, though this doesn't work for our purposes (e.g. we need `use_sysroot=false` since we're not using a Linux sysroot): ``` dcheck_always_on = true is_clang = true is_component_build = true is_debug = false llvm_force_head_revision = true symbol_level = 1 target_os = "win" ```

[uazo]

hello everyone, I'm joining the discussion which interests me since I also do a cross build. I hope I am not in the way.

I could not find that anyone has distilled those instructions into a working setup that others can use as a starting point

you are right about that, instructions are not complete and i often have problems generating dependencies. i would like to have an automatic process

They have put together a Python script (vsdownload.py) that can download any of Microsoft's current SDK offerings,

interesting, thanks for finding it! Do you mind if I take a cue from it?

The source is available here.

correct, at the beginning (last year) I would download it and build it (https://github.com/uazo/bromite-buildtools/blob/master/images/chr-source/prepare-build.sh#L77) but then I saw that it is already available (pre-compiled by google https://github.com/uazo/cromite/blob/master/tools/images/bromite-build/pre-start.sh#L70)

Many guides for performing cross builds have some variation of a "put the Windows SDK on a VFAT-formatted filesystem" step, to recreate the case-insensitive lossage of the target OS.

This is what I do, I use a case-fold disk (the runner is on my server) but your solution is intriguing

And with all that, I've put together tooling to build a Docker image Unfortunately, the build is not fully working yet

you can find my solution in my repo if you want to compare it, I use docker in docker and sysbox for the build but I think it is also applicable in the github runner if disk space permits. the build has been working for years without any problems, you can also find the images in docker hub

[iskunk]

Hi @uazo!

They have put together a Python script (vsdownload.py) that can download any of Microsoft's current SDK offerings,

interesting, thanks for finding it! Do you mind if I take a cue from it?

Go right ahead! The msvc-wine project figured out a solution to a rather tricky problem, and we should all take advantage of it.

Just be sure to include appropriate caching logic, so that the MSVC files are not downloaded from Microsoft more than necessary. The vsdownload.py script has a --cache option, similar to other package-downloading tools, that helps with this.

The source is available here.

correct, at the beginning (last year) I would download it and build it (https://github.com/uazo/bromite-buildtools/blob/master/images/chr-source/prepare-build.sh#L77) but then I saw that it is already available (pre-compiled by google https://github.com/uazo/cromite/blob/master/tools/images/bromite-build/pre-start.sh#L70)

It's better to avoid non-distro binaries, just on principle. While I doubt Google would provide a bad binary, if (for example) you ever want to run the build on a different platform, then you'll have to reach for the source code anyway.

Many guides for performing cross builds have some variation of a "put the Windows SDK on a VFAT-formatted filesystem" step, to recreate the case-insensitive lossage of the target OS.

This is what I do, I use a case-fold disk (the runner is on my server) but your solution is intriguing

There's no need for the full generality of a case-folding filesystem, because SDK files are not accessed via arbitrary case permutations---in all instances I've seen, the build needs the lowercase name, a canonical mixed-case (or all-uppercase) name, and that's it. So symlinks are enough for that.

you can find my solution in my repo if you want to compare it, I use docker in docker and sysbox for the build but I think it is also applicable in the github runner if disk space permits. the build has been working for years without any problems, you can also find the images in docker hub

Thank you, this is very helpful!

I tried building with your patches and GN args, and... I'm happy to say... I got a successful build :smiley:

I had to make some updates to case-fold.sh and a couple other scripts, which I've already pushed to my branch. And now I have to figure out the salient differences between your configuration which worked, and mine that didn't. But this unblocks me, so I greatly appreciate that!

Now, looking at your setup... I see you're using depot_tools, gclient, goma, the casefold location... there are a lot of moving parts. I set out from the beginning wanting to avoid all that kind of custom tooling; I wanted a build that was similar in "shape" to building a package for a Linux distro.

One benefit of having the Google tooling in working order, however, is being able to identify exactly what Microsoft SDK version they're using. The vsdownload.py script lets you specify a version, but otherwise it downloads the latest of what's available from Microsoft. Between last week and now, I noticed that the latest SDK version got updated (more specifically, I saw that the script downloaded some files in addition to using the --cache directory I provided, whereas before, it didn't download anything). We would be better off matching the SDK version we use with whatever Google's using, to avoid any build issues due to minor incompatibilities between the two. But we would need a (reliable, automatic) way of telling what version Google is using.

[uazo]

Just be sure to include appropriate caching logic, so that the MSVC files are not downloaded from Microsoft more than necessary.

thanks but i don't think it's necessary for my purposes. after all, it's not possible to distribute the result (the ms licence doesn't allow it) so it'll probably be something i'd launch with every change of major (once a month).

So symlinks are enough for that.

In fact, I like your solution. the only problem is that it must be maintained.

Thank you, this is very helpful!

I am glad.

I see you're using depot_tools, gclient, goma, the casefold location... there are a lot of moving parts

yes, some things I don't use anymore and I could get rid of them, but it's just a matter of priorities and unfortunately doing everything myself is quite a challenge. if you have any doubts, just ask!

is being able to identify exactly what Microsoft SDK version they're using.

sure, here it is https://source.chromium.org/chromium/chromium/src/+/main:build/vs_toolchain.py;l=43;drc=8e78783dc1f7007bad46d657c9f332614e240fd8?q=build%2Fvs_toolchain.py&ss=chromium%2Fchromium%2Fsrc

[iskunk]

thanks but i don't think it's necessary for my purposes. after all, it's not possible to distribute the result (the ms licence doesn't allow it) so it'll probably be something i'd launch with every change of major (once a month).

This would go e.g. in a runner's non-public storage area. Of course the files can't be redistributed, but that doesn't rule out retaining them for a future run.

So symlinks are enough for that.

In fact, I like your solution. the only problem is that it must be maintained.

At worst, it would be sporadic breakage due to some new MS header being pulled in via a mixed-case name. I would not expect that to happen on the regular.

Alternately, one could scan the MS headers and the Chromium source for new mixed-case names, and create those symlinks ahead of time. But that could end up being a lot of work up front to avoid a relatively small amount of work in the future...

I see you're using depot_tools, gclient, goma, the casefold location... there are a lot of moving parts

yes, some things I don't use anymore and I could get rid of them, but it's just a matter of priorities and unfortunately doing everything myself is quite a challenge. if you have any doubts, just ask!

Will do, thank you :smile: I hope my work can make the task easier to deal with.

is being able to identify exactly what Microsoft SDK version they're using.

sure, here it is https://source.chromium.org/chromium/chromium/src/+/main:build/vs_toolchain.py;l=43;drc=8e78783dc1f7007bad46d657c9f332614e240fd8?q=build%2Fvs_toolchain.py&ss=chromium%2Fchromium%2Fsrc

Ah, so that's where it's controlled. That covers the SDK version... now, there is also a VC tools version in the mix (i.e. $WINDOWSSDKDIR/VC/Tools/MSVC/14.38.33130/). The vs_toolchain.py script has that MSVC_TOOLSET_VERSION table, but it only yields VC143, not 14.38.33130. Would you know offhand how the specific tools version is obtained? I'm not seeing anything obvious in the script.

[uazo]

VC143, not 14.38.33130 Would you know offhand how the specific tools version is obtained?

no, sorry, I don't know.

[iskunk]

All right, I've pushed a major update to my branch. Here are the highlights:

• The prepare-source.sh script is gone, replaced by new patches under patches/general/ and a provided args.gn file. If you are building the upstream Chromium tarball, then this is what you'd use;

• Otherwise, I've started integrating this into the u-c build. New patches have been added to the patch series, and a couple of existing ones tweaked. NOTE: The non-cross Windows build should not be affected, as any different behavior is conditionalized on a Linux host platform;

• There is now a modified build.py script under cross-build/ to drive the build. Eventually, this should get merged into the top-level build script (mostly it's just conditionalizing out the Windows-isms);

• Re-generating IDL source files is needed in the u-c build (as the pre-generated files get pruned), and because this can only be done with Microsoft's MIDL, I've integrated the use of Wine into the Docker image. You can see the setup for this in the new windows-run-midl-via-wine.patch. Thankfully, this is all that Wine is needed for.

I've confirmed successful cross builds for Chromium 120.0.6099.109, with both the (lightly patched) upstream source, and the u-c patched source.

[uazo]

I've pushed a major update to my branch.

Uhm... sorry, where is it? can it be seen?

[iskunk]

I've pushed a major update to my branch.

Uhm... sorry, where is it? can it be seen?

Of course; everything is here: https://github.com/iskunk/ungoogled-chromium-windows/tree/feature/win_cross

[uazo]

everything is here

thanks!

I've confirmed successful cross builds for Chromium 120.0.6099.109

very complex for my skills (few to tell the truth), congratulations. I think I will try the part of the prerequisites and the casefold :) I will have to study it. I will let you know.

[iskunk]

very complex for my skills (few to tell the truth), congratulations. I think I will try the part of the prerequisites and the casefold :) I will have to study it. I will let you know.

The case-fold.sh script should now be complete, in terms of the symlinks it creates. It'll also cover x86 as well as x64 (though x86 cross builds are currently broken for other reasons).

I haven't had time to write up a proper README on this, unfortunately---I pushed this out now to get ahead of the holiday rush---but if you run into any difficulties, please feel free to ask about it. (Likewise for anyone else following this!)

[PF4Public]

@iskunk Would it be possible to document your findings as a readme for this repository?

[iskunk]

Hi @PF4Public! Another big update is in. I've added a README file with the new build process, not least because the one described at the top of this issue is obsolete. I saw Rust is now required for the build, and that has been covered as well.

The whole build is working for me. I get what are presumably working installers with presumably working browsers at the end of the run. But I have no means of testing them, because any Windows box over which I have a modicum of authority tends to become a Linux box ^_^ So at this point, I am really needing others to try out this build process, and especially to verify that the end result actually works.

One significant change: I was originally planning to augment the top-level build.py script so that it can drive the cross build as well as the regular one. But eventually, I figured... there's so much heavy lifting that script has to do on Windows, that is just not applicable in the Linux environment. This was basically going to end up spamming if arch == 'win32': everywhere. It was clear that I could write a much shorter and simpler build script in shell, hence the new build.sh file. As a bonus, it even supports re-starting a previously-interrupted build (implemented for my own sanity's sake, because unpacking that ginormous tarball takes a long time even on the monster systems I'm testing this on).

Again, I really need feedback on this. The build process itself, as far as I can tell, is feature-complete at this point. It's now about making this easy/robust for others to use, documenting it where needed, and of course, making sure that the resulting binary doesn't immediately go splat. There's not a whole lot further I can take this by myself.

(Note: I've just confirmed that x86 build support is currently broken, due to a missing 32-bit header file. Will be looking into that.)

[iskunk]

I've updated the branch, and the x86 build is working now. (I doubt it was working previously.) It needed a few additional packages in the build container, and oddly enough, a new patch.

[PF4Public]

Do you plan to submit a PR eventually?

[iskunk]

Of course. I could first use some independent validation of what's there, however. It's not a good look to say, "I don't know if the resulting build actually runs, 'cause no one's tried it." As well as the squishier points of approachability and ease of use---I'm too close to the work to evaluate this myself.

[uazo]

I'm too close to the work to evaluate this myself.

would you allow me? you should investigate the use of ciopfs by the chromium toolchain, I think it would be the ultimate solution, whether it works.

[iskunk]

Hi again @uazo,

By all means, feel free to try this out, and let me know if you run into any hard spots. If you are working on Windows, you may want to try using Rancher Desktop for the container runtime. Eventually, I would like that to be the path for non-technical users on Windows who want to run this build.

I get what ciopfs is doing, but anything involving mounts is dicey inside a container. Symlinks are a lot simpler to deal with. The only advantage that a case-insensitive filesystem has over that is that it will handle any new accessed files that come up in the future, whereas symlinks have the potential of breaking. But this does not appear to be a common occurrence---when I re-ran this build, after months of not touching it, there was no breakage due to missing symlinks.

[uazo]

By all means, feel free to try this out

I will definitely try the sdk generation script, I will let you know, but don't be in a hurry, it's in the queue, too many things to do alone.

If you are working on Windows, you may want to try using Rancher Desktop for the container runtime.

My setup is a bit more complex. I currently work in windows on a remote docker on linux connected to a specific docker context and mapped via an ssh socket on the filesystem. it allows me to quickly switch from one server to another (I currently have three at my disposal, one worse than the other :)

I get what ciopfs is doing, but anything involving mounts is dicey inside a container

Sorry, I didn't realise that. well, if you are in a container, I can already tell you that it doesn't work. I had tried before setting up my infrastructure.

But this does not appear to be a common occurrence---when I re-ran this build, after months of not touching it, there was no breakage due to missing symlinks.

certainly, I too have things that I know will work until they break... (and it normally happens sooner or later :)

[iskunk]

Hi @uazo, have you had a chance to try out the cross build?

I've been on a tear lately, pushing dozens of branch updates to get this up and running as a GitHub Workflow, and making numerous fixes/adjustments as a result. Still no test of the final binary, however.

[uazo]

have you had a chance to try out the cross build?

I'm sorry, I don't understand. you mean in your mode? I already do a windows cross build all the time.

[iskunk]

I'm sorry, I don't understand. you mean in your mode? I already do a windows cross build all the time.

I should have been clearer; yes. I could use some independent confirmation that the process I've put together is usable by more than just me, and hopefully feedback on any issues and potential improvements. A big part of this work is making the cross build more accessible, so that it can be run by the same enthusiast-but-not-developer users who currently compile Chromium/ungoogled-chromium themselves.

The framework remains a work in progress, however. On my end, the issue is no longer about getting the cross build working, but about making it work reproducibly. So I hope you're up for a few rounds...

[uazo]

sorry, do not take offence, but I am not interested to test your solution because I do not consider it reliable. i am interested in your work with the download of microsoft sdk, i will try that.

try to understand me, I am focused on development, and every toolchain-related problem for me is work that annoys me and wastes my time. I think you should ask the maintainer of this repo if he is interested in your work for testing, if he finds benefit in it.

[iskunk]

I'm only asking for testing and feedback, not for this to replace any aspect of your current development process.This work occupies a very narrow technical niche that relatively few developers even approach, so your feedback would be quite valuable. Toolchain work is not glamorous, but the payoff of a robust and reliable end product is worth it, I find.

As for the maintainer of this repo, I'm sure they would have commented already if they were interested.

[uazo]

Thank you @iskunk! I confirm that your windows sdk generation tool works perfectly. Here is my version.

Just one thing; the line is incorrect: pushd %~dp0..\..\..\n --> pushd %~dp0..\\..\\..\n

[iskunk]

Great to hear, @uazo! The msvc-wine/vsdownload.py script is the centerpiece of a much nicer way of dealing with Windows builds.

Good catch on the missing backslashes. I've incorporated that fix, and inspired by your post, have been working to update my cross-build tooling to current versions. It's time to start preparing all this for a PR.

Note that as of three days ago, msvc-wine now produces a Windows Kits directory instead of kits, since Clang et al. expect the former.