Compiling CefSharp failed with Spotify automated builds

[TKGNET]

First of all, cudos for a great idea: I went through compiling CEF Source manually the last week, and it took me a long time to get the environment and the GN Defines right. Mainly setting up VS 2017 with the right SDK and Toolchain took a long time.

Today I tried to compile CefSharp v67.0.0 against precompiled standard builds of CEF 3.3396 and run into an error: Error report:

Running: docker build --memory=30g -f Dockerfile_cef_create_from_binaries -t cef . in G:\CEFBinaries
10/29/2018 21:42:39 CEF Build took 1.6 secs
Running: docker build --memory=30g --build-arg BINARY_EXT="zip" -f Dockerfile_cef_binary -t cef_binary . in G:\CefSharpDockerfiles
Process docker build --memory=30g --build-arg BINARY_EXT="zip" -f Dockerfile_cef_binary -t cef_binary . exited with non zero code: 1 aborting!
At G:\CefSharpDockerfiles\functions.ps1:40 char:9
+         throw "Process $proc $opts exited with non zero code: $($p.Ex ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Process docker ...de: 1 aborting!:String) [], RuntimeException
    + FullyQualifiedErrorId : Process docker build --memory=30g --build-arg BINARY_EXT="zip" -f Dockerfile_cef_binary -t cef_binary . exited with non zero code: 1 abort 
   ing!

Versions.ps1:

$VAR_CHROME_BRANCH="3396";
$VAR_CEFSHARP_VERSION="67.0.90";
$VAR_CEFSHARP_BRANCH="cefsharp/67";
$VAR_BASE_DOCKER_FILE="microsoft/dotnet-framework:4.7.2-runtime-windowsservercore-1803"; #microsoft/dotnet-framework:4.7.1-windowsservercore-1709
$VAR_DUAL_BUILD="0"; #set to 1 to build x86 and x64 together, mainly to speed up linking which is single threaded, note may need excess ram.
$VAR_GN_DEFINES="";
$VAR_GYP_DEFINES="";
$VAR_CEF_BUILD_ONLY=$false;#Only build CEF do not build cefsharp or the cef-binary.
$VAR_CEF_USE_BINARY_PATH=".\..\CEFBinaries\"; #If you want to use existing CEF binaries point this to a local folder where the cef_binary*.zip files are. It will skip the long CEF build step then but still must make the VS container for the cefsharp building.  Note will copy a dockerfile into this folder.
$VAR_CEF_BINARY_EXT="zip"; #Do not change this off of Zip unless you are supplying your own binaries using $VAR_CEF_USE_BINARY_PATH above, and they have a different extension
$VAR_CEF_VERSION_STR="auto"; #can set to "3.3239.1723" or similar if you have multiple binaries that Docker_cefsharp might find
$VAR_HYPERV_MEMORY_ADD="--memory=30g"; #only matters if using HyperV, Note your swap file alone must be this big or able to grow to be this big, 30G is fairly safe for single build will need 60G for dual build.
if ($false){ #Sample 65 overrides
    $VAR_CHROME_BRANCH="3325";
    $VAR_CEFSHARP_VERSION="65.0.90";
    $VAR_CEFSHARP_BRANCH="master";

Docker settings:

{
  "registry-mirrors": [],
  "insecure-registries": [],
  "debug": true,
  "experimental": false,
  "data-root": "g:/docker",
  "storage-opts": [
    "size=250G"
  ]
}

Spotify standard build CEF Binaries in the G:\CEFBinaries folder :

cef_binary_3.3396.1785.ga27bbfa_windows64.zip
cef_binary_3.3396.1786.gd3e36d0_windows32.zip

I'd be grateful for suggestions what went wrong Pete

[mitchcapper]

Thanks pete, I will give it a shot shortly and let you know. Are you using the remote powershell? I would have expected more output on the docker build command otherwise. Nice detailed bug report.

[TKGNET]

Its the first time I use docker, so I have to dig into debugging first. And, no I'm not using remote powershell - yet ;) The docker report might be short, cause I started build.ps1 several times experimenting with other formats than ZIP. I'm not sure I used the right binaries though, but according to CefSharp Branch 67 it should be CEF 3.3396. My manually compiled version of 3.3396 is working flawlessly with CefSharp 67 Feel free to ask for changes and/or additional input

[mitchcapper]

OK I think I solve the issue, although some notes:

It should not matter which binaries you use (as you do not need to match the CefSharp version exactly so using newer is also fine). I noticed you have .zip files however I cannot seem to download any zip files from spotify. Did you just rename the .tar.bz2 files to .zip? As that would create an issue.

I downloaded the two files: http://opensource.spotify.com/cefbuilds/cef_binary_3.3396.1785.ga27bbfa_windows64.tar.bz2 http://opensource.spotify.com/cefbuilds/cef_binary_3.3396.1785.ga27bbfa_windows32.tar.bz2

then used the following version.ps1 file:

$VAR_CEFSHARP_VERSION="67.0.90";
$VAR_CEFSHARP_BRANCH="cefsharp/67";
$VAR_BASE_DOCKER_FILE="microsoft/dotnet-framework:4.7.2-runtime-windowsservercore-1803"; #microsoft/dotnet-framework:4.7.1-windowsservercore-1709
$VAR_DUAL_BUILD="0"; #set to 1 to build x86 and x64 together, mainly to speed up linking which is single threaded, note may need excess ram.
$VAR_GN_DEFINES="";
$VAR_GYP_DEFINES="";
$VAR_CEF_BUILD_ONLY=$false;#Only build CEF do not build cefsharp or the cef-binary.
$VAR_CEF_USE_BINARY_PATH=".\..\CEFBinaries\"; #If you want to use existing CEF binaries point this to a local folder where the cef_binary*.zip files are. It will skip the long CEF build step then but still must make the VS container for the cefsharp building.  Note will copy a dockerfile into this folder.
$VAR_CEF_BINARY_EXT="tar.bz2"; #Do not change this off of Zip unless you are supplying your own binaries using $VAR_CEF_USE_BINARY_PATH above, and they have a different extension
$VAR_CEF_VERSION_STR="auto"; #can set to "3.3239.1723" or similar if you have multiple binaries that Docker_cefsharp might find
$VAR_HYPERV_MEMORY_ADD="--memory=30g"; #only matters if using HyperV, Note your swap file alone must be this big or able to grow to be this big, 30G is fairly safe for single build will need 60G for dual build.

The ticket did help me find another bug however using using alternate extensions. If you use the above file make sure you sync with the repo first to support the .tar.bz2 for binary only builds.

I was mentioning the lack of output as its very clean between your running commands above there is no other output. It may be due to different windows versions outputting different things by default.

With the above (and latest repo changes) it would then build 100% without an issue for me.

If it still does not work try running the last docker command that fails by hand (just copy and paste what it tells you it is running into the powershell window). Report back with the output and I can help further. I will close this for now but if still having an issue just re-open.

[TKGNET]

I thought I'd report back on the (now closed) issue: Before tar.bz2 was not recognised , thats why I had to decompress the binaries and compress them back into zip files. Stack trace now is also much more descriptive.

My orginal problem was as nearly always sitting in front of the screen, shame on me. Accidentally I copied x32 and x64 binaries from the same branch, but from diferrent releases:

cef_binary_3.3396.1785.ga27bbfa_windows64.zip
cef_binary_3.3396.1786.gd3e36d0_windows32.zip

This caused a copy error in Cefsharp\build.ps1, as it should.

Nevertheless it helped me a lot to get into debugging docker, and I must admit I'm a big fan of it now.

Again: thanks for your help and your package, most of the work compiling Cef/CefSharp was setting the environment up exactly right. This is now much more painless ;)

Will try to compile Cef binaries with proprietary support next (have done it manually already) , and if this works I'll try the same using Azure.

Regard, Pete

[mitchcapper]

@TKGNET thanks for reporting back, I don't think you should run into any issue with the alternate build flags:)