Move the Windows "nightlies" over to this system?

[tkelman]

Also cc @ihnorton - something seems wrong or in need of manual intervention with the cross-compiled Windows binaries, whereas this new buildbot system looks like it's been building bottles and OSX nightlies pretty reliably. I was experimenting with making Docker images with all the cross-compiling dependencies recently, so I can help translate those into the format used here (mostly Vagrant?). Cross-compilation is doable from Ubuntu 14.04 if you use a PPA I made for the right configuration of MinGW cross-compiler, or from openSUSE, or possibly from Arch but I think the MinGW package there is already on 4.9 which might break binary compatibility with everyone's packages on Windows.

[ihnorton]

I think this might be running on the same machine I am using, but otherwise - sure, sounds good. I actually just reset things a couple hours ago on the build system - the linux julia build was broken so it couldn't do the uploads.

[tkelman]

For Ubuntu 14.04 in Docker, I was doing this:

apt-get update
apt-get install --no-install-recommends software-properties-common git make curl patch python g++ m4 p7zip-full -y
add-apt-repository ppa:ubuntu-wine/ppa -y
add-apt-repository ppa:tkelman/mingw-backport -y
dpkg --add-architecture i386
apt-get update
apt-get install --no-install-recommends wine1.7 g++-mingw-w64 gfortran-mingw-w64 -y

git clone git://github.com/JuliaLang/julia.git /home/user/julia
cd /home/user/julia
git submodule init && git submodule update
for XC_HOST in i686-w64-mingw32 x86_64-w64-mingw32; do \
      export XC_HOST; \
      echo 'LLVM_FLAGS = ac_cv_have_decl_strerror_s=no' > Make.user; \
      for i in openblas llvm fftw gmp mpfr suitesparse-wrapper arpack pcre; do \
        make -j4 -C deps install-$i || exit 1; \
      done; \
      $XC_HOST-ar cr usr/lib/libgtest.a; \
      $XC_HOST-ar cr usr/lib/libgtest_main.a; \
      tar -cJf $XC_HOST.tar.xz usr; \
      make distcleanall; \
      rm Make.user; \
      unset XC_HOST; \
    done

# to build:
echo 'VERBOSE = 1' > Make.user
echo 'override DEP_LIBS = uv openlibm openspecfun dsfmt Rmath mojibake' >> Make.user
export XC_HOST=i686-w64-mingw32 && tar -xJf $XC_HOST.tar.xz && make win-extras && make -j4 dist
# UPLOAD BINARY, TRIGGER TEST RUN, ETC
make distcleanall
rm -rf dist-extras Make.user

Although due to some strange Docker-Wine interaction, building the Julia sysimg would only work from a privileged interactive Docker session. Most VM-based systems seem fine though.

Equivalent setup steps for openSUSE 13.1:

zypper addrepo http://download.opensuse.org/repositories/windows:mingw:win32/openSUSE_13.1/windows:mingw:win32.repo
zypper addrepo http://download.opensuse.org/repositories/windows:mingw:win64/openSUSE_13.1/windows:mingw:win64.repo
zypper addrepo http://download.opensuse.org/repositories/Emulators/openSUSE_13.1/Emulators.repo
zypper --gpg-auto-import-keys refresh
zypper -n install mingw32-cross-gcc-c++ mingw32-cross-gcc-fortran
zypper -n install mingw64-cross-gcc-c++ mingw64-cross-gcc-fortran
zypper -n install mingw32-libstdc++6 mingw32-libgfortran3 mingw32-libssp0
zypper -n install mingw64-libstdc++6 mingw64-libgfortran3 mingw64-libssp0
# opensuse packages the mingw runtime dlls under sys-root/mingw/bin, not /usr/lib64/gcc
cp /usr/i686-w64-mingw32/sys-root/mingw/bin/*.dll /usr/lib*/gcc/i686-w64-mingw32/*/
cp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/*.dll /usr/lib*/gcc/x86_64-w64-mingw32/*/
zypper -n install --no-recommends git make tar wine which curl patch gcc-c++ m4 p7zip.i586

[staticfloat]

I'm all for this, and cross-compilation will be easy. Do we want to try native compilation, or is cross-compilation still the best way to go forward? I have scripts for creating boxes suitable for building Julia in the julia-vagrant repository (currently only Linux and OSX, I've been thinking of trying to use the Modern IE VMs Microsoft provides for windows), and I think the first way to move forward on this is to just generate a buildslave with these packages added and see how it goes with cross-compiling.

[tkelman]

Native compilation will be slower and a bit more difficult to script, but has the advantage of being able to run the tests. As long as failures get flagged somewhere, that would be huge. Does vagrant have tools for scripting setup of Windows VM's? Presumably in Powershell?

[staticfloat]

That's right. You can read about it here. Getting CentOS building RPMs is next on my list, but if you are impatient, feel free to experiment with the windows side of things. :)

[tkelman]

Yeah, googled for and read that right after posting. Looks like the vagrant package in Ubuntu 14.04 is a little too old, I'll have to install it from the deb instead.

[staticfloat]

Yes, vagrant and packer, which is what I use for building vagrant base boxes, both need to be installed from their binary packages, as they move more quickly than the distributions.

[tkelman]

Gotcha. I can figure out packer at some point. Amazingly https://vagrantcloud.com/ferventcoder/boxes/win7pro-x64-nocm-lite actually seems to work, with an up-to-date version of vagrant. Seems to need vagrant reload a bit more often than a Linux guest.

I would lean towards using Chocolatey to install Cygwin and build from there, technically also a cross-compile but can run the tests just fine. I don't see MSYS2 in Chocolatey and the MinGW setup process tends to be a few more manual steps with MSYS2 than it is using cyg-get to install them in Cygwin. I should really get around to documenting the Cygwin cross-compile method in README.windows.md already...

[tkelman]

Slowly figuring out how vagrant provisioning works. I think a powershell script should be good enough here, I can't figure out what we would need packer for if we already have access to a Windows Vagrant box that works. Although I'm not sure how the licenses work... https://github.com/ferventcoder/vagrant-windows-puppet

The single most important line in the Vagrantfile that makes shell provisioning work for Windows guests is config.vm.communicator = "winrm".

[tkelman]

Man, when I said "Native compilation will be slower," I didn't realize it would be continental-drift slow. It's taking about a day to build all the dependencies inside a Windows VM, and still going. I have a lot more sympathy now for Appveyor with all the performance and timing-out issues we had there. Windows on bare metal is already pretty slow, but inside a VM it's just awful.

[ihnorton]

How much control do you have over the VM - can you turn off the services I mentioned at the end of README.windows? Might help Building on my windows box became much saner after that.

On Sat, Sep 20, 2014 at 4:00 AM, Tony Kelman notifications@github.com wrote:

Man, when I said "Native compilation will be slower," I didn't realize it would be continental-drift slow. It's taking about a day to build all the dependencies inside a Windows VM, and still going. I have a lot more sympathy now for Appveyor with all the performance and timing-out issues we had there. Windows on bare metal is already pretty slow, but inside a VM it's just awful.

— Reply to this email directly or view it on GitHub https://github.com/staticfloat/julia-buildbot/issues/3#issuecomment-56260364 .

[tkelman]

Good question. If I can figure out how to do so via powershell, it's worth trying. This box is also apparently single core, which doesn't help. Not sure whether that's fixable without rolling a new vagrant box from scratch.

[staticfloat]

You can edit the vagrant config to set the number of cores and memory. On Sep 20, 2014 6:38 AM, "Tony Kelman" notifications@github.com wrote:

Good question. If I can figure out how to do so via powershell, it's worth trying. This box is also apparently single core, which doesn't help. Not sure whether that's fixable without rolling a new vagrant box from scratch.

— Reply to this email directly or view it on GitHub https://github.com/staticfloat/julia-buildbot/issues/3#issuecomment-56268129 .

[tkelman]

You can edit the vagrant config to set the number of cores and memory.

Oh. Well now I feel kinda dumb for letting it go for 20-odd hours. It even managed to crash vagrant at one point, possibly because the logs were too long? This is also running on an old machine that only has 2 cores. Anyway, it did finally finish building the dependencies, but as soon as it tries to start bootstrapping the sysimg, I get

socket: (10106) The requested service provider could not be loaded or initialized.

I'm not sure what that's about, but my guess is it has something to do with the "Unnecessary services are shut off." I'm trying netsh winsock reset catalog then restarting the VM, we'll see if that helps.

[tkelman]

@ferventcoder do you have any recommendations about enabling whichever service provider socket is complaining about in Windows-guest Vagrant boxes? I tried adding netsh ... to https://github.com/tkelman/julia-vagrant/blob/windows/support/choco_juliadev.ps1 but it didn't help.

[ferventcoder]

@tkelman you are using the windows 7 box? It is a lited box, which means YMMV. But it also means that some services are not just disabled, they've been ripped out of the ISO and subsequent install to keep the size down. I'd suggest looking at boxes over here: https://vagrantcloud.com/opentable - they have one or two images up :)

[ferventcoder]

I haven't run into issues with slowness. Here's what I do with VMs that keep them going really fast (at least from a Vagrant point of view): https://github.com/ferventcoder/vagrant-windows-puppet/blob/master/boxes/VagrantWindows.rb#L9-L12

[ferventcoder]

One thing about slowness. Do not use host shared folders if you are looking for some speed (especially with lots of reads/writes, like compiling would do). Prefer to clone things to local folders with a small depth (as in git clone --depth 10).

[tkelman]

@ferventcoder thanks so much for the input! I'll see if your pointers help get things running any better.

[tkelman]

I was able to get the opentable win2012r2 box to run (needed some manual password policy tweaking to work properly), but it's having issues with network connectivity. It can download and install chocolatey fine via (iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))), but was having trouble with choco install cygwin or cyg-get make. The box already has Cygwin on it, but not in the location that Chocolatey now installs it. choco install -force cygwin didn't work either, but maybe due to problems with ftp connectivity.

[tkelman]

It seems like we can close this as soon as the builders run the tests. Jameson just fixed the only failing test on master, and I have an idea that might fix the failing test on release-0.3.

[staticfloat]

Looks like Jameson's fixes went through. I ran the tests on my VMWare windows box on my MBP, and everything checked out, so it looks like things might be good here.

The last step before I close this is to automatically sign the executables. I did that manually for the 0.3.3 release, I'll write up a little script to do it automatically from here on out.

[staticfloat]

Automatic signing looks like it's working now. Yipee!

[tkelman]

Are the buildbots running the tests yet? If no, can they?

[staticfloat]

I've re-enabled testing in https://github.com/staticfloat/julia-buildbot/commit/9a99204f7556042fccbffb0d502b3b6b16383c56

[tkelman]

Apparently that didn't work so well, which makes me sad.

 /home/vagrant/buildbot/slave/package_win8_1-x86/build/usr/bin/julia.exe --check-bounds=yes -f ./runtests.jl all
    From worker 3:       * linalg2
    From worker 2:       * linalg1
    SUCCESSERROR: write: invalid argument (EINVAL)
 in write at stream.jl:735
 in print at string.jl:4
 in println at string.jl:5
 in println at string.jl:8
 in anonymous at C:\cygwin\home\vagrant\buildbot\slave\package_win8_1-x86\build\test\runtests.jl:57
 in cd_3B_4119 at 
 in include_3B_2225 at 
 in include_from_node1 at loading.jl:128
 in process_options_3B_4042 at 
 in _start_3B_4032 at  (repeats 2 times)
while loading C:\cygwin\home\vagrant\buildbot\slave\package_win8_1-x86\build\test\runtests.jl, in expression starting on line 42
fatal: error thrown and no exception handler available.
Base.UVError(prefix="write", code=-4071)
Makefile:9: recipe for target 'all' failed
make[1]: Leaving directory '/home/vagrant/buildbot/slave/package_win8_1-x86/build/test'
make[1]: *** [all] Error 1
Makefile:430: recipe for target 'testall' failed
make: *** [testall] Error 2

I am going to have to start using the term SUCCESSERROR now though.

[staticfloat]

Hah. Well, at least it's a new one. I'm thinking this might be a buildbot-related thing. The tests, when run manually via Base.runtests() worked fine on both buildbots over SSH.