Windows build

[peterwittek]

...and matching Travis check.

[jul1278]

Hi all, I've spoken with Peter over email and I'm going to be having a go at this. Welcome any advice/suggestions/etc on how I should proceed, otherwise I'll probably start looking into python library build.

[xgdgsc]

I haven' t seen any julia project that has windows travis running. I think you should first read the Bindeps Doc and try to build at your windows machine.

[jul1278]

So just a quick update, I've managed to build a dll with the mapDistanceFunctions code and successfully call getWeight(...) from the julia shell.

Also as far as I can tell, travis doesn't support windows yet.

[peterwittek]

Wow, that was amazingly fast! Can you run the test?

Both of you are right, Travis does not support Windows. If you can run the test successfully on Windows, I would be happy to merge your contribution.

[jul1278]

how does this look? for some reason my PyPlot got messed up and it took me ages to fix it, finally got it working I think.

[peterwittek]

Perfect' This looks exactly what it should look like. Could you send a pull request?

[jul1278]

So the test passes and everything works fine when I build the dll in g++ . I'm trying to set up at vs build config to build a dll but when I ccall the vs built .dll it fails with a "Could not load library '.../somoclu.dll' error", not quite sure what to do about this but I'll keep at it.

Other question, how do you tell the Somoclu.jl where to look for the library? atm I'm just putting a hardcoded path to my dll in the ccall method

[peterwittek]

The file deps/build.jl has to be modified to let BinDeps compile the library and it will also put it at the right place (which is deps/usr/lib independent of the platform) when you call Pkg.build("Somoclu").

[jul1278]

Haven't had a chance to work on this, (busy week), I think what I should be doing is modifying the Makefile to build a dll on windows, rather than worrying about the visual studio project. That way the user just has to call Pkg.build(...) etc.

As far as I can tell I need autotools to run configure and make on the makefile? I can't build anything running make in the src directory. Is this correct?

I think I've managed to get autotools with MinGW but it seems to be just a bunch of scripts

[peterwittek]

I am looking at other wrappers in the package index how they do it. I think you are right, MinGW is the way to go. For instance, take a look at CDDLib.

[jul1278]

Should I be able to run make in the src directory though? I get all sorts or errors "no rule to make..." "no target specified.." etc when I call make. I've tried make in the cygwin shell as well.

[peterwittek]

When you issue Pkg.build("Somoclu"), it downloads and extracts the source dependency in deps/src/somoclu-1.7.2. If you want to take over manually, you have to run ./configure in this folder, followed by make.

[jul1278]

Hey guys, I'm back from being sick/christmas holidays, hopefully get this issue closed soonish...

I'm getting this error trying to Pkg.add("Somoclu") ?

ERROR: unknown package Somoclu
 in macro expansion at .\pkg\entry.jl:53 [inlined]
 in (::Base.Pkg.Entry.##2#5{String,Base.Pkg.Types.VersionSet})() at .\task.jl:360
 in sync_end() at .\task.jl:311
 in macro expansion at .\task.jl:327 [inlined]
 in add(::String, ::Base.Pkg.Types.VersionSet) at .\pkg\entry.jl:51
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}})() at .\pkg\dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}}, ::String) at .\file.jl:48
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at .\pkg\dir.jl:31
 in add(::String) at .\pkg\pkg.jl:100

[peterwittek]

This is because Somoclu.jl is not yet merged with the Julia package repository. Installation has to be done as written in the Readme. Our first milestone is to bring this package to a level where we have a chance to be included in the official repository. Part of that is resolving this very issue, so your contribution is highly valuable.

[jul1278]

Alright, sorry It's taking a while, I realise now I'm a bit out of my depth here. Here are the tasks as I see them:

• modify somoclu source code to expose functions to window .dll Basically done, I can compile the somoclu.dll in gcc with a simple command line and successfully run the tests from julia

These are the tasks I'm having trouble with:

• Add a configuration to the makefile which actually builds the windows DLL I thought the best place to start would be to try to run the autogen.sh and configure which requires a sort of unix environment (MinGW/Cygwin). I think I've set these up incorrectly though as I'm getting a variety of errors when I try to run autogen.sh

$ autogen.sh
+ test -d Tools/config
+ aclocal -I Tools/config
      0 [main] perl 9568 fork_copy: linked dll data/bss pass 0 failed, 0x72E000..0x74E794, done 0, windows pid 4116, Win32 error 487
      0 [main] perl 7896 fork_copy: linked dll data/bss pass 0 failed, 0xE9E000..0xEBE794, done 0, windows pid 4156, Win32 error 487

I'm looking into this ^^ now.

• modify Build.jl to add a build process for windows

[peterwittek]

I think this is a good plan. I have zero insight on what the problem could be with aclocal. Sorry about that.

[jul1278]

Alright I'm not entirely sure how I did it but I eventually got my environment set up properly, so the next step is to add a configuration to the makefile to build the dll.

[jul1278]

Ok, I've managed to get the makefile to build the dll which I can successfully run the example in the readme

[peterwittek]

This is with Somoclu proper, right? Could you send a pull request there?

[blegat]

The easiest way to get windows build working is to use opensuse build service. For instance, for cddlib, I am using this spec file. The dll are cross-compiled for Windows on an opensuse server automatically and you can use them using WinRPM. The fist step would maybe be to make a working build script for Linux (see here for many examples: https://build.opensuse.org/project/show/science) and then adapt it to Windows (typically by adding _mingw64_ at the beginning of each command).

[jul1278]

Why would that be better given that we already have a working build for linux and 90% of a working build for windows?

Peter I'll see if I can submit a PR for somoclu proper tonight

[peterwittek]

Thanks.

[blegat]

@jul1278 I think it is better in the long term since it is built automatically and you do not need to do a manual build for each new version and that is the way Tony Kelman suggests to do it. However, if you are almost done, it is probably not worth the effort. For CDDLib I got help for Jan Engelhardt, otherwise, I agree that adding the support in the opensuse build system might take a rather long time.

[jul1278]

Peter I've forked somoclu and pushed my changes for the windows build, just need to clone it onto my macbook and check I haven't broken the unix build, hopefully get a chance to do that tomorrow.

[peterwittek]

I took a quick glimpse and it fails on Linux. I commented on your commit to highlight why.

[jul1278]

Ok, on my end it's building on mac and windows

[peterwittek]

I tried and it builds fine on Linux too in every possible combination (GCC, MPI, CUDA). Please send a pull request.

[peterwittek]

Travis builds the Linux and macOS versions without problems. Once we confirm that the new Windows build procedure works fine with the Python flavour, we can release a new version of Somoclu and close this. Many thanks!

[peterwittek]

In the meantime, could you check if the Windows version builds as it should? Delete Somoclu from Julia, and try to install it from scratch:

Pkg.clone("https://github.com/peterwittek/Somoclu.jl")
Pkg.build("Somoclu")
using Somoclu

Thanks.

[xgdgsc]

I checked with above commands and the example from readme.md. I got error:

julia> train!(som, data);
ERROR: UndefVarError: libsomoclu not defined
 in #train!#3(::Int64, ::Int64, ::Int64, ::String, ::Float64, ::Float64, ::String, ::Function, ::Somoclu.Som, ::Array{Float32,2}) at C:\Users\gsc\.julia\v0.5\Somoclu\src\Somoclu.jl:154
 in train!(::Somoclu.Som, ::Array{Float32,2}) at C:\Users\gsc\.julia\v0.5\Somoclu\src\Somoclu.jl:98

[jul1278]

I'll hopefully get around to the Julia side in the next few days

[jul1278]

So I've tried modifying the Build.jl so when on windows we pass in :Windows instead of :Unix or whatever. which fails to build.

I suspect the problem might be because when I build libsomoclu from the command line I actually have to make the Tools\config folder and then call autoconf -v not entirely sure what that command does but it seems to work, so I'm thinking I'd need to add those commands to autogen.sh and have them called on windows os, if possible.

Is there some way I can test Build.jl with local sources? I see there's an issue on the bindeps repo but it doesn't look like any progress has been made on it.

[peterwittek]

This is really weird. I have no idea why you would need an extra call to autoconf on Windows.

If you copy the modified source files in deps/downloads/somoclu-1.7.3-pre.tar.gz, it will not download it from GitHub, but use the file there when you call Pkg.build('Somoclu').

[jul1278]

so you mean .julia/v0.5/Somoclu/deps/downloads/ ?

should the link in build.jl remain the same?

[peterwittek]

Yes, that is the folder. You can leave the link in build.jl as it is.

[jul1278]

So when I extract somoclu-1.7.3-pre.tar.gz it doesn't contain autogen.sh is that correct?

[peterwittek]

It should. Run make dist in the Somoclu folder to generate it the correct zip. Then extract it, rename the extracted folder somoclu-1.7.3 to somoclu-1.7.3-pre, and zip it again to get the correct file.

[jul1278]

my download of https://github.com/peterwittek/somoclu/releases/download/1.7.2/somoclu-1.7.3-pre.tar.gz does not contain autogen.sh

[peterwittek]

That is right, I think it should not contain autogen.sh, only the generated files. This is a source distribution and not a zip of the git repo.

[jul1278]

so what commands should I use to build?

[peterwittek]

I lost you a bit. If you put this zip file to the folder mentioned above, Pkg.build('Somoclu') extracts it and compiles it.

[jul1278]

Yep, so I'm trying to debug why Pkg.build() fails.

I'm assuming Pkg.build() calls configure and make etc?

[peterwittek]

Yes. This is what is specified in line 8 of deps/build.jl.

[jul1278]

So what I want to do is extract the .zip file and run the commands that julia will run to build to see where it fails, but if there is no autogen.sh then I'm not sure how I can do that.

[peterwittek]

Could you tell me the exact steps you are doing? I don't see how autogen.sh plays any role in this. On Linux, I would do something like this:

1. Delete the existing installation:

   rm -fr ~/.julia/v0.5/Somoclu

2. Clone it again from Julia:

   Pkg.clone('https://github.com/peterwittek/Somoclu.jl')

3. Make a source distribution from the modified Somoclu (this is to be done in the respective folder).

   make dist

4. Unzip the generate file, rename the folder to somoclu-1.7.3-pre, and tar-zip it again.
5. Copy this file to the Julia folder:

   cp somoclu-1.7.3-pre.tar.gz ~/.julia/v0.5/Somoclu/deps/downloads

6. Build it in Julia. This will invoke the configure script, but not autogen.sh.

   Pkg.build('Somoclu')

   Then every time you make a change in Somoclu, repeat the process from step 3.

[jul1278]

Ok, I think I've found the problem, the src/Windows folder is missing from the zip? I'm guessing make dist doesn't include it for some reason

[jul1278]

I followed those instructions and put the src/Windows folder inside the zip... still no luck.

Is there something I can do to the make file (maybe print to the Julia terminal or write a text file somewhere) just so I can be certain that julia is actually trying to build the somoclu inside deps/downloads/?

[jul1278]

here's the error

[peterwittek]

I don't know. I just tried the above instructions and they work just fine, apart from replacing single quotes by double quotes. I also tried what the error message suggests (BinDeps.debug("Somoclu")), but it was not particularly edifying.

[jul1278]

Ok, Well I'm not quite sure how to proceed from here.

If I just add a mkdir C:\make_test in the makefile inside the .tar.gz, I'll be able to confirm that it is in face the makefile being used?