Windows build - Githubissues

ocefpaf commented 8 years ago

We should build should build nco from source on Windows.

See http://nco.sourceforge.net/nco_qt_msvc.shtml

ocefpaf commented 8 years ago

Ping @czender. If you have any hints for us here it would really help us get the Windows builds going.

czender commented 8 years ago

the expert on windows builds of NCO is @pedro-vicente

ocefpaf commented 8 years ago

Obrigado Pedro! I'll take give it a try. We have almost all of those in conda-forge. I need to work on antlr though. Em 13/06/2016 23:06, "Pedro Vicente" notifications@github.com escreveu:

@ocefpaf https://github.com/ocefpaf

Building NCO from source is a time consuming task that for a first timer could take several frustrating days to finish. On the NCO main page you will see a list of library dependencies

http://nco.sourceforge.net/

on the link you posted above there are several environment variables listed

LIB_NETCDF LIB_HDF5 LIB_HDF5_HL LIB_ZLIB LIB_SZIP LIB_ANTLR LIB_GSL LIB_CURL LIB_UDUNITS LIB_EXPAT

Additionnaly, the following environment variable must also be defined for the location of the netCDF, UDUnits, GSL and Antlr header files: HEADER_NETCDF HEADER_UDUNITS HEADER_GSL HEADER_ANTLR

By their name it's obvious to which library they refer. So what you have to do is:

1) Obtain the source code for all the libraries above. (ANTLR must be version 2.7) LIB_UDUNITS LIB_EXPAT these 2 it's better to send an email to Ward Fisher on the netcdf mailing list because I think the netcdf group ported a semi-official port of these for Windows (that by now might be official)

2) Build the libraries, and define the above environment variables

Boa sorte !

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/conda-forge/nco-feedstock/issues/6#issuecomment-225760882, or mute the thread https://github.com/notifications/unsubscribe/AA6BL3SeUf5OC7kd4VWgXT1_LjiXIIgKks5qLgy2gaJpZM4H5hSE .

rsignell-usgs commented 8 years ago

@ocefpaf, seems like recent progress allowing antlr window builds https://github.com/antlr/antlr4/pull/1268

ocefpaf commented 8 years ago

@ocefpaf, seems like recent progress allowing antlr window builds https://github.com/antlr/antlr4/pull/1268

See my comment in https://github.com/conda-forge/pynco-feedstock/issues/2#issuecomment-251936898

I never got time to look at the Windows builds but it should be easier now that conda brings msys2 in the default channel.

czender commented 8 years ago

that's good news but beware that NCO requires antlr version 2.7 not antlr version 4.x. not sure whether the antlr4 package will cut the mustard for NCO.

kwilcox commented 8 years ago

@ocefpaf I have a windows laptop and an afternoon... if you want to give me a 10 minute brief I can hack away at it.

We can start with a non-ncap2 build for Windows and make incremental improvements as needed if its blocked upstream somewhere.

ocefpaf commented 8 years ago

@kwilcox I am a little bit swamped but you can start by setting your laptop up with conda and a MSVC.

https://github.com/conda/conda-recipes/wiki/Building-from-Source-on-Windows-32-bit-and-64-bit

Once that is done we need to check if we have all the dependencies already in conda and compiled on Windows.

[X] LIB_NETCDF, HEADER_NETCDF
[X] LIB_HDF5
[X] LIB_HDF5_HL # ~~not sure~~ I cofirmed and it is present in conda-forge's packages
[X] LIB_ZLIB
[ ] LIB_SZIP # absent but I believe it is not mandatory (alternative to zlib)
[ ] LIB_ANTLR, HEADER_ANTLR # absent
[ ] LIB_GSL, HEADER_GSL # absent but I believe it is not mandatory (ncap2?)
[X] LIB_CURL
[X] LIB_UDUNITS, HEADER_UDUNITS
[X] LIB_EXPAT

czender commented 8 years ago

you are correct that antlr and gsl are only required for ncap2, and that szip is not mandatory (NCO does not require it, though netCDF4 may require it depending on how it is built). so building NCO with --disable-ncap2 might be the easiest way to start.

rsignell-usgs commented 8 years ago

@pedro-vicente, any chance you could take a stab at getting the settings right for a conda build on windows without ncap2?

I might be able to help you address conda issues...

ocefpaf commented 8 years ago

Based on http://nco.sourceforge.net/nco_qt_msvc.shtml we need Qt's qmake. @ccordoba12 do you know if that is possible with anaconda's Qt? Any experience with qmake?

@czender would it be complicated to switch to cmake?

@pedro-vicente if you want to try this please feel free to bug me with any conda/conda-build question.

czender commented 8 years ago

@ocefpaf A switch from autotools to cmake might be conceptually simple, yet I've never written cmake build scripts and my time is limited. If someone else provided a working cmake build system then I would be happy to include it in NCO with the intent of maintaining both cmake and autotools build systems so long as each has significant usage.

ccordoba12 commented 8 years ago

qmake comes with all our Qt builds :-) I don't have much experience with it though.

ocefpaf commented 8 years ago

qmake comes with all our Qt builds :-) I don't have much experience with it though.

Thanks @ccordoba12!

I never worked with conda, and my time is limited to learn yet another build system. But if you post here or email me some instructions of where to start, I can give it a try, but no promises about time.

I started a PR to show you what we can do so far. See https://github.com/conda-forge/nco-feedstock/pull/22

Conda is not that complicated as it is a layer on top of other build systems.

On my post above from June 13, there are the steps for building with the supplied NCO Qt build: what you have to do is, with Visual Studio, build the NCO dependencies in some way.

We have "cmake" builds for all but ANTLR and GSL.

https://github.com/conda-forge/hdf5-feedstock/blob/master/recipe/bld.bat https://github.com/conda-forge/libnetcdf-feedstock/blob/master/recipe/bld.bat https://github.com/conda-forge/zlib-feedstock/blob/master/recipe/bld.bat https://github.com/conda-forge/curl-feedstock/blob/master/recipe/bld.bat https://github.com/conda-forge/udunits2-feedstock/blob/master/recipe/bld.bat https://github.com/conda-forge/expat-feedstock/blob/master/recipe/bld.bat

not sure how well those will play with nco but we can make adjustments to those recipes if necessary.

ocefpaf commented 8 years ago

Do the above bld.bat files get the libraries from some "official" location (like github) or do you host them somewhere?

All official sources. In some rare cases we patch them to enable the build.

I believe there is no "official" cmake script for Windows for ANTLR 2.7 and GSL that you could do the same.

That is in the background for now. We are focused on getting a build without GSL and ANTLR.

I have Visual Studio 2013 solutions to build ANTLR 2.7 and GSL, but I don't have any way to make these available, like a public machine for download.

You can send it via e-mail or upload right here on the GitHub issues. (Drag-and-drop should work for most file formats.) BTW people discussing the GSL Windows build at https://github.com/conda-forge/gsl-feedstock/issues/7 will be quite happy with that :smile:

pedro-vicente commented 8 years ago

@ocefpaf @czender I volunteer to write a CMake build.

czender commented 8 years ago

Much appreciated, @pedro-vicente

ocefpaf commented 8 years ago

I volunteer to write a CMake build.

That is awesome @pedro-vicente! Please feel free to bug me for any help you need.

pedro-vicente commented 7 years ago

The Cmake build is only for after NCO 4.6.2

ocefpaf commented 7 years ago

The Cmake build is only for after NCO 4.6.2

Was that released? If not we can open the PR with a pre-release just to get the ball rolling.

pedro-vicente commented 7 years ago

NCO release dates are here http://nco.sourceforge.net/

czender commented 7 years ago

Latest version is 4.6.2-beta03. Final 4.6.2 probably this coming week.

pedro-vicente commented 7 years ago

I pushed to github a prototype for a CMake build. The file /doc/index.shtml contains build instructions, which are

CMake build. To build with CMake, do:

mkdir build
cd build
cmake ..
make

The CMake script tries to find header and dependency libraries on standard system locations. If these are not found, they can be set with (the locations in full path are examples)

cmake .. -DNETCDF_INCLUDE:PATH=/my/netcdf/include/path -DNETCDF_LIBRARY=/my/netcdf/library/file -DHDF5_LIBRARY=/my/hdf5/library/file -DHDF5_HL_LIBRARY=/my/hdf5 high level/library/file -DSZIP_LIBRARY=/my/szip/library/file -DZLIB_LIBRARY=/my/zlib/library/file -DCURL_LIBRARY=/my/curl/library/file

Right now only the minimal dependencies are detected: netcdf header file, netcdf, HDF5, HDF5_HL, zlib, szip, curl libraries

czender commented 7 years ago

Where does this issue stand? @pedro-vicente have you produced a working feedstock that builds a conda-installable NCO (including basic ncap2) on Windows?

pedro-vicente commented 7 years ago

@czender The cmake build status is the same as the above post. Binaries are built for these NCO operators with just the minimum dependencies (netCDF, HDF5, szip, zlib) ncks ncbo ncecat ncflint ncpdq ncra ncrename

Regarding the conda build, can anyone please remind me what is needed? Is it a NCO cmake build, including library dependencies? Like I said before, I can spend some time on the NCO cmake build but I don't have time to learn conda.

czender commented 7 years ago

That's a great start. I see the CMakelists.txt file in the top level directory. I think @ocefpaf or someone else familiar with the conda Windows environment ( @rsignell-usgs ? ) needs to give that a try and work it from the feedstock end, and tell us what mods they need in the upstream. @pedro-vicente please install conda on your windows machine so you can iterate/verify that the install works.

ocefpaf commented 7 years ago

That's a great start. I see the CMakelists.txt file in the top level directory.

Sorry. I am a bit lost. You mean a custom CMakelists.txt file somewhere or are you referring to the one shipped with the source code? (Sorry I never looked into that part of the code so maybe this is something obvious.)

I think @ocefpaf or someone else familiar with the conda Windows environment ( @rsignell-usgs ? ) needs to give that a try and work it from the feedstock end, and tell us what mods they need in the upstream.

I am in the processing of moving to another city and I won't be able to try that for at least 2 weeks (sorry).

@pedro-vicente please install conda on your windows machine so you can iterate/verify that the install works.

@pedro-vicente conda is not that complicated. I will get a simple PR that tries to build this so we can iterate together.

pedro-vicente commented 7 years ago

@ocefpaf I you do git pull of NCO, there is a CMakelists.txt file in the top level directory. Do mkdir build cd build cmake ..

If the dependencies header files/libraries are not found they can be with these options

for example cmake .. -DNETCDF_INCLUDE:PATH=/mypath -DNETCDF_LIBRARY=/mypath/netcdf.lib

NETCDF_INCLUDE NETCDF_LIBRARY HDF5_LIBRARY HDF5_LIBRARY HDF5_HL_LIBRARY ZLIB_LIBRARY SZIP_LIBRARY CURL_LIBRARY

ocefpaf commented 7 years ago

conda-forge does not package SZIP because of its license. How can I bypass that in the cmake call?

CMake Error at CMakeLists.txt:62 (message):
  szip library not found

czender commented 7 years ago

FYI it is possible (and quite common) to build netCDF without SZIP, so that NCO will never need to link to SZIP. It should be possible to either eliminate all references to SZIP from CMakelists.txt, or to ignore SZIP when the package is not found.

ocefpaf commented 7 years ago

it is possible (and quite common) to build netCDF without SZIP, so that NCO will never need to link to SZIP.

Our netCDF is not built with szip and cmake is still trying to find it.

It should be possible to either eliminate all references to SZIP from CMakelists.txt, or to ignore SZIP when the package is not found.

I see that in sometimes SZIP=0 does the trick, would that work with nco? I want to avoid patching CMakelists.txt for off/on flags like that.

czender commented 7 years ago

Yes. The conda-forge Windows build is the raison d'etre for the CMakelists.txt. So please just submit any desired changes you need as pull-requests and @pedro-vicente or I will quickly merge them. After conda-forge Windows works, we can go back and improve CMakelists.txt for non-conda situations.

ocefpaf commented 7 years ago

The conda-forge Windows build is the raison d'etre for the CMakelists.txt. So please just submit any desired changes you need as pull-requests and @pedro-vicente or I will quickly merge them. After conda-forge Windows works, we can go back and improve CMakelists.txt for non-conda situations.

Got it. I though it was already "generic." I will play with it here then and submit any patches if I later.

monocongo commented 7 years ago

I have run cmake on Windows, below is the output. Are there next steps?

"C:\User Programs\CLion\CLion 2016.3.2\bin\cmake\bin\cmake.exe" -DCMAKE_BUILD_TYPE=Debug -G "CodeBlocks - MinGW Makefiles" C:\home\git\nco -- Found netcdf.h header file at: C:/home/Anaconda3/Library/include -- Found netcdf library at: C:/home/Anaconda3/Library/lib/netcdf.lib -- Found hdf5 library at: C:/Anaconda3/Library/lib/hdf5.lib -- Found hdf5 high level library at: C:/Anaconda3/Library/lib/hdf5_hl.lib -- Found zlib library at: C:/Anaconda3/Library/lib/z.lib -- Found curl library at: C:/Anaconda3/Library/lib/libcurl.lib -- Configuring done -- Generating done -- Build files have been written to: C:/home/git/nco/cmake-build-debug

pedro-vicente commented 7 years ago

@ocefpaf

szip library not found

To avoid this error , you can, for now

1) get SZIP , build, install, from here

https://github.com/erdc-cm/szip

2) comment in your local copy of CMakelists.txt the lines regarding SZIP detection

Because NCO has many possible combinations of library dependencies, I wanted to keep the dependencies as much simple as possible, so for now I assumed all base libraries are required. Ideally cmake would detect if SZIP is needed by inspecting HDF5 (it's HDF5 that uses SZIP, not netCDF).

for now I am going to change the SZIP detection to a warning in the script. Of course if SZIP is required, then the build will fail. So my preferred solution would be just to install SZIP, even if it's not needed.

I would like to be the only maintainer of the cmake script, so if you have any change to propose, post here, or email me directly

pedro-vicente commented 7 years ago

@monocongo ok, the generation of the build files succeeded. Now, only thing to do is build :-)

-G "CodeBlocks - MinGW Makefiles"

I never used CodeBlocks, but this should have created a CodeBlocks project or a file called "Makefile". If you have a "Makefile" just do

make

most people use Visual Studio

cmake -G "Visual Studio 14" ..

this creates a Visual Studio solution file and projects

ocefpaf commented 7 years ago

To avoid this error , you can, for now

get SZIP , build, install, from here

Like I mentioned in https://github.com/conda-forge/nco-feedstock/issues/6#issuecomment-278359639 SZIP is a no-no in conda-forge because it does not allowed for binary re-distribution. However, I did check the page to see if that changed in the past years and I found this statement in http://www.compressconsult.com/szip/:

The sourcecode for szip is here. There is a general public license for distribution and usage of szip included in the package, so no need to worry.

But I downloaded the source and could not find a license file stating that :unamused: so that is still a no-no.

comment in your local copy of CMakelists.txt the lines regarding SZIP detection

Did that in https://github.com/ocefpaf/nco-feedstock/commit/e76fa6e94896cde3a754b1b697b8c43d3767a93a with no avail.

See the full log in: https://ci.appveyor.com/project/ocefpaf/nco-feedstock/build/1.0.27/job/0x1ddiggwwky1j6d

I would like to be the only maintainer of the cmake script, so if you have any change to propose, post here, or email me directly

I prefer open development on GitHub with PRs and reviews instead of message posts and e-mails. Once we achieve a successful build here I can send a PR to nco's repository with a configuration for continuous integration with AppVeyor to test the Windows builds at every new PR.

pedro-vicente commented 7 years ago

The CMake build was officially released. Notes here http://nco.sourceforge.net/#cmake

czender commented 7 years ago

@ocefpaf @rsignell-usgs Thanks to @pedro-vicente, NCO 4.6.9, just released contains a functional CMake build that works on Windows, and szip can be omitted to avoid license issues. Build can be as simple as

cd nco/cmake
cmake .. -DFIND_SZIP=OFF -DCMAKE_INSTALL_PREFIX=/usr/local

or with fully specified paths to some libs, e.g.,

cmake .. -DFIND_SZIP=OFF -DCMAKE_INSTALL_PREFIX=${HOME} -DNETCDF_INCLUDE:PATH=${NETCDF_INC} -DNETCDF_LIBRARY:FILE=${NETCDF_LIB}/libnetcdf.a -DHDF5_LIBRARY:FILE=${NETCDF_LIB}/libhdf5.a -DHDF5_HL_LIBRARY:FILE=${NETCDF_LIB}/libhdf5.a

or see example build.bat for a Window-ish environment. Hope someone volunteers to create a new PR based on CMake for the conda feedstock for Windows.

pedro-vicente commented 7 years ago

@czender little correction :-) The SZIP and ZLIB detection are now automatic, so -DFIND_SZIP=OFF was removed.

the script CMakeLists.txt includes comments on how this is done. NOTE, in particular for Windows

#HDF5 can be optionally linked with the SZIP library (Science Data Lossless Compression Program) and ZLIB
#Symbol to detect in ZLIB can be only H5Z_DEFLATE, a structure only defined in H5Zdeflate.c if the filter is enabled
#For SZIP the structure can be only H5Z_SZIP, defined in H5Zszip.c if the filter is enabled
#check_library_exists() tries to link a temporary program with these symbols
#On MSVC for this detection to work, the HDF5 library MUST HAVE as additional dependencies the ZLIB and SZIP libraries,
#which is not a requirement for the library to build successfully

so, what this means is that by checking the symbols in the libraries (on MSVC) is not error proof, because it depends on the way the HDF5 library was built on Visual Studio, an option that can be turned off/on.

Another way would be to get this information (need to link with SZIP) from a header file, or a --config provided by the HDF5 distribution files. But it seems there is not an official standard way to get this information, so the feature is still waiting for user feedback, complaints.

rsignell-usgs commented 7 years ago

There is also this drop in replacement for SZIP

https://www.dkrz.de/redmine/projects/aec/wiki/SZIP

pedro-vicente commented 7 years ago

Just to add to the SZIP/ZLIB issue. This is an example output one gets after doing

cmake ..

regarding a needed ZLIB

-- Detecting if HDF5 library /usr/local/lib/libhdf5.so needs the ZLIB library...
-- Looking for H5Z_DEFLATE in /usr/local/lib/libhdf5.so
-- Looking for H5Z_DEFLATE in /usr/local/lib/libhdf5.so - found
-- ZLIB library is needed...

and here and example for SZIP detection

-- Detecting if HDF5 library /usr/local/lib/libhdf5.so needs the SZIP library...
-- Looking for H5Z_SZIP in /usr/local/lib/libhdf5.so
-- Looking for H5Z_SZIP in /usr/local/lib/libhdf5.so - not found
-- SZIP library is not needed...

For this example, Cmake tries to 1) find a ZLIB library, if not found exit with fatal error; if found, add it (e.g libz.a) to the list of libraries to link 2) for the SZIP case, nothing is done

The inclusion of SZIP/ZLIB is pretty much an individual choice for who builds the HDF5 library (e.g linux packages distributions) Downstream users, starting with netCDF and ending with NCO have no control over this.

So, the option -DFIND_SZIP=<ON/OFF> made no sense, it was just a temporary manual fix because there was no automatic detection.

ocefpaf commented 7 years ago

See https://github.com/conda-forge/nco-feedstock/pull/43#issuecomment-331497257 for the latest attempt. Seems to get quite far but it is failing with:

[ 96%] Linking C executable ncra.exe
failed to create symbolic link 'ncea': No error

pedro-vicente commented 7 years ago

@ocefpaf

I am not sure why that error happens (maybe because it says "No error" :-))

but I did some changes to that part do a new pull from git hub and try again

It seems the error is on the build part

on the configure part, this should print

cmake ..

-- Installing links ncea;nces;ncrcat;ncdiff;ncunpack to: /home/pvicente/nco_install/bin

and on the build

make install

[ 71%] Linking C executable ncra
Making symbolic link /home/pvicente/nco_install/bin/ncra -> ncea

pedro-vicente commented 7 years ago

ok I see

[ 96%] Linking C executable ncra.exe
failed to create symbolic link 'ncea': No error

this was in Windows. This fails in Windows because Cmake's create_symlink is not for Windows. Some days ago I added a

if (UNIX)

test so that that error does not happen in Windows. It seems you are still using an old version before that change

ocefpaf commented 7 years ago

I am using the latest release. I guess the changes you are referring to did not make into the release then.

On Sep 22, 2017 2:27 PM, "Pedro Vicente" notifications@github.com wrote:

ok I see

[ 96%] Linking C executable ncra.exe failed to create symbolic link 'ncea': No error

this was in Windows. This fails in Windows because Cmake's create_symlink is not for Windows. Some days ago I added a

if (UNIX)

test so that that error does not happen in Windows. It seems you are still using an old version before that change

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/conda-forge/nco-feedstock/issues/6#issuecomment-331509811, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6BL2UTjALAVPviOA59uqdlU4KVWujVks5sk-3jgaJpZM4H5hSE .

czender commented 7 years ago

@pedro-vicente I am preparing to release 4.7.0. Any testing you can do to ensure that CMake build (still) works with Windows would be appreciated. Hopefully everything is set so that the conda feedstock will "just work" with Windows as of 4.7.0.

pedro-vicente commented 7 years ago

@czender CMake/NCO builds for 4.7.0

xylar commented 7 years ago

I have a PR #47 in for updating to 4.7.0. Are there additional changes needed to add windows support to the recipe? Maybe that can be a follow-up PR, if so.

ocefpaf commented 7 years ago

I have a PR with the RC that has the changes. I'll submit it tomorrow. It is "kind of" passing, the build completes but a ncks -m $url test fails :unamused:

conda-forge / nco-feedstock

Windows build #6