CMake scripts and Windows compatibilty

[alex85k]

I am currently working on Windows compatibility of Osmium in my fork. The goal is to support MinGW 32/64 and MSVC. (Certain features can be disabled by #ifdefs before someone rewrite them in native code).

Currently I have CMake build scripts and slightly patched code working at least for Mingw32 and Ubuntu, planning also to support Visual Studio. Only disabled features for now is execute() method in osmfile.hpp and mmap_anon storage.

CMake file support installation and automatic testing, missig features for now are valgrind and cppcheck.

Can you please give me some advice on how to make the code suitable for future inclusion? https://github.com/alex85k/osmium/tree/mingw

For example, CMake generates the makefiles that overwrite existing ones. They are not to be committed (we want to keep old building system too).

[joto]

I am looking through your changes, here are some comments:

• You have some changes from shared_ptr to boost::shared_ptr. Those should not be necessary, see include/osmium/smart_ptr.hpp. I have added this file to make switching from boost smart ptrs to C++11 smart ptrs easier.
• Whats the difference between WIN32 and _MSC_VER macros?
• simple_map.hpp contains too many things just copy-and-pasted from sparse_table.cpp
• sparse_table.cpp vs. sparse_map.cpp?
• I don't like the inclusion of the stuff in include/osmium/windows. Is this really necessary? Can't we find some other way? For the timing stuff: We probably don't need the usec resolution anyway, so maybe we can find an easier interface with second resolution. inttypes.h should really be available in the system.
• I don't like the CMake files. Don't you have a normal "make"? Keep in mind that Osmium is a header only library, all the make files are for is to show you how the examples can be compiled. Everybody who uses Osmium in their own applications can choose whatever build system they want. I do not want to impose a complex build system on everybody when make is enough to show how things fit together.

[alex85k]

Thank you very much for your time and review!

I'll try to answer the questions here.

• On Windows you can use different compilers with their own set of standard libraries. The are no standard places for libraries, includes, etc. Visual Studio compiler and Intel compiler are the most advanced, but their set of libraries are the most specific (native for Windows), though the standard libraries are implemented.
• MinGW compiler have the includes most similar to Unix, except for glibc functions and Unix system calls (have unistd, inttypes, getopt but does not have sys/mman , sys/times, sys/wait, netinet/in, strptime , timegm. WIN32 macros is defined.
• _MSC_VER macros is set when the source is compiled using Visual Studio compiler. It has less Unix functions than MinGW (only those that are in standards, even not all of them) no unistd, gettimeofday, getopt, round.
• I'll try to avoid adding boost::shared_ptr (was problem on Visual Studio)
• simple_map.hpp and sparse_map.cpp are very rough (I wanted to run tagstats on low-memory machine). simple_map does not need sparsehash library but can consume too much memory for PBF>200 Mb. Sparse_map was added by intuition. I guess it should take less memory than sparse_table but more than simple_map and work little slower than sparse_table. I'll fix the comments and copypasted functions, maybe provide some benchmark to prove they are useful (or not).
• About include/osmium/windows : all files are found by Googling for windows incompatibility issues. systime and inttypes stubs can be eliminated by adding more #ifdefs or changes to the code. Microsoft library have stdint.h, so this should not be very hard.
• mmap.hpp is needed to re-implement Unix system api by WinApi . We can place it somewhere else but without this (or analogous) code there is no chance to get mmap_file working.
• on Visual Studio there is still unsolved problem with getopt - 4 executables use it but I have found no headrer-only C++ implementation, so these examples does not compile.

[alex85k]

Complex build system for header-only library is not actually needed. The reason to use it was the same as makefiles - to provide examples for cmake-based projects that use Osmium and automate compilation of examples and test on system that does not have Unix-compatible make and libraries in standard paths.

• I have committed Cmake-generated Makefile in examples by mistake
• We should keep the makefiles as the main demonstration system they are simpler, but it would be very useful to place CMake scripts somewhere (many of the libraries supports both CMake and autotools or Cmake/makefiles - see libgd, libpng, expat, geos) when they are ready.
• CMake scripts for examples are useful to run them or create new project that uses the multitude of libraries that Osmium needs. It asks user "give me Boost", "give me expat", "where is libgd" if there is no standard place to find them (like on Windows).
• CMake can provide convenient installation and packaging alternative to makefiles. No more install ... .root ... and hardcoded path. CPack allows to easily create distribution (ZIP, TGZ) for download.
• Testing is rather good with CTest.You can choose which tests to run etc. (try ctest in source folder after building).
• There are no standard makefiles on Windows, mingw makefile are Unix-incompatible, MSys or Cygwin are little hard to install, Microsoft nmake is a different thing too...
• One of the main reasons that CMake seems to be industrial standard for cross-platform C++ libraries is that it does not need any "standard" tools and libraries and can create both console makefiles and IDE projects. Many of the Windows developers do not use the console at all and we want osmium to be popular :)

[alex85k]

Here is the bechmark of storages for small file RU-BA.osm.pbf (24Mb), example osmium_mpdump.exe (changed typedef storage_sparsetable_t):

SparseTable 117s 830-915Mb Vector 112s 100Mb SparseMap 126s 114Mb SimpleMap 115s 247Mb

I ignored the Vector class when adding my classes, because I did not remember that the input was sorted by ID. Maybe map storages are not needed after all.

[alex85k]

The same file on Ubuntu (same machine) is processed 3-4 times faster: SparseTable 31s 670 Mb Vector 28s 115 Mb SparseMap 40.5s 112 Mb SimpleMap 30 s 250 Mb (MinGW is slow or maybe some library is built without optimization. Visual Studio variant runs as fast as Linux or faster)

[alex85k]

Visual Studio comaptible build is ready. I have removed redundant inttypes.h and boost:: and placed getopt into separate library on Windows. (Leaving gettimeofday and mmap in separate osmium/windows folder for now).

I am not sure if simple_table and simple_map sta are needed in master branch - Vector seems to be faster (no files with unsorted IDs?).

CMake files can be generated in separate folder (if user chooses it), so original makefiles remain untouched as "standard" build system.

Can you have a look at the code once again?

[joto]

I have taken out some non-controversial bits and pieces from your changes (some of them as is, some slightly changed) and committed them in 8f77d4e. But that were the easy parts...

The rest of the stuff is somewhat difficult to handle, one reason is that there are several issues mixed up: cmake, general windows support, storage variantes, ...

Can you move all the cmake stuff into it's own directory? I can't test and maintain it, so until I am sure that somebody is actually maintaining this in the long term, I'd rather not have it mixed up with everything else.

I'll have a look at the gettimeofday stuff and see whether we actually need that.

Can we pull in the mmap stuff from some other project as a sub-module or something?

[joto]

I have added some changes in the last commits that should fix some more of the problems. Can you try that and then rebase your changes to the current version to see what else needs to be done.

The progress handler is not important enough to warrant lots of special windows stuff. We should just document it as unix-only. This way we don't have to do the gettimeofdays stuff.

[alex85k]

Ok, tomorrow I'll check and rebase the changes, separating compatibility from all other additions.

[alex85k]

Thank you! I have checked the changes and merged them to my mingw branch (do not know exactly how to use rebase and if it is suitable).

Currently, no examples compile on MinGW because of osmfile.hpp issue - sys/wait (fork, pipe, pid_t, ,,, are not supported on Windows except maybe Cygwin emulation).

If it is patched ( https://dl.dropboxusercontent.com/u/63393258/mingw_min.patch ) then all tests and most of examples can be compiled and they pass OK.

To compile all examples we need one of following:

• add some h-file alternative to sys/mmap.h (maybe you can find the better one)
• drop MMap storage support on Windows (not very good),
• or put header (and maybe library) to separate repository (like I did with wingetopt, but in this case the header is sufficient and short).

We also need to solve problem with progress.hpp (maybe implement it in some cross-platform way?) or drop its support on Windows. Maybe using simple clock() is better (with ifdefs, if needed)?

To allow compilation of headers in VS in other projects, we need some extra patching:

• patch position.hpp , xml.hpp, smatr_ptr.hpp
• patch mmap_file.hpp, progress.hpp according to chosen solution of mmap and progress problems

CMake is needed to generate and run Visual Studio projects for examples and tests and is optional (I am improving and separting it to always keep existing makefiles untouched). Makefile theoretically can be used under Mingw+Msys, but it is very hard to make working system with gdal-config etc. (simpler to patch makefile by hand, hardcoding libraries and include dirs).

[alex85k]

I have updated my sources and made them windows-compatible again. Here are the changes we need to make the most headers windows-compatible: https://github.com/alex85k/osmium/commit/e486c4988791b0447565f40a73490f55a7986b5e

And here are the cmake scripts (building with main makefile overwriting is prohibited now): https://github.com/alex85k/osmium/commit/7fda597dba7eb25c9539a6fcddbbe7cceca12de1

[joto]

I am closing this issue. This will not be fixed in osmium any more, but the newer libosmium works fine on Windows.