Windows version availability

mmd-osm commented 7 years ago

Recently a number of people reported issues on GIS SE to get osmium-tool up and running, mostly because it appears to be too difficult for them. As a windows version is created as part of appveyor scripts anyway I wonder if that binary file result could be published somewhere.

joto commented 7 years ago

I personally don't know anything about Windows. So I have no idea whether the binaries we build on appveyor would run anywhere else than there, if we need several versions for different Windows versions and how to make those available in a way that Windows users could install them easily. If somebody wants to tackle this, please go ahead.

mmd-osm commented 7 years ago

Thanks for your feedback. Maybe @BergWerkGIS has some ideas on this, as he's provided the Windows AppVeyor script originally.

wilhelmberg commented 7 years ago

The AppVeyor build script could be changed to collect all necessary files and push them to an artifact. So each successful build would have a binary package attached, that would show up under the Artifacts tab on AppVeyor: https://ci.appveyor.com/project/Mapbox/osmium-tool/build/1.0.281/job/yui64kpudl67t9ub/artifacts

@joto @mmd-osm how does that sound?

joto commented 7 years ago

@BergWerkGIS is this something that would work out-of-the-box on any Windows system? We would also need some documentation how to find and use it. We can't just link to an always changing appveyor link. We could download those artifacts for every release and put them on osmcode.org or so. But that should be automated somehow, I don't need that extra work for each release. And I can't check whether whatever is built there actually works and is complete etc. I can integrate this on osmcode.org, but somebody needs to write the scripts and instructions.

wilhelmberg commented 7 years ago

is this something that would work out-of-the-box on any Windows system?

Yes and no 😏 We are using Visual Studio 2015 thus the Microsoft Visual C++ 2015 Redistributable is required.

On most up to date systems this should already be installed as a lot of programs also install it as a dependency.

We can't just link to an always changing appveyor link.

This link always shows the latest build: https://ci.appveyor.com/project/Mapbox/osmium-tool We would have to add to the readme how to click on a job and then on the Artifacts tab. But that might prove difficult if the latest build wasn't successful: we would have to add intructions to click on History find the last ✅ build.

We could download those artifacts for every release and put them on osmcode.org or so. But that should be automated somehow, I don't need that extra work for each release. And I can't check whether whatever is built there actually works and is complete etc. I can integrate this on osmcode.org, but somebody needs to write the scripts and instructions.

@joto we could use FTP to automatically publish to osmcode.org after a successful build, if that were ok for you. The credentials could be added via the project settings on AppVeyor or into the appveyor.yml via encrypted environment variables (https://www.appveyor.com/docs/build-configuration/#secure-variables).

If we choose to go that route: REMINDER! remove SET from the build script as it shows all environment variables in raw format (even the previously encrypted ones): https://ci.appveyor.com/project/Mapbox/osmium-tool/build/1.0.281/job/va8388o8synhjii8#L7

lonvia commented 7 years ago

Appveyor has stable links to artifacts via its API. Examples from osm2pgsql:

https://ci.appveyor.com/api/projects/openstreetmap/osm2pgsql/artifacts/osm2pgsql_Release.zip (latest build) https://ci.appveyor.com/api/projects/openstreetmap/osm2pgsql/artifacts/osm2pgsql_Release.zip?tag=0.92.0 (version 0.92.0)

In osm2pgsql we simply pack all necessary dlls with the artifact. That seems to work okay, at least there have been no complaints.

wilhelmberg commented 7 years ago

Thanks @lonvia, that's good news - I wasn't aware about those stable links.

joto commented 7 years ago

We definitely only want to link to builds of released codes, master might compile or not, but in any case it could be broken.

FTPing to osmcode.org doesn't work. osmcode.org is a jekyll setup published through github.io mechanism. But I have to do a bunch of things anyway for releases, so this could be added somehow. I have to think about this.

Okay, so if I understand this correctly we have this workflow:

[ ] every appveyor build creates some artifacts
[ ] whenever i do a release i need to link to the artifact created for that release somehow. That link is stable, except for the release version number which I know, so that's easy to do
[ ] we need some kind of documentation that tells a windows user how to turn the artifact file into something that runs.

wilhelmberg commented 7 years ago

every appveyor build creates some artifacts

Only successful builds create artifacts.

we need some kind of documentation that tells a windows user how to turn the artifact file into something that runs.

The arctifact will be a ZIP file: so, unzip and run :smirk: Maybe we need to document how to set some environment variables or put everything in the root of the ZIP (don't remember from the top of my head what is needed to run).

wilhelmberg commented 7 years ago

Took a quick stab at this and created a branch that (after successful build and tests)

copies osmium.exe and all DLLs of lib-osmium deps into a directory (sorry, didn't have time to investigate which DLLs are needed and which not)
runs osmium.exe from there
packages that directory into a ZIP
pushes that ZIP to AppVeyor artifacts, eg.:
- Dev: https://ci.appveyor.com/project/Mapbox/osmium-tool/build/1.0.284/job/m9ye1tolkglwqmas/artifacts
- RelWithDebInfo: https://ci.appveyor.com/project/Mapbox/osmium-tool/build/1.0.284/job/2yg5eio0hhxt3ux7/artifacts

As stated above by @lonvia it's possible to have a permalink to the artifact of the latest successful buid: https://www.appveyor.com/docs/packaging-artifacts/#permalink-to-the-last-successful-build-artifact

mmd-osm commented 7 years ago

I just ran a few tests using RelWithDebInfo on Windows 10! After unzipping the archive to an empty directory, I checked a couple of osmium commands on a .pbf extract, like converting pbf to opl, xml, check-refs,...

This all looked good so far, no error messages about missing DLLs.

joto commented 7 years ago

Okay, so there is a ZIP file with an osmium.exe and a bunch of DLLs. How does Windows know where those DLLs are when you call the exe? Do they have to be in the same directory as the exe file? In the current directory?

Do we actually need those DLLs? I am pretty sure that most of them aren't needed, zlibwapi.dll and libexpat(w).dll probably are, but the others should not be needed. We should only include the DLLs we need and we need to tell the user what they should do with them. We need a README file anyway with, at least, license information.

mmd-osm commented 7 years ago

libexpat.dll and zlibwapi.dll are both required here. Just extract all files to same directory, and then call osmium.exe, maybe put that directory in your PATH, if you don't like providing the full path all the time. That's basically all I had to do.

joto commented 7 years ago

Okay, we should only put those DLLs into the ZIP file than. @mmd-osm Can you write up a README that explains to the newbie how to use this that we can then also put into the ZIP file?

mmd-osm commented 7 years ago

I'm really not that talented when it comes to writing end user documentation, so please bear with me.

First draft:

Osmium Tool Installation on Windows

To install Osmium Tool manually, do the following:

When upgrading from a previous osmium tool version, all existing files belonging to the previous Osmium Tool installation need to be removed/replaced.
Make sure you are logged in as user with administrative privileges. Otherwise you may only be able to extract Osmium Tool to your own user directory.
Choose an installation location, such as C:\osmium. If this location does not yet exist, create it first in Windows Explorer.
Extract the installation archive to the installation location
Add the installation location to your local path settings - or - call Osmium Tool by using the full path: C:\osmium\osmium

Nacktiv commented 7 years ago

I tried that in Windows 7. When calling osmium.exe I get error: Der Prozedureinsprungpunkt "CreateFile2" wurde in der DLL KERNEL32.dll nicht gefunden. which means something like "entry point CreateFile2 not found in KERNEL32.dll"

In osm2pgsql we simply pack all necessary dlls with the artifact. That seems to work okay, at least there have been no complaints.

The osm2pgsql artifact from appveyor works for me

ontheebrink commented 7 years ago

@Wolfgang16 I'm having the same issue. Did you install osm2pgsql? I'm at my wits end here trying to get this thing to work.

mmd-osm commented 7 years ago

@ontheebrink : what's your environment like? Also running Windows 7!?

ontheebrink commented 7 years ago

@mmd-osm Yes, windows 7.

Nacktiv commented 7 years ago

ontheebrink: I have got osm2pgsql from the links lonvia posted above. Installing is not necessary, simply unzip it into a folder and run it in a console. You might need Microsoft Visual C++ 2015 Redistributable.

mmd-osm commented 6 years ago

Someone reported a similar issue in osm2pgsql and tracked it back to a zlib regression & wrong zlib version used: https://github.com/openstreetmap/osm2pgsql/issues/812#issue-288278912

According to appveyor logs, we have the same issue here as well:

Found ZLIB: C:\projects\osmium-tool\libosmium-deps\zlib\lib\zlibwapi.lib (found version "1.2.8") -- Looking for pthread.h

mmd-osm commented 6 years ago

While the zlib issue seems to be resolved by using a more recent version, I don't seem to find any artifacts for Windows anymore: https://ci.appveyor.com/project/Mapbox/osmium-tool/build/job/qnnovk4mnoi1whdj/artifacts

I think an after-build and artifacts section are currently missing in appveyor.yml. I'm referencing the corresponding parts for osm2pgsql here, as their build still seems to work.

https://github.com/openstreetmap/osm2pgsql/blob/master/appveyor.yml#L83-L95 https://github.com/openstreetmap/osm2pgsql/blob/master/appveyor.yml#L112-L113

OndrejSpanel commented 6 years ago

I confirm I was able to download Windows artifacts from https://ci.appveyor.com/project/Mapbox/osmium-tool/build/1.0.390 and the tool is running fine on my Win10 x64 machime.

Thanks a lot for this, it makes installing on Windows much much easier.

I also confirm I cannot see the the artifacts in the recent versions.

trumpeta commented 6 years ago

I can't download from appveyor.com Can I get a mirror please

mmd-osm commented 6 years ago

Close in favor of #105.

SchmollTroll commented 4 years ago

I confirm I was able to download Windows artifacts from https://ci.appveyor.com/project/Mapbox/osmium-tool/build/1.0.390 and the tool is running fine on my Win10 x64 machime.

Hi there, stumbled over this thread by looking for a Win version of Osmium.

I looked at the history and for the past four month all builds fail. I understand that data must be deleted after a time but wouldn't it be great to keep the last succeeded build?

OndrejSpanel commented 4 years ago

For the record, I gave up on OSM processing using Windows versions, Linux seems to be much better supported. I have switched my OSM processing to use Windows Subsystem for Linux, Ubuntu 18.04 LTS, and I do not regret.

wimpie3 commented 3 years ago

Windows builds aren't available anymore on appveyor, can someone take a look at this?

IldarKhayrutdinov commented 2 years ago

Artifacts are now completely unavailable on appveyor server, even for fresh builds.

ImreSamu commented 2 years ago

@IldarKhayrutdinov : please test the Conda-forge distribution (linux-64, osx-64, win-64) https://anaconda.org/conda-forge/osmium-tool To install this package with conda run: conda install -c conda-forge osmium-tool

tp-duarte commented 2 years ago

@ImreSamu could you tell me if the conda-forge distribution install all the dependencies needed for osmium ?

I'm trying to use time-filter function and isn't working, although the help command of the package works just fine.

IldarKhayrutdinov commented 2 years ago

@ImreSamu @tp-duarte I manual download from https://anaconda.org/conda-forge/osmium-tool/files but trying to run shows an error message about not founded files: libexpat.dll, boost_program_options.dll, zlib.dll, LIBBZ2.dll.

tp-duarte commented 2 years ago

@IldarKhayrutdinov , I think this might be the problem then. Try to download the dependencies listed in the error in your machine to see what happens. The links of the packages are available in the readme of osmium-tool.

I'm also going to try to do this in this week and update my results in the issue here.

ImreSamu commented 2 years ago

@tp-duarte:

could you tell me if the conda-forge distribution install all the dependencies needed for osmium ?

I only tested it on linux ( with conda install -c conda-forge osmium-tool ) and it seems to have installed the necessary dependencies. see my minimal test log: https://github.com/osmcode/osmcode.github.io/pull/11#issue-1235957982

if the default and recommended conda install -c conda-forge osmium-tool is not working; please report to -> https://github.com/conda-forge/osmium-tool-feedstock ( with the full debug info , steps, logs, etc ... )

@IldarKhayrutdinov

I manual download from https://anaconda.org/conda-forge/osmium-tool/files

IMHO: this is not the recommended way!

my suggestion: 1.) install conda on windows: https://conda.io/projects/conda/en/latest/user-guide/install/windows.html 2.) run the following: conda install -c conda-forge osmium-tool

tp-duarte commented 2 years ago

Ohh @ImreSamu, in linux I only tried install osmium with sudo apt get.

Will try with conda and report my results here. If there is something wrong I'll also report in the link you sended here.

tp-duarte commented 2 years ago

@ImreSamu the installation with conda-forge in linux happened ! but I'm still unable to use time-filter correctly. Will report that in another issue, thanks ! 😉

IldarKhayrutdinov commented 1 year ago

@ImreSamu @tp-duarte I manual download from https://anaconda.org/conda-forge/osmium-tool/files but trying to run shows an error message about not founded files: libexpat.dll, boost_program_options.dll, zlib.dll, LIBBZ2.dll.

@ImreSamu @tp-duarte Late reply. Yes, of course, these errors mean that there are missing dependencies. I just wanted to do without additional tools (conda). But I was confused by the fact that the older version of osmium that I had at that time (as suggesed @OndrejSpanel https://github.com/osmcode/osmium-tool/issues/59#issuecomment-390255364) had one single .exe file (without additional .dll files) and worked. Or maybe I'm confusing something...

I think there were dependencies then too, maybe they was linked into single .exe file. I don’t know, I already lost this old version, on artifacts it’s also removed.

Be that as it may, I decided to compile the utility myself. The latest builds, like in conda, has dependencies: boost_program_options-vc143-mt-x64-1_81.dll, bz2.dll, libexpat.dll, lz4.dll, zlib1.dll.

UPD Earlier uses static libs and linking into single .exe: https://ci.appveyor.com/project/Mapbox/osmium-tool/build/1.0.390/job/skx61j610842y80u#L440

osmcode / osmium-tool

Windows version availability #59

Osmium Tool Installation on Windows