Add an executable for windows in the bin directory

[bchapuis]

The current distribution contains a bash script. The equivalent for windows systems should be created. Probably something like a baremaps.bat file containing:

@echo off
set dir=%~dp0%
set LIB="%dir%..\lib\*"
java -cp "$DIR/../lib/*" com.baremaps.cli.Baremaps %*

[veikkoeeva]

I tried and received Error: Could not find or load main class com.baremaps.cli.Baremaps. Indeed, I don't see a jar file with that name (I'm not familiar with Java but assume a jar file with name should be found). I double-check later it's not something about uncompressing the release.

[bchapuis]

The LIB variable was not set correctly, sorry. Could you try with the following snippet. I don't have a windows machine at hand, so I hope it helps.

@echo off
set dir=%~dp0%
set LIB="%dir%..\lib\*"
java -cp %LIB% com.baremaps.cli.Baremaps %*

[veikkoeeva]

I get

Unable to get Charset 'cp65001' for property 'sun.stdout.encoding', using default windows-1252 and continuing. Usage: baremaps [COMMAND] A toolkit for producing vector tiles. Commands: execute Execute queries in the database. import Import OpenStreetMap data in the database. update Update OpenStreetMap data in the database. export Export vector tiles from the database. serve Serve vector tiles from the the database.

This is a different problem. Using chcp tells Active code page: 65001, which is UTF-8 but I take this means things work.

I think the next step is to download an osm file and see if it can be processed to Mapbox vector files. It was my initial idea. As an aside, I was also thinking if OsmSharp could be used to load things into PostgreSQL. Not important here otherwise than that to note that if the PostgreSQL files would be as files (and loaded and executed dynamically" the core contribution of this project would be the SQL queries while allowing potential modularity on loading the system and maybe preprocessing the data. :)

It's almost midnight here and I was erecting this so a bit of delays here. :)

[bchapuis]

It's almost midnight here and I was erecting this so a bit of delays here. :)

Nice week-end project ;)

I published a new release (0.3.3) that includes a preliminary bat file.

Regarding the importation process, there are some simple examples in the current repo and a more complexe stylesheet in the following repository.

[veikkoeeva]

@bchapuis About hundred people getting sore muscles. :)

I thank you for this, check examples and see how things go when I try this. Hopefully Windows users find this more broadly. :)

[veikkoeeva]

I add a bit to this thread, I have a running PostgreSQL and I added database baremaps, user baremaps with password baremaps with the rights to login and create objects to the database.

When I try to run .\baremaps.bat execute --database='jdbc:postgresql://127.0.0.1:5432/baremaps?&user=baremaps&password=baremaps' --file='res://osm_create_extensions.sql' --file='res://osm_drop_tables.sql' --file='res://osm_create_tables.sql' --file='res://osm_create_gist_indexes.sql'--file='res://osm_create_gin_indexes.sql' (from OpenStreetMap example) I get

.\baremaps.bat : Missing required option: '--file=FILE'
At line:1 char:1
+ .\baremaps.bat execute --database='jdbc:postgresql://127.0.0.1:5432/b ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Missing required option: '--file=FILE':String) [], RemoteExcepti 
   on
    + FullyQualifiedErrorId : NativeCommandError

Usage: baremaps execute [--enable-s3] --database=DATABASE
                        [--log-level=LOG_LEVEL] --file=FILE [--file=FILE]...
Execute queries in the database.
      --database=DATABASE     The JDBC url of the database.
      --enable-s3             Enable Amazon S3 integration.
      --file=FILE             The SQL file to execute in the database.
      --log-level=LOG_LEVEL   The log level.
'user' is not recognized as an internal or external command,
operable program or batch file.
'password' is not recognized as an internal or external command,
operable program or batch file.

This is likely caused by running a bat file that takes quoted parameters (otherwise puzzling). Maybe useful to record here these next steps even if I haven't gone more into details about this.

[bchapuis]

Thank you for reporting this. Have you tried to remove the "'"? I definitely need to setup a windows machine and perform more tests to see how the tool behave in a prompt.

[veikkoeeva]

@bchapuis I tried several variations. I look deeper into this when I get a bit of time cleared up. I want to be a bit more systematic. I did try that and a few other variations quickly and didn't have success (in a hurry, so I may have just erred). I think also one thing I should do is to move everything to lib and use java from the plain command line with only the lib envronment variable pointing to that folder (either as a path, dot or whatever I come up with it). This way I can establish a maybe functioning baseline so I know piping quotes between processes isn't at fault. I may also read some code, maybe I don't install Java tooling to actually debug (but we'll see).

[veikkoeeva]

A quick update, running

java -cp "C:\baremaps\lib\*" com.baremaps.cli.Baremaps execute --database 'jdbc:postgresql://localhost:5432/baremaps?&user=baremaps&password=baremaps' --file 'res://osm_create_extensions.sql' --file 'res://osm_drop_tables.sql' --file 'res://osm_create_tables.sql' --file 'res://osm_create_gist_indexes.sql' --file 'res://osm_create_gin_indexes.sql'

will result in

java : java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: permission denied to create 
extension "postgis"
At line:11 char:1
+ java -cp "C:\\baremaps\\lib\\*" com.baremaps.cli.Baremaps execute --d ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (java.lang.Runti...nsion "postgis":String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

  Hint: Must be superuser to create this extension.
    at com.baremaps.cli.Execute.lambda$call$0(Execute.java:59)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)
    at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
    at java.util.concurrent.CountedCompleter.exec(Unknown Source)
    at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.util.concurrent.ForkJoinPool.helpComplete(Unknown Source)
    at java.util.concurrent.ForkJoinPool.externalHelpComplete(Unknown Source)
    at java.util.concurrent.ForkJoinTask.externalAwaitDone(Unknown Source)
    at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
    at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
    at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.util.stream.ReferencePipeline.forEach(Unknown Source)
    at java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
    at com.baremaps.cli.Execute.call(Execute.java:54)
    at com.baremaps.cli.Execute.call(Execute.java:22)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
    at picocli.CommandLine.access$1200(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
    at picocli.CommandLine.execute(CommandLine.java:2058)
    at com.baremaps.cli.Baremaps.main(Baremaps.java:43)
Caused by: org.postgresql.util.PSQLException: ERROR: permission denied to create extension "postgis"
  Hint: Must be superuser to create this extension.
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
    at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:310)
    at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:296)
    at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:273)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:268)
    at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
    at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
    at com.baremaps.cli.Execute.lambda$call$0(Execute.java:57)
    ... 26 more

Instead of giving baremaps user super user priviledges, I installed to the baremaps database the following extensions using pgAdmin GUI in browser

• postgis
• postgis_topology

I do not know if other PostGIS extensions are needed. Probably wouldn't hurt and depends on operations.

In any event, then re-runing the batch file results in...

java -cp "C:\\baremaps\\lib\\*" com.baremaps.cli.Baremaps execute --database 'jdbc:postgresql://localhost:5432/baremaps?&user=baremaps&password=baremaps' --file 'res://osm_create_extensions.sql' --file 'res://osm_drop_tables.sql' --file 'res://osm_create_tables.sql' --file 'res://osm_create_gist_indexes.sql' --file 'res://osm_create_gin_indexes.sql'
[INFO ] 2020-12-03 22:46:59.814 [main] Execute - 16 processors available
[INFO ] 2020-12-03 22:46:59.837 [main] Execute - res://osm_create_extensions.sql
[INFO ] 2020-12-03 22:47:00.044 [main] Execute - res://osm_drop_tables.sql
[INFO ] 2020-12-03 22:47:00.060 [main] Execute - res://osm_create_tables.sql
[INFO ] 2020-12-03 22:47:00.097 [main] Execute - res://osm_create_gist_indexes.sql
[INFO ] 2020-12-03 22:47:00.106 [main] Execute - res://osm_create_gin_indexes.sql

All looks good thus far. Next steps:

(<Edit: The path to lib in the above also works with single slashes.)

1. Download an .osm map. Likely a small one like Luxembourg.
2. Import this file to the database.
3. Export it out as vector tiles. Preferably in format so it can be uploaded to cloud as files that can be used directly without a server.

Hopefully this week. I think this establishes that there is something iffy with the batch file. For instance, I'm not sure if that %* expands all variables, i.e. starting at position 0 or what's the problem here.

[bchapuis]

Thanks a lot for digging into this. I wasn't aware that installing the extensions through the sql script would require superuser privileges on windows.

Regarding the upload of the vector tiles there is an option to deploy them on an s3 bucket. This requires to install and configure aws-cli and to configure it with an AWS account.

[veikkoeeva]

@bchapuis No problem at all! Would you have tip for the following issue?

I tried java -cp "C:\baremaps\lib\*" com.baremaps.cli.Baremaps import --database 'jdbc:postgresql://localhost:5432/baremaps?&user=baremaps&password=baremaps' --file 'https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf'

and it fails with

[INFO ] 2020-12-07 16:10:20.906 [main] Import - 16 processors available
java : Exception in thread "main" java.lang.UnsatisfiedLinkError: unknown
At line:14 char:1
+ java -cp "C:\baremaps\lib\*" com.baremaps.cli.Baremaps import --datab ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Exception in th...kError: unknown:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

    at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:93)
    at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries(NativeLibrary.java:71)
    at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress(NativeLibrary.java:50)
    at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress(NativeLibrary.java:60)
    at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:138)
    at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:86)
    at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:44)
    at jnr.ffi.LibraryLoader.load(LibraryLoader.java:392)
    at jnr.ffi.LibraryLoader.load(LibraryLoader.java:371)
    at org.lmdbjava.Library.<clinit>(Library.java:125)
    at org.lmdbjava.Env$Builder.open(Env.java:486)
    at org.lmdbjava.Env$Builder.open(Env.java:512)
    at com.baremaps.cli.Import.call(Import.java:116)
    at com.baremaps.cli.Import.call(Import.java:45)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
    at picocli.CommandLine.access$1200(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
    at picocli.CommandLine.execute(CommandLine.java:2058)
    at com.baremaps.cli.Baremaps.main(Baremaps.java:43)'

It appears I miss some native library, but I do do not know which one. I dropped https://jdbc.postgresql.org/download.html to the lib directory, just in case (didn't help).

[bchapuis]

Ok, this is an issue associated with LMDB that will require some investigations (the binary does not seems to be loaded correctly). LMDB is supposed to work on windows but I must admit that I'm not using windows on a regular basis. I opened another issue to track this and will try to come up with a fix later this week. Thanks a lot for reporting this.

[veikkoeeva]

@bchapuis No problem. I was thinking that one next. I'll check too if there's something I could do by just adding binaries or maybe a C runtime or something.

I'm testing this on 64 bit Windows Server 2019. One thing to be aware of that it doesn't support Linux Docker images. I took deliberately the difficult option. :D

[veikkoeeva]

@bchapuis I had both 32 and 64 bit Java runtime installed and the 64 bit one disabled. When I enabled it, loading succeeded. I also tested by enabling just the 64 bit Java and it worked (as expected, but wanted to make sure) – apart for having a duplicate key exception.

A few takes:

• If someone asks or searches, this is now documented here.
• I do not know if this possible to add checks based on architecture and loading or catch this particular native library exception and add a possible explanation so users would have easier time to solve this without hassle.

It's past midnight, but I probably try to produce maps tomorrow (<- edit: a bit later today, hah, have to always make the same mistake :)).

[bchapuis]

Out of curiosity, are you using Windows Subsystem for Linux (WSL)? I was very thrilled by this feature when it came out but I'm not really able to gauge its popularity.

[veikkoeeva]

@bchapuis It is somewhat popular and I have it installed on a desktop. I don't use it often, some do, but it depends on what they do. I just happened to start this on a temporary Windows Server 2019 installation so I might as well follow it through now.

My goal was to actually just to learn to produce vector tiles from .osm files and this looked like one way of doing it while I'm not that familiar with any of this. :)

Now that you bring this up, there is maybe one way to make using/cooperation easier. It might be of interest to some to replicate this using .NET, at least to people like me who are more comfortable with that, but I suppose the same type reasoning would work for other platforms (Python?) From this perspective this would be easier to me if:

1. If the scripts came just as .sql scripts to both create the database and basic ones to create the vector map tiles.
2. Note what does JTS do before the tables are loaded. So this can be replicated using language specific libraries (.NET examples here and here (supports also producing vector tiles, I do not know about performance or quality).
3. Bulk load data to database in a way or another but cross-feeding on ideas in different platforms. I see LBDB used here, might be other options too.

Then the language specific part is more about synchronizing .osm files and bulk loading the database, which may be project dependent in any case. Then cooperation on creating more effective scripts and queries migh be the value add (also GDAL). This is a though I would like to try in the future. I don't know if this sort of an idea is interesting in general or if there is value add there. :) I see there is more development like web server and other features done or planned, but I would guess those too would be nice to have in more language/framework/platform particular way.

[veikkoeeva]

Final update that generating vector tiles seemed to succeed. At least I got out .pbf files in a directory hierarch that corresponds to /tiles/{z}/{x}/{y}.pbf convention as described at https://docs.mapbox.com/archive/ios/maps/api/5.9.0/tile-url-templates.html. I used https://github.com/baremaps/baremaps/blob/master/examples/openstreetmap/config.yaml as the configuration file, so it may have the wrong postion regarding Luxemburg but probably doesn't invalidate the process other than they likely need to be considered too (how to consider it?).

[veikkoeeva]

One more added note, I got the Linux Docker container to run on Windows Server 2019. I installed Docker for Desktop and switched to Linux containers. I do not know why I couldn't get this to work first, may be related to https://github.com/OneGet/MicrosoftDockerProvider/pull/55 kind of issues. This may help someone who gets also an error running the Linux Docker image on Windows and somehow switching the container from Windows to Linux doesn't seem to work.

[bchapuis]

Thanks a lot for looking into this. Yes, the bounds in the configuration file have to be changed according to the bounds of the input PBF file, otherwise, the generation will succeed but the vector tiles will be empty.

So if I understand correctly, #150 can be addressed by using the 64 bit Java runtime. I closed it for now, do not hesitate to reopen it if I miss-understood.

I have only little experience with docker on Windows, is there something I could do when generating the images to make them work on windows more easily?

[veikkoeeva]

@bchapuis I think it can be closed. I do knot know Java, but I think it would be possible to check the environment bitness and JVM bitness and if they are incompatible, issue a clear warning telling it – or maybe trying to load the another 32/64 version of the library.

I am not sure if there is anything you can make the Docker images work smoother apart them being Windows machines. I could imagine it's not worth the hassle since Docker for Desktop has worked just fine for me. It just happened that I used Server 2019 and got into trouble, partly because I'm not a Docker guru (especially not on Windows). Then I chose to go the non-Docker route and got into trouble again. The upside is that now there's some public documentation of the trail.

I tried a bit of my hand on OSM processing with .NET. I think I try replicate the contents of the tables your system produce. To get a better feeling of this data and processing times. If I get far enough, I most likely put the code to GitHub. In that case I think I would go for a more like single-binary commandline kind (ASCII graphics?) of an approach (that loads the scripts as-is and the program only cares about types and parameter names as required by the protocols, this way one can load data to other DBs also by re-creating the scripts).

Do tell if you if you this could be something worth your time. In that case I put code to GH earlier and may ask something, though my intention isn't to squeeze your valuable time. Here's something I dabbled during the few evenings. I'm not sure if that's correct, i.e. all ways in Liechtenstein are lines? Makes sense but can't be sure with my mad skillz. :P This is of course unoptimized (and copy-pasta formatting bad).

using(var fileStream = new FileInfo("../../../../data/liechtenstein-latest.osm.pbf").OpenRead())
            {
                var source = new PBFOsmStreamSource(fileStream);
                Polygonizer p = new Polygonizer(true);
                //WKBWriter wKBWriter
                const int OpenStreetMapProjection = 4326;
                GeometryFactory geometryFactory = new GeometryFactory(new PrecisionModel(PrecisionModels.Floating), srid: OpenStreetMapProjection);

                var allNodesLookup = source.Where(i => i.Type == OsmGeoType.Node).Select(i => i as Node).ToDictionary(key => key.Id, val => val);
                var allWays = source.Where(i => i.Type == OsmGeoType.Way).Select(i => i as Way).ToList();
                var allRelations = source.Where(i => i.Type == OsmGeoType.Relation).Select(i => i as Relation).ToList();

                //Could be parallel (and LINQ and alltogether do better with CPU cache)...
                foreach(var way in allWays)
                {
                    //The order must be preserved with ways, see https://wiki.openstreetmap.org/wiki/Way
                    //and see polygon geometry case. The same with relations? Cache coordinates if same ones
                    //are used, what's this with nullable IDs? Sure a flaggable error?
                    var wayNodeCoordinates = way.Nodes
                        .Select(i => new Coordinate(allNodesLookup[i].Latitude.GetValueOrDefault(), allNodesLookup[i].Longitude.GetValueOrDefault()))
                        .ToArray();

                    Geometry geometry;
                    if(wayNodeCoordinates.Length > 3 && wayNodeCoordinates[0] == wayNodeCoordinates[wayNodeCoordinates.Length - 1])
                    {
                        //If the first and last coordinates are the same and there are more than three nodes, must be a polygon/polyline.
                        geometry = geometryFactory.CreatePolygon(wayNodeCoordinates);
                    }
                    else if(wayNodeCoordinates.Length > 1)
                    {
                        //If more than three coordinates but first and last are not the same, then this is a line.
                        geometry = geometryFactory.CreateLineString(wayNodeCoordinates);
                    }
                    else
                    {
                        //Malformed cases?
                    }
                }
            }

            //No polygons in Liechenstein ways?

<edit reading through https://github.com/OsmSharp/core/blob/c62b6d445187e15df35a310bac72e6300d8219d2/src/OsmSharp.Geo/DefaultFeatureInterpreter.cs, interesting, interesting (while not fastest?).

[UsulPaul]

Hi @bchapuis,

I am trying to use baremaps to serve tiles from a postgis database and I cannot connect to the database. I get a message saying that _user_ and _password_ is not a recognized command. I have tried with "" '' and without, removed the ?. The credentials are definitely correct as I am currently connected to the database from QGIS. Also, the yaml files is not seen for some reason, although I added the path. I am no coding expert, but I do understand GIS very well. I enjoy baremaps as it seems very simple to use, compared to other tools that do a similar thing. Any pointers will be highly appreciated.

C:\Users\frant>baremaps serve -- database='jdbc:postgresql://127.0.0.1:5432/central_america&user=postgres&password=pjQK5erhCci4' config='C:\wgetdown\baremaps\config.yaml' --watch-changes Missing required option: '--config=YAML' 'user' is not recognized as an internal or external command,operable program or batch file. 'password' is not recognized as an internal or external command, operable program or batch file.

[veikkoeeva]

@frantiucalexandru I think you need to see that -- is next to database (there's a space now) and remove the = signs from the command-value pairs. E.g here's what worked for me java -cp "C:\baremaps\lib\*" com.baremaps.cli.Baremaps export --database 'jdbc:postgresql://localhost:5432/baremaps?allowMultiQueries=true&user=baremaps&password=baremaps' --config 'config.yaml' --repository 'tiles/'.