Add jdbc import

[afischerdev]

This is according to discussion #486 and adds tags via database call to the way generation. New is forest to find nice ways in or near forest river to file ways along water noise to avoid bigger streets town to avoid bigger citys traffic was used before but is now generated by database call

The classpath in generation needs a jdbc driver and the OsmFastCutter a call with some database parameter Please see readme.

There is also a new lookups.dat that contains the new tags - before the node context. This should be neutral to older versions. In the next generation lookups.dat (#458) I would prefer these new tags more moved up to get smaller rd5 files.

This needs also an update for the profiles to make use of the new tags.

[abrensch]

I tried to run this branch on the "mapcreator" account of new dev server, but I'm stuck with a very strange effect: Call to OsmFastCutter is returning immediatly, but the process is running on, so script is continuing with the second step (PosUnifier) but that fails...

Probably has noting to to with the jdbc option, more with the new PBF-Parser?

[afischerdev]

@abrensch I rechecked the jdbc handling. wrong driver wrong database wrong user wrong password On all errors the process is killed by System.exit And next process stops with 'bordernids.dat not found'.

So sounds as this is jdbc problem. I could change this: remove System.exit with a smoother handling

[abrensch]

So sounds as this is jdbc problem. I could change this: remove System.exit with a smoother handling

A non-zero exit code would be smooth enough :-)

However, I still do not understand why the process keeps running. Looks like a second thread forked or the like (?)

[afischerdev]

@abrensch Ok I will change to exi(1). On my run next process stops with 'bordernids.dat not found'. May be no cleaning first?

I was preparing for smoother handling, but not sure if it's a good idea to continue a process when no database call is in progress. The (incorrect) results will therefore only be displayed to you once the entire process has been completed.

[afischerdev]

@abrensch @mjaschen Update is done. Zip file contains new lib

[afischerdev]

• .sql and .lua don't use a consistent format, perhaps run some formatter/beautifier

Not easy to find a formatter, most breaking on postgres functions. I used https://sqlformat.darold.net/ online formatting for now, but also this breaks with errors on database generation. That needs more time.

• documentation in mapcreation folder: Should we keep it there or move it to docs?

Yes, we should. But at the moment this is more a todo list for finding a right script for generation. At the end we have a new db_gen.sh and new rd5_gen.sh. May be an db_update.sh All this is not ready.

• absolutly no tests: It requires quite some setup to import OSM data into postgis and process it. For tests we should perhaps use some (in-memory) SQLite database which has an all_tags table.

Yes, and I don't have an idea for a change. The scripts need postgres functions. So a SQLite could only hold pregenerated values. But that is a fake test.

• Is it possible to update the database using osm2pgsql?

I hope so. But it is not developed or tested.

• Should the database use more (materialized) views instead of tables because we don't need to modify the data?

This is not tested as well. But yes, it is an idea for the future.

[abrensch]

I don't get it.... OsmFastCutter keeps putting itself in the background as soon as there is a 13. argument looking slighty like a jdbc-url. Not for "xyz", but for "jdbc:oracle:..." and even without the postgres driver in the classpath And from the code I would expect that the JDBC-Logic is only initialized when the first Osm-Way is decoded (which would be after half an hour decoding nodes), but it happens immediaty...

[afischerdev]

I would expect that the JDBC-Logic is only initialized when the first Osm-Way ...

That is the way it works. Please see this line in nextWay routine. I was able to check it in my logs. OsmFastCutter only forwards the jdbc argument to OsmCutter (please see here).

But the mapping to "jdbc:oracle:..." sounds strange.

[abrensch]

OsmFastCutter keeps putting itself in the background as soon as there is a 13. argument looking slighty like a jdbc-url

Blame on me... the "&" in the jdbc-url is interpreted by the shell... just needs quotes.

[zod]

Yes, and I don't have an idea for a change. The scripts need postgres functions. So a SQLite could only hold pregenerated values. But that is a fake test.

It's pretty common to use mocks/fakes/doubles to test functionality on a unit or module level. This also establishes a clear boundary because you can easily see which data is expected by generateTagsFromDatabase.