Boundaries lack date information

[gregallensworth]

Created in response to a comment https://github.com/OpenHistoricalMap/openhistoricaltiles/issues/3#issuecomment-418573034 that this shape is not showing its start_date in the dateissues map.

http://www.openhistoricalmap.org/way/198180496 https://openhistoricalmap.github.io/openhistoricaltiles/dateissues/#7.38/44.91845/-69.47637/

Confirmed here, that boundaries lack any date info. I have comments in here, that boundaries seem to be imported via import-osmborder which was a separate process (and one which may be separate from the later diff-consuming process???), and that these boundaries lacked date information in the osm_border_linestring table and its derivatives.

[gregallensworth]

Collecting some URLs and notes for reference:

The import-osmborder image, does not specify to use the start_date and end_date fields.

https://github.com/OpenHistoricalMap/import-osmborder/blob/master/import_osmborder_lines.sh#L18-L21

Not that the generalized tables would also need modification to include the new fields: https://github.com/OpenHistoricalMap/import-osmborder/blob/master/import_osmborder_lines.sh#L40

The source CSV used by import-osmborder is generated by the makecsv-osmborder command, which in turn runs https://github.com/OpenHistoricalMap/osmborder-docker/blob/master/build-osm-borders.sh to download a ohmplanet$DATE.osm.bz2

I do already have a ohm_planet.osm.bz2 dated August 2, and the resulting osmborder_lines.csv. The CSV does seem to have only the 6 fields specified in https://github.com/OpenHistoricalMap/import-osmborder/blob/master/ and no more than that. So this would be something further back in osmborder

Judging by the docs at https://github.com/pnorman/osmborder and the source code https://github.com/pnorman/osmborder/blob/b421a906de8d2fd0a5637775d8c1e480658d2111/src/adminhandler.hpp#L113-L165 it looks like the osmborder tool does specifically fetch these fields and only these, and would need modification and compile-from-source in order to add the start_date and end_date fields to the output CSV.

[gregallensworth]

Tasks:

• [x] fork and modify the osmborder tool, to include OHM's fields; should also add the name field while we're at it

• [x] alter the makecsv-osmborder docker, to use this new version of osmborder and confirm, that the output CSVs are formatted as expected

• [x] alter the import-osmborder docker, to include these 3 new fields, re-run to check

• [x] set up scheduled task to run this whole series every few hours; this chain of tools doesn't use imposm and its diff-consuming capability, so looks as if it'll be a drop-and-load?

[gregallensworth]

I have modified the tools and re-run these two steps, and the osm_border_linestring tables now properly have start_date, end_date, and name fields. I have modified the OpenMapTiles CREATE VIEW statements as well, to expose these new fields into vector tiles.

Looking good so far:

I do see some testing and improvements to the dateissues map, but good news is that boundaries now have dates and names.

[gregallensworth]

There is now a scheduled task in place, which runs the modified makecsv-osmborder and import-osmborder from the OHM planet file, then re-creates the boundary_zX views for use in vector tiles.