[WIP] Updater

[SlowMo24]

Write update-package that fulfils framework set out here: https://confluence.gistools.geog.uni-heidelberg.de/display/oshdb/Update+process+framework

Missing implementations are:

Updater Package

• [x] implement different backends (ignite, h2 etc)
• [x] insert new tags into tagtranslator-db BEFORE creating new entity
  • [x] update TagTranslator cache if new tag or role is inserted to keytables or handle dublicate insertion exception
• [x] account for back-references of OSHEntites (e.g. ways for nodes for changed ways and changed nodes)
  • [x] account for back-references of backreferences of OSHEntites (e.g. relations depending on ways depending on changed nodes)
  • [ ] Relations are excluded for now, but might get important in a later update? @rtroilo
• [x] implement flushing of updates to real database
• [x] implement Kafka promoter (do we actually want this, its not hard to do but also a feature with no use-case so far)
• [x] collect OSHEntites before processing to avoid multiple calls to same method
  • [ ] test if this actually is an improvement
• [x] collect OSHEntites into insertCells before flushing to prevent multiple overwriting of the same Cell
• [ ] resolve comments in code
• [ ] assure concurrency (most of all, check if locking in flusher is safe)
• [ ] optimise Code (very important last step!)
• [ ] analyse and clean dependencies
• [ ] extensive testing!
• [ ] codeStyle, staticCodeAnalyses etc (resrouce handling)
• [ ] documentation
• [ ] correct errors introduced by https://github.com/GIScience/oshdb/pull/69/commits/2ab2f883d7804ee9963170f9691a7e9281a7fef8
  • [ ] check if functionality was preserved
• [ ] there might be a mixup of CellId.getId() and CellId.getLevelId() in the Flusher and possibly elswhere! Should be investigated!

OSHDB-API

• [x] write class to append entities from update-db to stream when querying over OSHDB-API (I need help to point me to the right place where this would be implemented @tyrasd ?)
  • [x] implement ScanQuery and LocalPeek OSHDB-API-Ignite Backends
  • [ ] move streaming of updates up do MapReducer-Class. It is only copy code now.
  • [x] group 100 or 1'000 entities together in one 0,0 GridCell in OSHDB-API to avoid overhead of CellCreation (and calling of celliterator methods)
• [ ] improve testData for updates to speed them up (Area to big atm)

ETL-Backend

• [ ] implement ETLFileReaderClass (help needed from @rtroilo )

Dependencies

• [ ] check if ignite-geospatial is actually necessary as a dependency or only needs to be available at cluster startup on the server side. If so, remove artifact from artifactory and remove dependency.
• [ ] check if h2gis is necessary as a dependency

[SlowMo24]

😢😢😢