Import framework

[fb71]

Sub tasks:

• [x] 27: WMS import (http://github.com/Polymap4/polymap4-p4/issues/issue/27) - create catalog entry
• [ ] 28: Shapefile import (http://github.com/Polymap4/polymap4-p4/issues/issue/28) - actually import something
• [x] 29: Archive file import (http://github.com/Polymap4/polymap4-p4/issues/issue/29) - prepare data for other importers

  Specification

The goal of an "import" ist to transform a given input into something that can be transfered into a standard data store. Currently we are about to import feature data und image data (WMS) but this might be expanded to work with raster data in the future. It is out of scope of the import to shape or beautify data. This ist done on the imported data.

There is a huge number of input sources and formats in many different flavours. A lot of them are not well specified and even if so the particular input might diverge. So the import is about assumtions, guesses and questions. For example:

• what is the CRS of the shapefile? (if *.prj is missing)
• what is charset of a *.zip file? (filesnames and content)
• what is delimiter of a CSV?
• what are column name and types of CSV?
• does a WMS require authentication?
• what layers to import from a WFS? (?)
• is the URL a downloadable dataset or a service URL?
• ...

The basic job of the import is to:

• present current status to the user: fact/info (green), assumtions (yellow), questions (red)
• present the current state of the import (files of a zip, table of CSV)
• allow user to answer questions and validate assumptions
• loop to top

Verifies assumptions and answered questions are decisions. Decisions may depend on other decisions. Decisions are:

• a description of status and question or problem
• a single click action (verify, delete, multiple choice)
• full blown UI to gather more information from the user

Data source / input

• archive files (zip, tar, whatever... )
• single files
• URLs (download or service)

[joergreichert]

Reminds a bit of Common Google Maps Engine Workflows, now in CartoDB.

Other aspects:

• import is a one in time import or a continuous / polling import (e.g. executing a scraper, think of import.io, morph.io = batch job)
• import is a transformation / aggregation of several sources (e.g. bus stops + current construction side news + accident news, when a bus stop isn't served due to a construction or an accident, this bus stop shouldn't be imported)
• import requires additional user input / decisions (from different persons), so the user have to be notified by e.g. email, that he have to login into the system to make a decision or acknowledge a decision
• user permissions to import / see / change data

[stundzig]

Data sources could also be specialized importers/adapters to different external system. For example i would implement a TomTom Webfleet Importer. Therefore i need the login data to the external system from the user and a schedule interval. With this schedule interval, the TomTom Webfleet Importer connects to the external system, reads all car informations including the current GPS Position and updates the internal feature collection.

I think, these part, specialized importers for enterprise APIs could be a real benefit.

Additionally, a ArcGIS Importer makes sense in this scenario. Internal its a specilization of a WMS/WFS importer, but for the user it requires only the ArcGIS login/pw and shows the user a list of all possible importable Datasets from his ArcGIS account. So the user has no need to find the WMS URL and copy/paste them into mapzone.

[stundzig]

If i import a PNG file, i don't see any error message. And also in the logs i got [WARN] UploadService : Upload handler already registered for id: 1378271589

[fb71]

Feature import basically works.

To do:

• [x] auto create layer for new resource?
• [ ] update UI after import: CatalogPanel (currently unnecessary)
• [x] update UI after new layer
• [ ] progress dialog while feature import
• [ ] check if schema name already exists -> https://github.com/Polymap4/polymap4-p4/issues/68

[fb71]

Open enhancement and bugs are worked on in separate issues.