track ingestion process for robustness and reporting of status / issues

[marc-portier]

Currently the setup and actual processing involves quite a number of steps. When some error happens along the line the user / administrator does not have an easy view on where things went awry.

Typical / Practical encountered issues - puzzling situations:

• when adding source
  • wrong url / gives status != 200 ok
  • content unreadable (parsing error - bad ttl, jsonld, ...)
  • in case of LDES --> unclear how many pages fetched, until the end,
• when analyzing source (in case of LDES)
  • no properties are actually read
  • total amount of fetched entities limited at some count, possibly LDES fetching halted at some page ?
• when running unification
  • how many of the found entities have been passed to index / when was the last one ?
  • how many of those can be found in the index (counting all) / when was the last one ?

? other

[nvdk]

We believe this will be covered in #37 and #38