At the moment, the process for tracking progress between stages 1 and 2 of the harvest is extremely manual.
It would be possible to ameliorate this by:
Adding a field in the harvester-stage-2-state which holds the value of updatedLastSeen as a Date rather than epoch value (e.g. updatedLastSeenAsDate)
Creating an index pattern of the form 'harvester*' in Kibana to allow querying across indices.
Creating visualisation comparing the before and after values of 'updated' in the harvester-raw index to give a rough sense of how much processing remains to be done.
At the moment, the process for tracking progress between stages 1 and 2 of the harvest is extremely manual.
It would be possible to ameliorate this by:
Adding a field in the harvester-stage-2-state which holds the value of updatedLastSeen as a Date rather than epoch value (e.g. updatedLastSeenAsDate)
Creating an index pattern of the form 'harvester*' in Kibana to allow querying across indices.
Creating visualisation comparing the before and after values of 'updated' in the harvester-raw index to give a rough sense of how much processing remains to be done.