Getting metrics from ES for datasets that are missing format types - Githubissues

DataONEorg / metrics-service

An efficient database and REST API for delivering aggregated data set metrics to clients.

Apache License 2.0

2 stars 1 forks source link

Getting metrics from ES for datasets that are missing format types #50

Open rushirajnenuji opened 5 years ago

rushirajnenuji commented 5 years ago

There are roughly ~4 Million logs in ES that do not have SOLR metadata. See this document for top 1000 such datasets

Without this metadata the metrics service can't decide the metric type, so it skips the datasets while processing metric counts. Also lots of .

Possible reasons identified for missing metadata from ES:

Dataset PID was not synced with the CN (synchronization error) - less possible; but true for pid doi:10.5063/F1BG2KX8
Log processor running at the MN level writing erroneous logs. (having only a single / after http: Note: the error was resolved sometime back in 2015 or 2016 - however the logs were not reprocessed!

Actionable items:

Rely on Metrics Service to retreive the metadata from SOLR for all the datasets.
Reprocess the log events having single / ater http: