search

AtlasOfLivingAustralia / data-management

Data management issue tracking
7 stars 0 forks source link

Track daily data resource counts #804

Closed peggynewman closed 2 years ago

peggynewman commented 2 years ago

We need to keep track of DR counts every time we load a new index, so we can see daily movements but also use for metrics reporting.

djtfmartin commented 2 years ago

A simple solution for this that could be a jenkins script or airflow script that:

1) Run the ./la-pipelines validation-report tool (this reads all the metrics files and produces a CSV) 2) Pushes the CSV into a github repo (github formats CSVs and shows them as tables)

You can then use the github UI to see the difference day-to-day.

As an added bonus, it would be nice if it only pushed to the github repo if something had changed.