search

tl-its-umich-edu / unizin-validation

Unizin Validation Scripts
1 stars 4 forks source link

Compare values in csv with previous run #3

Closed jonespm closed 3 years ago

jonespm commented 5 years ago

We would like to write the values of the csv and compare it with the previous run of this job. This would allow us to perform some additional checks.

jonespm commented 5 years ago

This is going to require switching to the database or keeping files on the filesystem to work because we're deleting the previous runs now.

jonespm commented 4 years ago

We have a MySQL database setup for production to be able to store this data. Either add a MySQL to a docker-compose locally or just use SQLLite locally to test. But the values from the previous day can be stored in a database. Maybe just use SqlAlchemy and Alembic to keep it light?

jonespm commented 3 years ago

I don't think this task is worth the effort. Maybe if we ever started storing the runs in a database, but that would have to come first.