search

codeforIATI / iati-tables

https://iati-tables.codeforiati.org
MIT License
1 stars 1 forks source link

Datasette: update log/stamp #3

Closed stevieflow closed 8 months ago

stevieflow commented 3 years ago

Currently on https://datasette.codeforiati.org/ we can't readily access the lastupdate dateTime - which we know is at https://iati-tables.codeforiati.org/

stevieflow commented 3 years ago

maybe also point to https://ocdsdata.opendataservices.coop/tree?dag_id=run-iati&root= in both instances (@kindly : should / could we put this process in its own Airflow? Does that make any sense?)

kindly commented 3 years ago

It was not worth the effort of making a whole new airflow instance for this task and why I did not do it.

There may be value in a code-for-iati airflow instance where lots of the long running tasks are managed as at the moment everyone has there own not very transparent cron jobs.

All the logs are stored in the S3 bucket and I need to think of a good way of gettting the iati-tables website and datasette.

I also would need to work out how to customise datasette a bit before any information could be displayed in it.

odscjames commented 11 months ago

Currently on https://datasette.codeforiati.org/ we can't readily access the lastupdate dateTime

We can create a special meta table and write it to that. It won't be totally obvious in the UI but people can still do a couple of clicks / 1 simple SQL and see the value.

Also that way it will make it to all the downloads (PG & CSV as well as SQLite), which is good.

tillywoodfield commented 8 months ago

There is now a table called metadata in datasette and all the other output formats containing two columns data_dump_updated_at (taken from the IATI data dump used for processing), iati_tables_updated_at (the time when Tables processed the data)