search

CLIMADA-project / climada_python

Python (3.8+) version of CLIMADA
GNU General Public License v3.0
291 stars 115 forks source link

Data freshness #856

Closed IanHopkinson closed 2 weeks ago

IanHopkinson commented 4 months ago

Is there a way to establish whether there is new/changed data being delivered by the CLIMADA API?

I'm mainly interested in large changes, for example another year of historical data being added to an indicator or the basis year of something like LitPop being changed.

The context for this question is that I am looking for a trigger to refresh the data on HDX. In the absence of a simple trigger I'll probably just set a yearly schedule to update data.

I considered using versions of this repo as a guide but as I understand this simply reflects changes in the mechanism of data delivery rather than a change in the underlying data.

emanuel-schmid commented 3 months ago

Thanks for the question. There is the 'activation_date' that is an attribute of every dataset. Perhaps that could be helpful? There is no "greater than" condition for it so you would have to fetch all datasets, e.g. into a data frame and query that:

from climada.util.api_client import Client
client = Client()

datasets = client.into_datasets_df(client.list_dataset_infos())
since_last_november = datasets[datasets.activation_date > "2023-11-01"]
peanutfun commented 2 weeks ago

This question seems to have been answered appropriately. Feel free to reopen this issue if something remained unclear.