search

dataobservatory-eu / observatory_workflow

Testing our Datathlon workflow
0 stars 4 forks source link

Try to create a few indicators to all observatories #14

Open antaldaniel opened 3 years ago

antaldaniel commented 3 years ago

Use as an example this, and just save whatever you can into one single file per observatory (music, greendeal, economy)

The indicator package is now in the open. Its core functions (data collection and imputation) are well-covered by unit tests, but the database metadata functions are not, and I think they do not correctly count the number of forecasted, approximated values. But it is not a big issue now. Any problems - please report them there.

Curated Indicators

pitkant commented 3 years ago

Should indicators for all three observatories be saved in one database file or should I create separate databases for each observatory?

nvm, read the first paragraph again and the question answered itself

pitkant commented 3 years ago

It is now done and uploaded to Zenodo. I hope they fulfil the minimum requirements. Please let me know if there is something more you want from the datasets.

There were some difficulties with impute_indicator function and some economics related datasets such as bd_hgnace2_r3, ipr_ta_popr, ipr_da_popr and nama_10r_3gdp. I don't know why but these irregularities could warrant a closer look in the near future.

antaldaniel commented 3 years ago

Hi Pyry, I think actually the most natural place for the test observatories would the data-raw/ of hte indicator package. If you save there, and report any irregularities in an issue, we can actually improve the code right away.

pitkant commented 3 years ago

Test databases now uploaded in my fork and a pull request has been created. I will open issues about the irregularities later this day

antaldaniel commented 3 years ago

@pitkant, great idea to start with COFOG. I created an 'own' indicator after selecting two existing ones from the Eurostat gov_10a_exp statistical product. But I would not experiment on Zenodo, it is permanent, and they will kick you out if you spam it.

I am adding the code as an example to indicators/data-raw/centralization_of_gov_cultural_spending.R

Zenodo example

I very strongly suggest you to first upload a few meaningful examples manually to Zenodo, or even Zenodo sandbox, because Zenodo is permanent!

If you later want to automate Zenodo tutorial, we can eventually put this on the rOpenGov / observatory blogs.

pitkant commented 3 years ago

The files are in the pull request: dataobservatory-eu/indicator#3

Also in Zenodo: http://doi.org/10.5281/zenodo.4776796 http://doi.org/10.5281/zenodo.4775787 http://doi.org/10.5281/zenodo.4775313

I agree that it's probably best not to upload many more datasets in Zenodo in the short term, but in case such examples are needed for testing they are now there, linked to their respective observatories (pending approval).