search

epiforecasts / covidregionaldata

An interface to subnational and national level COVID-19 data. For all countries supported, this includes a daily time-series of cases. Wherever available we also provide data on deaths, hospitalisations, and tests. National level data is also supported using a range of data sources as well as linelist data and links to intervention data sets.
https://epiforecasts.io/covidregionaldata/
Other
37 stars 18 forks source link

Germany data not loading from source #408

Closed RichardMN closed 3 years ago

RichardMN commented 3 years ago

Fails in our tests and fails in my covidregionaldatagraphs run. Not finding a column "Meldedatum" in the inputs?

Failure with the source data url: https://opendata.arcgis.com/datasets/dd4580c810204019a7b8eb3e0b329dd6_0.csv

Loading it from a web browser gives

{"type":"Feature Service","serviceUrl":"https://services7.arcgis.com/mOBPykOjAyBO2ZKk/arcgis/rest/services/RKI_COVID19/FeatureServer/0","sourceLastModified":"2021-08-22T03:06:34.075Z"}
seabbs commented 3 years ago

Nice flag and debugging @RichardMN. Will try and find some time to look into.

RichardMN commented 3 years ago

A bit more fiddling.

Using the RKI restful API explorer I generated an url which should generate just the five columns we actually use: https://services7.arcgis.com/mOBPykOjAyBO2ZKk/arcgis/rest/services/RKI_COVID19/FeatureServer/0/query?where=1%3D1&outFields=Bundesland,Landkreis,AnzahlFall,AnzahlTodesfall,Meldedatum&outSR=4326&f=json

But when I run this, I get JSON (which we could probably work with) and more critically the warning (in the JSON)"exceededTransferLimit":true

{"objectIdFieldName":"ObjectId","uniqueIdField":{"name":"ObjectId","isSystemMaintained":true},"globalIdFieldName":"","fields":[{"name":"Bundesland","type":"esriFieldTypeString","alias":"Bundesland","sqlType":"sqlTypeNVarchar","length":2147483647,"domain":null,"defaultValue":null},{"name":"Landkreis","type":"esriFieldTypeString","alias":"Landkreis","sqlType":"sqlTypeNVarchar","length":2147483647,"domain":null,"defaultValue":null},{"name":"AnzahlFall","type":"esriFieldTypeInteger","alias":"AnzahlFall","sqlType":"sqlTypeInteger","domain":null,"defaultValue":null},{"name":"AnzahlTodesfall","type":"esriFieldTypeInteger","alias":"AnzahlTodesfall","sqlType":"sqlTypeInteger","domain":null,"defaultValue":null},{"name":"Meldedatum","type":"esriFieldTypeDate","alias":"Meldedatum","sqlType":"sqlTypeOther","length":0,"domain":null,"defaultValue":null}],"exceededTransferLimit":true,"features":[{"attributes":{"Bundesland":"Schleswig-Holstein","Landkreis":"SK Flensburg","AnzahlFall":1,"AnzahlTodesfall":0,"Meldedatum":1605052800000}},

...

So it looks as though ways forward may be:

Edit - I've created a draft PR #409 which has standalone code which successfully downloads lines from the RKI data (using httr and jsonlite). It is neither clean nor pretty but it might form the basis for a new download function specific for Germany.

RichardMN commented 3 years ago

This issue appears to have been resolved upstream.

seabbs commented 2 years ago

Hi Richard,

Sorry to have not got to this. Perfect storm of annual leave and moving house. Just going through the PRs etc but from what your saying looks like they have resolved this?