Open nesnoj opened 3 months ago
Hello @nesnoj, you're right: due to IT security reasons we had to make some drastic changes to our opendata plattform which included a complete re-design of the former way we stored and retrieved the data. We have not been able to re-build the former database with all its functionality (e.g. filtering with operators like gt / ge etc.).
Not knowing the internals of the dissaggregator tool I cannot really tell how many adaptions you need to implement to make the tool fully functional again, but you figured out the most important ones already (i.e. different URL, unsupported operands in the query parameters).
Now regarding your specific problem: I just checked and the year 2022 is not available for id_spatial 71 in the dataset. Are you sure this existed at some point or did you just choose these parameters randomly?
Also, the URL is not quite correct as you need to use
https://api.opendata.ffe.de/demandregio/demandregio_spatial?id_spatial=71&year=2016
instead of https://api.opendata.ffe.de/demandregio_spatial?id_spatial=71&&year=2022
.
You can refer to the new API documentation for further details.
Thx for your quick reply @jferstl! And thanks for the URL, now some queries do work to some extend again. E.g.
spatial.disagg_households_power(
by="households",
weight_by_income=True,
year=2022,
scale_by_pop=True,
)
builds query https://api.opendata.ffe.de/demandregio/demandregio_spatial?id_spatial=13
(after chucking away), returns data but crashes
2024-05-22 08:51:56 disaggregator.config: ERROR id_spatial 13
title Stromverbrauch nach Haushaltsgröße
oep_metadata {'name': 'id_spatial=13', 'title': 'Energy con...
ffe_metadata {'id_region_type': [3]}
data [{'id_spatial': 13, 'id_region_type': 3, 'id_r...
dtype: object
Traceback (most recent call last):
File "/some/path/disaggregator/disaggregator/config.py", line 127, in database_raw
df = pd.read_json(BytesIO(requests.get(host + query).content),
...
The query in the first post was an example which was generated by the disaggregator and worked before, it's still not working with the fixed URL though.
There're multiple bugs (see other issues) we fixed in a fork but I don't think we will rework the API.
Hello, I also used the DemandRegio disaggregator for my work. I want to generate some new data right now, but as already mentioned in this issue, the disaggregator does not work anymore because of the change in the API and the data structure. I have managed to get around some of the problems using local files, but there are still many datasets for which I can't find the exact replacement. For example, in the data.database_get()
function
if dimension in ['spatial', 'temporal']:
id_name = 'id_' + dimension
if dimension == 'spatial':
if cfg['use_nuts_2016'] and table_id in cfg['nuts3_tables']:
table = 'v_demandregio_spatial_lk401'
# table = 'demandregio_spatial'
else:
table = 'demandregio_spatial'
else:
table = 'demandregio_temporal'
I can not find the v_demandregio_spatial_lk401' dataset so far. This means I can only access data under
https://api.opendata.ffe.de/demandregio/demandregio_spatial`, which doesn't give me the same result as what is shown in the example notebooks.
After the relaunch of the FfE open data portal I tried to get the disaggregator working:
tl;dr The interface must be adapted, the disaggregator is useless. As of now the only way of using the data is to use the python way described in the how-to.
Details: I updated the remote URL and had to restict ruamel.yaml to <0.18.0 to get it installed and running (cf. forked version).
However, the API seems to be incompatible. E.g. when running
I get
Apparently, the new API has a different format (cf. how-to), e.g. the former operators such as
eq.
are missing. So I removed the prefixes indata.py
but without success, still getting