search

khaeru / sdmx

SDMX information model and client in Python
https://sdmx1.readthedocs.io
Apache License 2.0
24 stars 18 forks source link

Eurostat / unable to run the ten-line example #181

Closed nicolas-graves closed 4 months ago

nicolas-graves commented 4 months ago

Hi, thanks for your work on this package.

Has Eurostat updated its policy on download agent tolerance?

I'm able to get this URI through my browser, but neither through python-requests nor wget. https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/nrg_bal_c/latest

It doesn't seem to be related to missing certificates :

requests.get("https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/nrg_bal_c/latest", verify="/home/graves/spheres/work/cired/git/hybridization/eurostat.pem")

will not work either, despite having the right certificate downloaded from dnschecker.org.

Do you also have the issue on your side? This makes it not possible to even run the ten-lines example. Thanks!

nicolas-graves commented 4 months ago

Seems that it works again when forcing IPv4 on my end, with curl -4 for instance. The requests configuring hack here: https://stackoverflow.com/a/72440253 is a valid temporary solution. I've sent an inquiry to eurostat upstream.

khaeru commented 4 months ago

Hi —thanks for reporting this issue and your work-around!

You don't say directly: was sdmx1/requests/urllib3, on your system/network, attempting to use IPv6 to make this connection? Was there a particular error message that indicated so? If so, that or the fact that curl -4 worked would seem to imply that the Eurostat API endpoint is not available via IPv6.

Within this repo, I have this nightly workflow that attempts to access all the known REST web services/data sources. You can inspect the latest run (on GitHub, or the summary page at https://khaeru.github.io/sdmx/) and see that the ESTAT and ESTAT3 endpoints both are reachable. I guess this implies that on the GitHub Actions runners the code chooses IPv4 for the same queries.

Please do report if Eurostat has any reply about this detail of their service. The best-case is always that each data provider/data source operator gives clear and up-to-date description of the limitations/idiosyncratic features of their own API endpoint. Then, this package can adapt to those. While it is also possible to adapt to unadvertised/unacknowledged idiosyncrasies, or merely document work-arounds, that can involve time-consuming guesswork, so I prefer to avoid it.