search

ideal-postcodes / postcodes.io

UK postcode & geolocation API, serving up open data
https://postcodes.io
MIT License
1.23k stars 244 forks source link

Unexpected and maybe incorrect discrepancy in API response (NUTS, ITL) #786

Closed Andras1000 closed 3 years ago

Andras1000 commented 3 years ago

Compare responses from curl https://api.postcodes.io/postcodes/TS68BD and curl https://api.postcodes.io/postcodes/TS6%208BD

They are mostly the same, but codes/nuts is different, "UKC12" for the first (correct) and "TLC12" for the second (I think incorrect).

To me this is unexpected and maybe even incorrect behaviour.

UPDATE: Have the same issue when the postcode is lower case. curl https://api.postcodes.io/postcodes/ts68bd

cblanc commented 3 years ago

Hi @Andras1000. This was a caching issue. I've manually wiped the cache, let me know if you're continuing to see stale data

The infrastructure and deployment pipeline was overhauled this month to increase capacity and make scaling more straightforward. In that process I introduced a bug where cache busting wasn't triggered on Cloudflare at the end of the deploy.

Andras1000 commented 3 years ago

curl https://api.postcodes.io/postcodes/TS68BD is now returning "TLC12" for code/nuts, which I don't think is a valid NUTS code for the UK, see https://en.wikipedia.org/wiki/NUTS_statistical_regions_of_the_United_Kingdom.

As I understand, all NUTS codes start with the letters "UK".

cblanc commented 3 years ago

NUTS has been replaced with ITL in the most recent release of ONSPD.

International Terratorial Levels (ITL) (Former Nomenclature of Units for Territorial Statistics (NUTS). The national LAU1-equivalent code for each postcode. Pseudo codes are included for Channel Islands and Isle of Man. The field will otherwise be blank for postcodes with no grid reference. 

As of May 2021. NUTS has changed to International Territorial Levels (ITL). Postcodes.io will report ITL in nuts and codes.nuts to preserve backwards compatibility. 

Following the UK’s withdrawal from the EU, a new UK-managed international statistical geography - ITL (International Territorial Levels) - was introduced from 1st January 2021, replacing the former NUTS classification. They align with international standards, enabling comparability both over time and internationally. To ensure continued alignment, the ITLs mirror the NUTS system. They also follow a similar review timetable – every three years.

I've tried to extract NUTS from the most recent ONSPD release, but the LAU1 codes used to identify them were reorganised on January 1st. In other words, it's not possible to take the latest key and use the old NUTS lookup table (https://github.com/ideal-postcodes/postcodes.io/blob/13.1.0/data/nuts.json)

Here's the new NUTS (ITL) translation dataset and reorganised LAU1 keys so you can see the difference: https://github.com/ideal-postcodes/postcodes.io/blob/master/data/nuts.json

According to ONSPD release notes:

Lookups between ITLs and NUTS will be maintained and published until 2023...

I've searched https://geoportal.statistics.gov.uk and https://www.ons.gov.uk and failed to turn up a LAU1 -> NUTS or ITL -> NUTS lookup table. In the end, I concluded what they might have meant was that ITL format will continue to mirror the NUTS format until 2023.

Just how they will be continue to be mirrored is apparently somewhere on the ONS website, but this too eludes me. The link in ONSPD takes the reader to the ONS homepage. I suspect the pattern going forward is the 2 letter UK prefix will be replaced by TL. i.e. UKJ46 is now TLJ46 and UKG24 is now TLG24. But this is only a guess

If you require the old NUTS prefix, you can resurrect an old copy of the DB with docker. Anything < 14.0.0 would suffice. (i.e. change latest to 13.1.0 in docker-compose.yml)

Andras1000 commented 3 years ago

I see the issue as resolved. The behaviour changed due to changes in the official UK geostatistical definitions.

The discrepancy was due to caching.