Create process to check for upstream updates to GNAF database - Githubissues

LukePrior / nbn-upgrade-map

Interactive map showing premises eligible for the NBN FTTP upgrade program.

https://lukeprior.github.io/nbn-upgrade-map/

MIT License

101 stars 11 forks source link

Create process to check for upstream updates to GNAF database #216

Open lyricnz opened 11 months ago

lyricnz commented 11 months ago

And reprocess to create our cut-down version. https://github.com/LukePrior/nbn-upgrade-map/tree/main/extra/db

This doesn't happen very often - roughly quarterly https://data.gov.au/data/dataset/geocoded-national-address-file-g-naf

LukePrior commented 11 months ago

We could make a weekly action that checks if an upstream Docker container exists with a newer version?

lyricnz commented 10 months ago

The update script already fails cleanly if we have the current version. While we could make a process to run this automatically, it has the potential of breaking things if the schemas change....

Will PR a change to actually download the dump file if it doesn't exist.

lyricnz commented 10 months ago

Perhaps it's better to make a GHA action that checks for an updated upstream image, and notifies (create an issue?) - rather than upgrading the image automatically. This would be safer.

LukePrior commented 10 months ago

~Maybe it could open a PR with the changes, but just opening the issue is probably enough~

lyricnz commented 10 months ago

I don't think there's actually any changes? We use :latest in all the automations, and there's already a GHA that builds and pushes the new image.

https://github.com/LukePrior/nbn-upgrade-map/blob/main/.github/workflows/publish-db-image.yml

LukePrior commented 10 months ago

My bad was getting confused, you're right

lyricnz commented 10 months ago

@LukePrior There's a new GNAF 202308 release. https://data.gov.au/dataset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/details

It's already integrated in our upstream https://github.com/minus34/gnaf-loader/pull/81 but he hasn't created a release yet.

However the DB dump files etc that our Dockerfile uses are all there - so the build works if you explicitly set the GNAF_LOADER_TAG=202308

lyricnz commented 10 months ago

From the government data release: G-NAF Release Report August 2023.pdf

Nationally, the August 2023 update of G-NAF shows an increase of 45,413 addresses overall (+0.29%). The total number of addresses in G-NAF now stands at 15,530,436 of which 14,704,939 or 94.68% are principal.

The release notes also mention addresses being tagged as "retired", and some "complex" addresses with duplicate address_string. Maybe we should look into that as part of our transformation/cut-down.

lyricnz commented 10 months ago

Rebuilding the image locally, and running a single suburb seems to work fine. A bunch of small differences in coordinates

LukePrior commented 10 months ago

Cool, will run GHA