Addresses missing from map

[lyricnz]

Manually generated the map output for Somerville, VIC per #33

FWIW a bunch of addresses seem to be missing from the map (those on "Eramosa Road East" and "Eramosa Road West" for example). Possible mishandling of street_suffix in DB?

SELECT * FROM gnaf_202302.address_principals WHERE locality_name = 'SOMERVILLE' AND state = 'VIC' and address like '%ERAMOSA ROAD %' LIMIT 100000

[lyricnz]

For that suburb, 946 of 5957 failed to lookup the locID

[a for a in addresses if 'locID' not in a]

[LukePrior]

Hmmmm I guess a better way to extract the address in a format that NBN likes from the database is required.

[lyricnz]

Example '12 ERAMOSA ROAD EAST SOMERVILLE 3912'

'https://places.nbnco.net.au/places/v1/autocomplete?query=12%20ERAMOSA%20ROAD%20EAST%20SOMERVILLE%203912'

{'source': 'google',
 'suggestions': [{'formattedAddress': '12 Eramosa Road East, Somerville VIC '
                                      '3912, Australia',
                  'id': 'EjQxMiBFcmFtb3NhIFJvYWQgRWFzdCwgU29tZXJ2aWxsZSBWSUMgMzkxMiwgQXVzdHJhbGlhIjASLgoUChIJ183ot0vg1WoR-v6VW4q_f5QQDCoUChIJtYUilcbh1WoRAmUGlVGFYnE',
                  'zoom': 16},
                 {'formattedAddress': 'Hearing Australia Somerville, The '
                                      'Natural Health and Wellness Clinic, 12 '
                                      'Eramosa Road East, Somerville VIC 3912, '
                                      'Australia',
                  'id': 'ChIJpeO8uUvg1WoRuax9tXdALO0',
                  'zoom': 16}],
 'timestamp': 1684816721187}

Which isn't a loc_id

If you do it in the web interface, you end up (after two levels of suggestions) with 12 ERAMOSA RD E SOMERVILLE VIC 3912 Australia

(E vs East)

[lyricnz]

Looking at web traces, it looks like the rollout map does a proximity search based on a location it got earlier (I can't immediately see how) https://places.nbnco.net.au/places/v1/nearby?lat=-38.2240566&lng=145.1860325&source=website_rollout_map

[lyricnz]

Actually, it looks like the id returned by the lookup work with the detailUrl even when they don't start with LOC, they just don't seem to return altReasonCode...?

[lyricnz]

Removing that filter if not locID.startswith("LOC"): and putting some dummy value in altReasonCode makes all the addresses appear again, at least. Maybe it needs a different colour or something, to say there's some issue there...

[lyricnz]

I've got a possible fix for this. If we retrieve the details for the property, even if it doesn't start with "LOC", the result includes a new addressSplitDetails that we can use to resubmit the request?

{'address1': '102 Eramosa Rd E', 'locality': 'Somerville', 'postcode': '3912', 'state': 'VIC'}

[lyricnz]

That works for some addresses. In Somerville, VIC 576/946 were fixed by using the addressSplitDetails. The rest still failed :/

        if not locID.startswith("LOC"):
            # if we have the wrong locID, try again with the address from the API
            new_address = ' '.join(status['addressSplitDetails'].values())
            locID = cached_get_json(lookupUrl + urllib.parse.quote(new_address))["suggestions"][0]["id"]

[LukePrior]

May be relevant: https://github.com/LukePrior/nbn-upgrade-map/issues/47

[lyricnz]

Maybe from there, use the "nearby" search, and pick the first one? (if any) This might be a little unreliable, as it may hit the neighbour. Also, it looks like some place-names have been renamed, and the NBN database might be using the old name. For example

15 WORWONG AVENUE SOMERVILLE 3912 => 15 BLACKS CAMP RD SOMERVILLE VIC 3912 Australia

[lyricnz]

Any code fix would be helped by #36

[lyricnz]

Addresses that cannot be converted to LocID should be included on the map, but in a colour that doesn't say "no upgrade available"

[lyricnz]

PR https://github.com/lyricnz/nbn-upgrade-map/pull/2 shows previously missing nodes as grey/unknown.

[lyricnz]

https://github.com/LukePrior/nbn-upgrade-map/pull/64 attempts to resolve the missing addresses by using the address-split details from the first query.

'102 ERAMOSA ROAD EAST SOMERVILLE 3912' ==> ID 'ChIJeXgl98nh1WoRMpmNmD7KSGk'

Fetching that ID from NBN /places/v2/details returns

addressSplitDetails = {'address1': '102 Eramosa Rd E', 'locality': 'Somerville', 'postcode': '3912', 'state': 'VIC'}

(Note the "East" -> "E"). Resubmitting that to the first query returns a valid LocID of 'LOC000058033404'

Now the map looks like

[lyricnz]

I think #64 solves addresses "missing" from the map, though we're still not resolving a few (6.2% in Somerville, VIC) addresses to LOCID. Will raise separate issue for that.