Investigate changing how files are named.

[LukePrior]

I've been considering changing how files are named to avoid having to classify into states/territories and avoiding issues with duplicate names.

The current solution is just the suburb name in the corresponding state folder. Then the results file needs to contain several fields which are of minimal use.

Consider switching to just generating a unique identifier for each suburb or even just using an identifier from GNAF.

This could also be paired with the gzip compression to significantly reduce file size, would need to be paired with a slight rework of the front-end which is already on the cards anyway.

[lyricnz]

I'm not sure how changing the filename would help? Suburb names are unique within each state, aren't they? Agree on desire to remove redundant/derivable fields from results file - intending to do that in the refactor PR.

[LukePrior]

Main concern was as you note with the results file so if it's going to be addressed in PR it'll probably be fine. Only other concern is some of the characters used in suburb names but should be fine.

[LukePrior]

Ahh yep that was the issue I was envisioning with this, it'll be quite involved change

[lyricnz]

There are 134 regions with >1 postcode (though this might not be an error - maybe a region can span postcodes in some circumstaces?)

select locality_name || ',' || state suburb_state, postcode, count(*) into states from address_principals group by locality_name , state, postcode

select suburb_state, count(*) into duplicate_states from states group by suburb_state having count(*) > 1

select * from states where suburb_state in (select suburb_state from duplicate_states)

suburb_state	postcode	count
ALBION,QLD	4010	3444
ALBION,QLD	4822	41
ALISON,NSW	2259	48
ALISON,NSW	2420	83
ALLIGATOR CREEK,QLD	4740	471
ALLIGATOR CREEK,QLD	4816	744
ASCOT,QLD	4007	4183
ASCOT,QLD	4359	56
ASCOT,VIC	3364	55
ASCOT,VIC	3551	1091
BACK CREEK,NSW	2372	22
BACK CREEK,NSW	2390	9
BACK CREEK,NSW	2422	18
BACK CREEK,NSW	2484	25
BACK CREEK,NSW	2622	26
BACK CREEK,NSW	2671	41
BAKERS CREEK,NSW	2359	19
BAKERS CREEK,NSW	2422	28
BAKERS CREEK,NSW	2447	24
BALMORAL,NSW	2283	318
BALMORAL,NSW	2571	234
BARRY,NSW	2340	3
BARRY,NSW	2799	144
BELCONNEN,ACT	2615	29
BELCONNEN,ACT	2617	6973
BIG HILL,VIC	3231	44
BIG HILL,VIC	3555	125
BIRNAM,QLD	4285	53
BIRNAM,QLD	4352	43
BLACK CREEK,NSW	2439	33
BLACK CREEK,NSW	2729	7
BOAT HARBOUR,NSW	2316	638
BOAT HARBOUR,NSW	2480	70
BROADWATER,NSW	2472	502
BROADWATER,NSW	2549	81
BURRA,NSW	2620	398
BURRA,NSW	2653	26
CABBAGE TREE ISLAND,NSW	2430	12
CABBAGE TREE ISLAND,NSW	2477	4
CANNON CREEK,QLD	4310	85
CANNON CREEK,QLD	4380	46
CARRINGTON,NSW	2294	1474
CARRINGTON,NSW	2324	63
CATTLE CREEK,QLD	4407	14
CATTLE CREEK,QLD	4626	46
CEDAR CREEK,NSW	2325	42
CEDAR CREEK,NSW	2484	11
CEDAR CREEK,QLD	4207	403
CEDAR CREEK,QLD	4520	342
CLONTARF,QLD	4019	4968
CLONTARF,QLD	4357	47
COLO,NSW	2756	64
COLO,NSW	2795	14
DALWOOD,NSW	2335	49
DALWOOD,NSW	2477	108
DARLINGTON,NSW	2008	1766
DARLINGTON,NSW	2330	241
DEEP CREEK,NSW	2440	49
DEEP CREEK,NSW	2460	11
DEEP CREEK,NSW	2469	33
DOUGLAS,QLD	4354	98
DOUGLAS,QLD	4814	3183
ELDERSLIE,NSW	2335	65
ELDERSLIE,NSW	2570	3368
ELLERSLIE,NSW	2648	36
ELLERSLIE,NSW	2729	28
ELPHINSTONE,QLD	4361	79
ELPHINSTONE,QLD	4742	32
ENMORE,NSW	2042	2281
ENMORE,NSW	2350	63
FAIRY DELL,VIC	3561	43
FAIRY DELL,VIC	3875	3
GILBERTON,QLD	4208	39
GILBERTON,QLD	4871	10
GLENROY,NSW	2640	1767
GLENROY,NSW	2653	54
GOLDEN POINT,VIC	3350	1542
GOLDEN POINT,VIC	3451	88
GOLDEN POINT,VIC	3465	6
GOWRIE,NSW	2330	134
GOWRIE,NSW	2340	68
GREEN HILLS,NSW	2365	17
GREEN HILLS,NSW	2730	2
GREEN POINT,NSW	2251	3408
GREEN POINT,NSW	2428	333
GREENDALE,NSW	2550	49
GREENDALE,NSW	2745	172
GREENLANDS,NSW	2330	74
GREENLANDS,NSW	2631	27
GREENMOUNT,QLD	4359	394
GREENMOUNT,QLD	4751	269
GREGORY RIVER,QLD	4660	60
GREGORY RIVER,QLD	4800	256
HAPPY VALLEY,VIC	3360	55
HAPPY VALLEY,VIC	3549	36
HAWKER,ACT	2614	1458
HAWKER,ACT	2617	1
HIGHLAND PLAINS,QLD	4401	41
HIGHLAND PLAINS,QLD	4454	48
HILL TOP,NSW	2575	1220
HILL TOP,NSW	2628	69
HILLGROVE,NSW	2350	246
HILLGROVE,NSW	2650	22
HILLSBOROUGH,NSW	2290	264
HILLSBOROUGH,NSW	2320	23
HILLSIDE,VIC	3037	6301
HILLSIDE,VIC	3875	183
HUNTLEY,NSW	2530	32
HUNTLEY,NSW	2800	87
IRONPOT,QLD	4610	75
IRONPOT,QLD	4701	101
KILLARA,VIC	3312	15
KILLARA,VIC	3691	594
KINGS PLAINS,NSW	2360	12
KINGS PLAINS,NSW	2799	86
KINGSWOOD,NSW	2340	469
KINGSWOOD,NSW	2550	3
KINGSWOOD,NSW	2747	6180
LAMBS VALLEY,NSW	2335	171
LAMBS VALLEY,NSW	2370	61
LANSDOWNE,NSW	2163	110
LANSDOWNE,NSW	2430	395
LILLI PILLI,NSW	2229	532
LILLI PILLI,NSW	2536	537
LILYVALE,QLD	4352	35
LILYVALE,QLD	4723	42
LIMEBURNERS CREEK,NSW	2324	163
LIMEBURNERS CREEK,NSW	2444	44
LONG PLAIN,NSW	2360	26
LONG PLAIN,NSW	2629	1
LONG POINT,NSW	2330	27
LONG POINT,NSW	2564	90
LYNDHURST,NSW	2350	53
LYNDHURST,NSW	2797	285
MACKENZIE,QLD	4156	804
MACKENZIE,QLD	4702	32
MARYLAND,NSW	2287	3246
MARYLAND,NSW	4377	73
MARYVALE,QLD	4370	458
MARYVALE,QLD	4703	21
MAYFIELD,NSW	2304	5947
MAYFIELD,NSW	2540	8
MAYFIELD,NSW	2580	33
MAYFIELD,NSW	2787	6
MELBOURNE,VIC	3000	76966
MELBOURNE,VIC	3004	15116
MERRIJIG,VIC	3723	537
MERRIJIG,VIC	3875	5
MILROY,NSW	2380	71
MILROY,NSW	2850	35
MOGO,NSW	2536	374
MOGO,NSW	2850	26
MONTROSE,QLD	4370	2
MONTROSE,QLD	4413	54
MOONLIGHT FLAT,VIC	3450	52
MOONLIGHT FLAT,VIC	3465	69
MOUNT ARCHER,QLD	4514	171
MOUNT ARCHER,QLD	4701	66
MOUNT OLIVE,NSW	2330	95
MOUNT OLIVE,NSW	2787	34
MOUNT PLEASANT,QLD	4521	202
MOUNT PLEASANT,QLD	4740	2546
MYALL,VIC	3533	6
MYALL,VIC	3579	60
NELSON,NSW	2550	6
NELSON,NSW	2765	225
NEWTOWN,QLD	4305	703
NEWTOWN,QLD	4350	5916
NEWTOWN,VIC	3220	5970
NEWTOWN,VIC	3351	93
NORTH BRANCH,QLD	4356	72
NORTH BRANCH,QLD	4370	36
OXLEY,NSW	2711	25
OXLEY,NSW	2824	9
PADDYS FLAT,NSW	2469	7
PADDYS FLAT,NSW	2632	7
PADDYS RIVER,NSW	2577	57
PADDYS RIVER,NSW	2653	22
PALING YARDS,NSW	2580	24
PALING YARDS,NSW	2795	19
PRESTON,QLD	4352	321
PRESTON,QLD	4800	240
PRETTY BEACH,NSW	2257	299
PRETTY BEACH,NSW	2539	5
PUNCHBOWL,NSW	2196	8636
PUNCHBOWL,NSW	2460	7
RANGEMORE,QLD	4352	14
RANGEMORE,QLD	4806	7
RED HILL,NSW	2347	20
RED HILL,NSW	2720	3
RED HILL,NSW	2824	16
RED HILL,QLD	4059	3268
RED HILL,QLD	4413	98
RICHMOND,QLD	4740	551
RICHMOND,QLD	4822	750
ROCKY CREEK,NSW	2371	22
ROCKY CREEK,NSW	2390	28
ROCKY POINT,QLD	4873	102
ROCKY POINT,QLD	4874	1082
ROCKY RIVER,NSW	2358	167
ROCKY RIVER,NSW	2372	89
ROSE VALLEY,NSW	2534	41
ROSE VALLEY,NSW	2630	24
ROSEWOOD,NSW	2446	102
ROSEWOOD,NSW	2652	135
SANDY CREEK,NSW	2338	21
SANDY CREEK,NSW	2835	39
SCRUBBY CREEK,QLD	4478	10
SCRUBBY CREEK,QLD	4570	47
SHERWOOD,NSW	2440	65
SHERWOOD,NSW	2450	39
SHERWOOD,NSW	2474	24
SILVERWATER,NSW	2128	2161
SILVERWATER,NSW	2264	153
SMITHS CREEK,NSW	2460	18
SMITHS CREEK,NSW	2474	39
SMITHS CREEK,NSW	2484	158
SOUTH ARM,NSW	2449	171
SOUTH ARM,NSW	2460	109
SPRING CREEK,QLD	4343	150
SPRING CREEK,QLD	4361	225
SPRING CREEK,QLD	4420	22
SPRING HILL,NSW	2500	35
SPRING HILL,NSW	2800	258
SPRINGFIELD,NSW	2250	1947
SPRINGFIELD,NSW	2630	25
SPRINGFIELD,QLD	4300	3150
SPRINGFIELD,QLD	4871	1
SPRINGFIELD,VIC	3434	102
SPRINGFIELD,VIC	3544	5
ST CLAIR,NSW	2330	15
ST CLAIR,NSW	2759	7244
ST HELENS,QLD	4356	25
ST HELENS,QLD	4650	130
STOCKYARD,QLD	4344	33
STOCKYARD,QLD	4703	41
STONEHENGE,QLD	4357	74
STONEHENGE,QLD	4730	121
STONY CREEK,NSW	2550	63
STONY CREEK,NSW	2850	34
STONY CREEK,VIC	3371	15
STONY CREEK,VIC	3957	211
SUGARLOAF,QLD	4380	318
SUGARLOAF,QLD	4800	167
SUMMER HILL,NSW	2130	4657
SUMMER HILL,NSW	2421	69
SUNDOWN,QLD	4382	4
SUNDOWN,QLD	4860	122
SWAN BAY,NSW	2324	335
SWAN BAY,NSW	2471	153
TABLELANDS,QLD	4605	111
TABLELANDS,QLD	4680	11
THE BLUFF,QLD	4340	40
THE BLUFF,QLD	4355	17
THE GAP,NSW	2472	21
THE GAP,NSW	2650	90
THE GAP,QLD	4061	7086
THE GAP,QLD	4825	621
THE ROCKS,NSW	2000	1888
THE ROCKS,NSW	2795	42
TORRYBURN,NSW	2358	20
TORRYBURN,NSW	2421	89
WATTAMOLLA,NSW	2232	2
WATTAMOLLA,NSW	2535	59
WEST END,QLD	4101	10237
WEST END,QLD	4810	2958
WESTDALE,NSW	2340	1413
WESTDALE,NSW	2653	4
WHITE ROCK,QLD	4306	279
WHITE ROCK,QLD	4868	2404
WILLOW VALE,NSW	2534	38
WILLOW VALE,NSW	2575	368
WOODBURN,NSW	2472	535
WOODBURN,NSW	2538	42
WOODLANDS,NSW	2536	37
WOODLANDS,NSW	2575	130
WOODSTOCK,NSW	2360	17
WOODSTOCK,NSW	2538	114
WOODSTOCK,NSW	2793	665
YARRAS,NSW	2446	37
YARRAS,NSW	2795	11
YELLOW ROCK,NSW	2527	85
YELLOW ROCK,NSW	2777	433

[LukePrior]

Hmmm, it's interesting because some of these should probably be combined if adjacent while others like Mogo probably not...

[lyricnz]

Maybe there's more fields in the upstream DB that can be use to merge/split suburbs/postcodes.

[lyricnz]

It probably does mean we need a postcode in the data/files/map so that people select the "right" Mugo etc.

[lyricnz]

Also:

Some towns and suburbs have two postcodes—one for street deliveries and one for post office boxes. https://simple.wikipedia.org/wiki/Postcodes_in_Australia#Geography

[lyricnz]

It looks like both postcodes are being combined at the moment. And the postcode is shown when you click on a dot. So current behaviour isn't "wrong", just a bit unexpected.

eg https://lukeprior.github.io/nbn-upgrade-map/?suburb=belconnen&state=ACT

vs

❯ grep name results/ACT/belconnen.geojson | tr -d '",' | awk '{print $NF}' | sort | uniq -c
  29 2615
6973 2617