invalid byte sequence for encoding "UTF8": 0xc9 CONTEXT: COPY address_site, line 94109

minus34 / gnaf-loader

A quick way to get started with Geoscape's open GNAF & Admin Boundaries

Apache License 2.0

185 stars 66 forks source link

invalid byte sequence for encoding "UTF8": 0xc9 CONTEXT: COPY address_site, line 94109 #45

Closed andrewharvey closed 4 years ago

andrewharvey commented 4 years ago

I'll try to investigate but it looks like an issue with the source data, any one else run into the same issue?

root        : INFO     SQL FAILED! : COPY raw_gnaf_202002.address_site FROM '/tmp/work/gnaf/G-NAF/G-NAF FEBRUARY 2020/Standard/NSW_ADDRESS_SITE_psv.psv' DELIMITER '|' CSV HEADER; : invalid byte sequence for encoding "UTF8": 0xc9
CONTEXT:  COPY address_site, line 94109

minus34 commented 4 years ago

Yes, a manual edit is required on that line - it has an invalid character which I presume was meant to be an E with an accent on it.

The Admin Bdys also have issues with four shapefiles - where there are extra, invalid columns. Issue is with ANZLIC and the PSMA.

andrewharvey commented 4 years ago

Yes I just noticed that second one too

root        : WARNING   Importing QLD_LOCALITY_POLYGON_shp.shp - Couldn't run Shapefile SQL
shp2pgsql result was: column "qld_locali" of relation "aus_locality_polygon" does not exist
LINE 4: ..." ("lc_ply_pid","dt_create","dt_retire","loc_pid","qld_local...

minus34 commented 4 years ago

A quick edit to remove the offending fields does the job