search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

AODN_NSW_OEH harvest failing on geometry check #517

Closed mhidas closed 1 year ago

mhidas commented 1 year ago

All recent uplooads to the AODN_NSW_OEH pipeline have been failing on the harvest with an error like this:

Exception in component tPostgresqlOutput_1 (HarvestSinglebeamShapefile)
java.sql.SQLException: Batch entry 0 INSERT INTO "aodn_nsw_oeh"."singlebeam_survey" ("file_id","location","survey_date","source","source_xyz","area","category","est_no","point_num","point_dens","comments","loca>
caused by: ERROR: new row for relation "singlebeam_survey" violates check constraint "singlebeam_survey_geom_check"
  Detail: Failing row contains (466, 3961, SussexInlet, 2021-04-27, , NSWOEH_20210427_SussexInlet_STAX, 836909.021634000004, Estuarine, 113, 0, 0, , Sussex Inlet, 0106000020E610000001000000010300000003000000830>
        at aodn_nsw_oeh.harvestsinglebeamshapefile_0_1.HarvestSinglebeamShapefile.sShapefileInput_1_GIProcess(HarvestSinglebeamShapefile.java:6187)
        at aodn_nsw_oeh.harvestsinglebeamshapefile_0_1.HarvestSinglebeamShapefile.tPostgresqlRow_1Process(HarvestSinglebeamShapefile.java:1416)
        at aodn_nsw_oeh.harvestsinglebeamshapefile_0_1.HarvestSinglebeamShapefile.tPostgresqlConnection_1Process(HarvestSinglebeamShapefile.java:1182)
        ...
mhidas commented 1 year ago

The data providers have tried fixing the geometry in the shapefiles included with these datasets. In some cases the updated version did pass the singlebeam_survey_geom_check in our database. In others it still fails.

My last email to the group (on Apr 24) was:

Sorry, I don't get any more specific details from the error. However, you might these documents helpful: ST_IsValid is the function we use in our database to check the geometry, which returning false in these cases. Definition of valid geometry

I don't think I can do much more than that. I haven't heard from them since then.

bpasquer commented 1 year ago

@mhidas what is the status of this issue. Did the providers manage to fix all their files? Can we close?

mhidas commented 1 year ago

I still haven't heard any more from NSW OEH (or whatever they're called now...), and they haven't uploaded any further files. I've been going on the assumption that even if there is still an issue, it's low priority. I think we can close it.