Load data : Victorian Biodiversity Atlas

[peggynewman]

https://collections.ala.org.au/public/showDataResource/dr1097 Currently ~7m records

Data is sourced from Vic Gov Spatial Data Mart, have to log in and order the files to be generated for download https://discover.data.vic.gov.au/dataset?sort=score+desc%2C+metadata_modified+desc&q=victorian+biodiversity+atlas&organization=&groups=&res_format=

4 files:

• Victorian Biodiversity Atlas flora records (unrestricted) for sites with high spatial accuracy
• Victorian Biodiversity Atlas flora records (unrestricted) for sites with moderate to low spatial accuracy
• Victorian Biodiversity Atlas fauna records (unrestricted) for sites with high spatial accuracy
• Victorian Biodiversity Atlas fauna records (unrestricted) for sites with moderate to low spatial accuracy

Recommend:

• Combine the 4 files into 1 DwC, and populating datasetName with the file name.
• Extract and mapping should be rerunnable given these input files. Scripts should go into Databox GH repo.
• File is ESRI Shapefile - extract SHP using `ogr2ogr -f "ESRI Shapefile" xxx-flora.shp
• Current unique key is catalogNumber. Ensure key mapping from the source files matches dataset in production. Update this ticket with key information
• This data contains museum and data from other organisations that may bring substantial duplicates. Analyse, report in this ticket, propose appropriate filters

Other checks:

• Does the DR metadata need updating?
• Licensing all ok? Is it per record or dataset?
• Note current record count, source file record counts, final SOLR record counts

[charvolant]

Changes between data:

• Record counts: Flora 2402546, Fauna 2035097, Currently in ALA 6905244 This needs to be checked out.

Record mapping

• The catalog number in the old version seems to have had a spurious ".00" appended to it, possibly a result of converting what looked like a floating point number. Appending a ".00" to new record numbers finds the same record.
• The existing dataset has a QUADID_ETC field matches the QUADID_ETC value in the flora file.
• locationID = SITE_ID. However, there now appears to be more precision 8348 -> 384830
• eventID = SURVEY_ID
• taxonID = TAXONID. Appears to have been updated.
• scientificName = SCI_NAME
• vernacularName = COMM_NAME
• scientificNameAuthorship = AUTHORITY. Not in original load
• samplingProtocol = SURVEY_TYPE
• basisOfRecord = HumanObservation (??)
• identificationVerificationStatus = ??
• georeferenceVerificationStatus = ??
• individualCount = TOTAL_COUNT (Appears to have been floating-pointed in original) LOW_COUNT, HIGH_COUNT, COUNT_ACC = count accuracy
• eventDate = START_DATE (Convert to ISO8601 from yyyy/MM/dd)
• year = START_YEAR
• month = START_MONTH
• verbatimElevation = ALTITUDE (Also add minimumElevationInMeters and maximumElevationInMeters)
• site_accuracy_km = MAX_ACC_KM
• coordinateUncertaintyInMeters = MAX_ACC_KM * 1000 (Currently a decimal, convert to integer)
• locality = LOCN_DESC
• FFG_Act_description = FFG_DESC
• Victorian_Advistory_List_Description = VICADV_DESC
• EPBC_Description = EPBC_DESC
• establishmentMeans = ORIGIN
• recordedBy =
• decimalLatitude = LAT_DD94
• decimalLongitude = LONG_DD94
• dataset_version_date = VERS_DATE (Should be dcterms:modified?)
• site_modification_date =
• taxon_modification_date =
• geodeticDatum = GDA94 (check projection - .prj file says GEOGCS["GCS_GDA_1994",DATUM["D_GDA_1994",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]] convert to EPSG?)
• dataGeneralisations =
• COVERABUND = COVERABUND
• FIRE_RESP = FIRE_RESP
• VIC_LF = VIC_LF
• NVIS_GF = NVIS_GF
• PROJECT_ID = PROJECT_ID (appears to have been decimalled)

Available but not used

• VICADV, FFG, EPBC Codes that correspond to the description
• ECOLOGY
• SCALE_USE
• UFI
• UFI_DATE

[peggynewman]

The record count disparity is likely because you've only loaded 2 of the 4 files. I only put 2 up onto the server.

[peggynewman]

Worthy of discussion with VBA are the dataGeneralisations. The loads are divvied up according to accuracy which looks like some generalisation has been applied. I also think the file name should go into datasetName.

[charvolant]

With all 4 files, approx 825337 present in the ALA not in current dump and 1673260 new records. 7753164 total records, 6905243 in the ALA. 6905243 + 1673260 - 825337 = 7753166 (probably correct when accounting for accidental header inclusion). Still, 825337 is a pretty large number, so check VBA to ensure that they're deletes.