Generate process to create JSON-LD & RDF from CSW service endpoint

[jmckenna]

Background

• many NODC/partners have catalogue services running that do not automatically generate/publish JSON-LD for each metadata record in their catalogue

• workarounds and extensions may exist, but not for all types of catalogue software

  • some partners were also providing CSV files, made by hand, that lists some of the metadata layers for their service

• since we know the (CSW) service endpoint for all partners, we can harvest the endpoint and develop a standalone process that partners can run locally (possibly nightly) to generate the JSON-LD for each record, and an RDF file containing all records

  Updates:

• standalone process exists now, that:

  • hits the catalogue endpoint live
  • parses all metadata records
  • pays particular attention to keywords (as the ISO standard defines 'theme', 'place', and 'discipline' keywords)
  • initial partners processed:
  • MASPAWIO (GeoNode)
    • harvester script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/maspawio-harvest.py
    • JSON-LD and RDF files: https://github.com/iodepo/odis-arch/tree/schema-dev-jm/code/notebooks/Exploration/data-maspawio
  • AMA [old African Marine Atlas Catalogue] (GeoNetwork)
    • harvester script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/ama-harvest.py
    • JSON-LD and RDF files: https://github.com/iodepo/odis-arch/tree/schema-dev-jm/code/notebooks/Exploration/data-ama
  • BCC [Benguela Current Convention] (GeoNode)
    • harvester script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/benguelacc-harvest.py
    • JSON-LD and RDF files: https://github.com/iodepo/odis-arch/tree/schema-dev-jm/code/notebooks/Exploration/data-benguelacc
  • Caribbean Marine Atlas (GeoNode)
    • harvester script: https://github.com/iodepo/odis-arch/blob/schema-dev-jm/code/notebooks/Exploration/caribbeanmarineatlas-harvest.py
    • JSON-LD and RDF files: https://github.com/iodepo/odis-arch/tree/schema-dev-jm/code/notebooks/Exploration/data-caribbeanmarineatlas

Statistics

MASPAWIO

Parsed 149 records
    134 exported to JSON-LD
    15 were invalid records

AMA

Parsed 596 records
    596 exported to JSON-LD
    0 were invalid records

BCC

Parsed 180 records
    180 exported to JSON-LD
    0 were invalid records

Caribbean Marine Atlas

Parsed 358 records
    358 exported to JSON-LD
    0 were invalid records

Validation

• I used the schema.org validator for the JSON-LD files

Discussion

• could be useful for some partners that cannot easily expose JSON-LD for all their layers
• data folder contains separate JSON-LD files, by the actual metadata ID (so partners can examine the related record)
• parsing the keywords need discussion (the 'place' keyword could be very important for a region search, for example)
• process leverages OWSLib (the same Python library that all of these catalogue software use deep underneath)
• I was running these in Jupyter Notebook as-is, but could generate a notebook file for each partner when needed

Lessons Learned

• DublinCore response from these catalogue endpoints failed (at the catalogue server, not local script here) for metadata records with special characters in the title/name of the record (even with names with exponents: 10²)
  • therefore these harvest scripts request the ISO metadata profile record

[jmckenna]

Updates:

• added the data url (as the ID) to the JSON-LD (arguably this is the most important part of the JSON-LD, as it can't be an arbitrary url, it must go to the actual record)
• this is actually the GeoNode or GeoNetwork front-end url
• parsed live in the ISO record result from the service
• I added this to the MASPAWIO harvester (ran on all 500+ results)
• validated through schema.org validator
• example JSON-LD (the important addition is the working link to the dataset: http://maspawio.net/layers/geonode%3Agazi_fishing_intensity

  {
    "@context": {
        "@vocab": "https://schema.org/",
        "geosparql": "http://www.opengis.net/ont/geosparql#"
    },
    "@id": "http://maspawio.net/layers/geonode%3Agazi_fishing_intensity",
    "@type": "Dataset",
    "geosparql:hasGeometry": {
        "@type": "http://www.opengis.net/ont/sf#Polygon",
        "geosparql:asWKT": {
            "@type": "geosparql:wktLiteral",
            "@value": "POLYGON((39.3 -4.649999999999997, 39.3 -4.19, 39.709999999999994 -4.19, 39.709999999999994 -4.649999999999997, 39.3 -4.649999999999997))"
        },
        "geosparql:crs": {
            "@id": "http://www.opengis.net/def/crs/OGC/1.3/CRS84"
        }
    },
    "description": "This data shows high-resolution maps on fishing behavior and target resources of artisanal fishers in Gazi Kenya. The resource was developed using participatory mapping of fishing grounds using a handheld GPS mounted on selected fishing vessels. To access this resource, please contact innocentke@gmail.com or jmbugua@cordioea.net",
    "keywords": "fishing, Africa, East Africa, Kenya",
    "name": "Fishing intensity Gazi",
    "url": "http://maspawio.net/layers/geonode%3Agazi_fishing_intensity"
  }

[jmckenna]

Closing as this has been working for a while..