Harvest all geospatial items from DGA

[mwu2018]

The DGA is backed by magda and harvests very rich datasets from various public sources, including but not limited to ckan datasets. Currently the catalog group data.gov.au in the NationalMap only has very limited items common to DGA, which is because the NationalMap only harvests ckan datasets with a filter.

The NationalMap should harvest all geospatial datasets from DGA.

Technical Note

Search all organisations in DGA

https://data.gov.au/api/v0/search/organisations?limit=1000

{
  "hitCount": 883,
  "organisations": [
    {
      "acronym": [
        "A"
      ],
      "addrCountry": "Australia",
      "addrPostCode": "7001",
      "addrState": "Tasmania",
      "addrStreet": "School of Geography and Environmental Studies",
      "addrSuburb": "Hobart",
      "aggKeywords": "aadc",
      "datasetCount": 1604,
      "email": "ndholmes@utas.edu.au",
      "identifier": "org-aodn-AADC",
      "name": "AADC",
      "phone": "+61 3 6226 2463",
      "source": {
        "id": "aodn",
        "name": "Australian Oceans Data Network"
      },
      "website": "http://data.aad.gov.au"
    },
    ...
  ]
}

Search datasets with specified formats within an organisation

https://data.gov.au/api/v0/search/datasets?publisher=AADC&format=CSV&format=geojson&format=kml&format=kmz&format=wms&format=wfs&publishingState=published

Challenge

Currently a digital twin assumes that a catalog item from magda record will have terria aspect that is created by minion-terria. However, there is no minion-terria in DGA therefore no terria aspect available. When harvesting data from DGA, the properties similar to what are defined in terria aspect need to be created dynamically.

[mwu2018]

When https://github.com/TerriaJS/terriajs/issues/5220 is resolved, more datasets can be correctly added to the map.

[AnaBelgun]

Ana to organise a meeting on Search epic and start breaking down the ticket

[wibily]

ana: please check whether v8 still implements the filters for DGA mwu: If there is no limit from DGA, how do we implement that into a catalog for NatMap?

[wibily]

we can't use pagination for huge number of items as our current pattern for catalog explorer is through an expander so we need a UX redesign