Filter datasets harvested by custom fields

[t83714]

Filter data via custom fields

Some data portal might create their own customised field through CKAN customisation.

e.g. this dataset https://catalogue.data.wa.gov.au/api/3/action/package_show?id=mining-tenements-dmirs-003 contains a custom field access_level, which we might need to filter data by.

This ticket is about creating a new feature to filter datasets harvested by custom fields.

Possible solution:

• Option 1: This generic proposal might solve this: https://github.com/magda-io/magda/issues/3100
• Option 2: alternatively, we can work out a quick solution by filtering by ckan API query (needs to test if it's supports that custom field)

Update

We adopted solution option 1: https://github.com/magda-io/magda/issues/3100

[t83714]

closed via PR: https://github.com/magda-io/magda-ckan-connector/pull/13

Tested with config like:

connector-wa:
  config:
    id: wa
    name: Western Australia Government
    sourceUrl: https://catalogue.data.wa.gov.au/
    pageSize: 25
    customJsFilterCode: |
      if(type === "Dataset") {
        return jsonData["access_level"]=="open" ? true : false;
      } else {
        return true;
      }