Generate search index after catalog update

[mwu2018]

Currently the github action for search index generation is commented out.

[nf-s]

duplicate of https://github.com/TerriaJS/saas-catalogs-public/issues/118

[nf-s]

I generated a new catalog-index last night and there were 54224 datasets. This results in a ~40MB (uncompressed) JSON catalog index file. This is too large and makes the catalog index static file approach unsuitable.

The dramatic increase in number of datasets comes from data.gov.au - as there are some new organisations being harvested that have WMS services with over 1500 layers.

Options

• Continue with catalog-index + remove all data.gov.au datasets from the index (or from NatMap)
• Continue with catalog-index + remove problematic organisations from data.gov.au harvesting
  • Here we can continue with our blacklist approach (per org)
  • Or we can turn to a whitelist approach
• Turn off catalog-index and go back to client-side search - which basically doesn't work
• Create a server-side component to fix search
  • For example - add search endpoint to TerriaJS-server where it can handle much larger catalog-index file

[nf-s]

We now have a GitHub workflow in https://github.com/TerriaJS/saas-catalogs-public

1. Go to https://github.com/TerriaJS/saas-catalogs-public/actions/workflows/generate-catalog-index-pr.yml and press "Run Workflow"
2. Two PRs will be created - one for DEV and TEST - and another for PROD