search

archivematica / Issues

Issues repository for the Archivematica project
GNU Affero General Public License v3.0
16 stars 1 forks source link

Enhancement: Improve Elasticsearch setup #404

Open jraddaoui opened 5 years ago

jraddaoui commented 5 years ago

Please describe the problem you'd like to be solved.

From the Elasticsearch support upgrade, a few enhancements came out that could not be addressed in that period:

  1. Improve ES hosts declaration:

To allow setting a list of hosts separated by comma.

  1. Modify elasticSearchFunctions to a class:

To avoid using global variables and passing the client around. The class could be initiated in wsgi.py like it's being setup now.

  1. Improve mapping declaration:

We have dynamic indexes and a wild mapping in the AIP and AIP files indexes, with multiple elements from the METS file. Most of those fields are never used and probably shouldn't be indexed. Also, having strict indexes gives more control over what can be found in the documents.

  1. Normalize field names to snake_case in all indexes:

Field names are quite a mess. I didn't want to change this as external tools may be using the indexes but, since the indexes have changed anyway to be single document indexes, it may be a good moment to do so.

  1. Remove ES dependency in dashboard:

Only used to wrap some ES errors, but I think we could find a way to do it via archivematicaCommon.elasticSearchFunctions or fully remove it.

  1. Avoid index check and creation on the fly:

We could force to run the rebuild indexes commands on the install/upgrade process to avoid checking and creating the indexes on the fly. The check is a HEAD request, so it's not a huge deal, but I think we should not do it.

  1. Remove cluster health and index retry systems:

Trust timeout setting and let if fail.

  1. Use fixed ids in documents:

Use the UUID of each resource as the ES document _id to avoid making queries to get the document ids for deletes and updates.

  1. Improve backlog and archival storage searches:

Multiple requests are made on the views. This will require several changes and may not be doable in the time remaining but we should try to not do an extra count request (required by the pager system) and a query over the files indexes while querying later over the aips and transfers index to show the packages instead fo the files but make the query over the files.

  1. Improve boolean query generation:

To properly represent the user input in the search form. Explained in here.

  1. Allow to configure the index names:

This would allow to create an ES cluster that can be used by more than one Archivematica instance.

Describe the solution you'd like to see implemented.

Some of them are included above but this is open to discussion.

Describe alternatives you've considered.

Additional context

For Artefactual use: Please make sure these steps are taken before moving this issue from Review to Verified in Waffle:

scollazo commented 5 years ago

It would be also interesting to be able to set a custom name for the indexes

This would allow to create an ES cluster that can be used by more than one Archivematica instance.