Refactor the organisation index, views and templates to filter db records with elastic search results.

[Mike-Heneghan]

Currently, the ElasticSearch model is being used to display the results which are is good practice as it is not it's intended purpose. Instead, only fields which are searched against should be in the elastic search model. The hits from the search should be used to filter DB records to ensure the newest information is displayed.

Organisation.objects.filter(id__in=[])

ids = { x.id for x in result} Organisation.objects.filter(id__in=ids)

[Mike-Heneghan]

Updated so that the content displayed is from the db rather than the elastic search model.

It appeared that pagination was not working so the TemplateView was refactored as a ListView. This appeared not to be successful until the paginate_by was reduced to 5 which showed it was in fact working but the number of results was lower than expected.

Number of DB records:

>>> Organisation.objects.all().count()
25

Number of ElasticSearch responses:

>>> queryset = Search(index='organisation_search', doc_type='organisation').count()
>>> queryset
25

Number of returned orgs after it is executed:

>>> queryset = len(Search(index='organisation_search', doc_type='organisation').execute())
>>> queryset
10

[Mike-Heneghan]

Resolved the issue of the limited results by removing the .execute() call although I'm unsure when I introduced that in the first place. Also resolved an issue with the location and service partials not formatting correctly.

[Mike-Heneghan]

To do:

• [x] Remove the unnecessary fields from the Organisation search mapping.
• [x] Implement the filter by claimed using the DB records.
• [x] Find the solution for 10 hit max limit on ES search results.
• [x] Find ES best practice method for returning all IDs within a queryset. (handled by the keyword_order method)
• [x] Filter by has services using DB query.

[Mike-Heneghan]

From a comment online on ES

es_search = Search(...)
es_search = es_search.extra(_source={"includes": ["id"]})
res = es_search.execute()

[Mike-Heneghan]

At the moment returning to an old commit which displayed the ES results until furhter investigation is done.

The branch from the old commit is: https://github.com/Mike-Heneghan/ALISS/tree/enhance/organisation-search-es-display

Updated the index so that the services fields are removed and hence they do not appear in the search results.

Also rmeoved the filter for has services by commenting it out in the template.

[Mike-Heneghan]

For adding the related services function to avoid hitting the db again try to prefetch the data.

https://docs.djangoproject.com/en/1.11/ref/models/querysets/#prefetch-related or selected related.

[Mike-Heneghan]

Check that only published results appear for users who aren't logged in or have basic permissions.

[Mike-Heneghan]

When requesting records for services related to the organisations returned from the search to reduce trips to the db need to look into:

Prefetch related:

• See https://docs.djangoproject.com/en/1.11/ref/models/querysets/#prefetch-related

or

Select Related:

• From the docs it sounds like selected_related might not be ideal for our purposes as "select_related is limited to single-valued relationships - foreign key and one-to-one".

[Mike-Heneghan]

Checked out the enhance/organisation-search-view branch which was using the results from the es search to query the db.

Need to:

• [x] Update the branch with the recent changes from the master branch.
• [x] Integrate the new code for processing the es results for an ordered db query.
• [x] Find how to prefetch the related services for organisations which have matched the search. (unsure whether it's working or not.)
• [x] Ensure results are working as intended.
• [x] Ensure the filter methods are working as intended.

[Mike-Heneghan]

Means of filtering on number of services using QuerySet api.

orgs = Organisation.objects.annotate(num_services=Count('services')).filter(num_services__gt = 0)

[Mike-Heneghan]

Merged into master