CirrusSearch "An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later."

[Daniel-Mietchen]

Describe the bug

Doing a search on the wiki like https://portal.mardi4nfdi.de/w/index.php?search=data&title=Special%3ASearch&profile=default&fulltext=1 gives an error

An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later. 

Expected behavior

Submitting a search term should invoke a search and yield results.

To Reproduce Steps to reproduce the behavior:

1. Go to the search box on any page on the wiki
2. Enter a term
3. Hit return
4. See error

Screenshots

Additional context

• This is a common error likely related to the indexing for CirrusSearch, for which various reasons and fixes are discussed in various places, e.g. here

Checklist for this issue:

• [x] Assignee has been set for this issue
• [x] All fields of the issue have been filled
• [x] the main MaRDI project has been assigned as project

[physikerwelt]

Thank you for the report. I'll have a look.

[physikerwelt]

Updating alone did not help. I tried

root@e1cfc771e0ee:/var/www/html/extensions/CirrusSearch/maintenance#  php updateSearchIndexConfig.php --reindexAndRemoveOk --indexIdentifier now
Could not open input file: updateSearchIndexConfig.php
root@e1cfc771e0ee:/var/www/html/extensions/CirrusSearch/maintenance# ls
CheckIndexes.php          CopySearchIndex.php  ExpectedIndices.php   IndexNamespaces.php  RunSearch.php  SaneitizeJobs.php            UpdateOneSearchIndexConfig.php  UpdateSuggesterIndex.php  elasticsearch-scripts
CirrusNeedsToBeBuilt.php  DumpIndex.php        ForceSearchIndex.php  Metastore.php        Saneitize.php  UpdateDYMIndexTemplates.php  UpdateSearchIndexConfig.php     UpdateWeightedTags.php
root@e1cfc771e0ee:/var/www/html/extensions/CirrusSearch/maintenance#  php UpdateSearchIndexConfig.php --reindexAndRemoveOk --indexIdentifier now

and got

Indexing namespaces...Elastica\Exception\ResponseException from line 178 of /var/www/html/vendor/ruflin/elastica/src/Transport/Http.php: Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;

[physikerwelt]

ok, we need to upgrade further. It seems the WMDE images are outdated

This mainly allows writes to be properly applied. It was used on the WMF infrastucture to allow a zero-downtime upgrade migration plan. It is strongly advised to upgrade to elasticsearch 7.10.2 rather than using this compatibility layer.

[physikerwelt]

es 7.10.2 is not findable in the official docker repo. Let's try elasticsearch:7.11.2

[physikerwelt]

Let's wait a bit... Elasticsearch seems to need time. When trying to run the update script I see

index: /mw_cirrus_metastore_first/_doc/namespace-my_wiki-123 caused index [mw_cirrus_metastore_first] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];

[physikerwelt]

Maybe the problem is what is says.. the disk is full

df -h Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf /dev/mapper/vg_salt-root 19G 5,1G 13G 29% / udev 16G 0 16G 0% /dev tmpfs 16G 180K 16G 1% /dev/shm tmpfs 3,2G 2,9M 3,2G 1% /run tmpfs 5,0M 0 5,0M 0% /run/lock /dev/sda3 463M 126M 309M 29% /boot /dev/mapper/vg_salt-docker 109G 28G 77G 27% /var/lib/docker /dev/mapper/vg_salt-data 7,6G 7,0G 193M 98% /srv tmpfs 3,2G 0 3,2G 0% /run/user/6834

[physikerwelt]

Can someone from ZIB log into the server and free some space?

[physikerwelt]

Can someone from ZIB log into the server and free some space?

I moved the backups to /

So this is resolved.

[physikerwelt]

Now trying to rerun the indexing says

| error    | Only Elasticsearch 7.10.x |                                      |
|          |  is supported.  Your vers |                                      |
|          | ion: 7.11.2. 

[physikerwelt]

A new error

        Creating index...
<Error, collected 1 message(s) on the way, no value set>
+----------+---------------------------+--------------------------------------+
| error    | Custom Analyzer [extract_ |                                      |
|          | wb_quantity] failed to fi |                                      |
|          | nd filter under name [ter |                                      |
|          | m_freq]                   |                                      |
+----------+---------------------------+---------------

[physikerwelt]

updating to the wmf version and recreating the index works. I tested search and see results (while not everything is indexed though) this will take a few more hours.