Index suggested items that are approved

[dpancic]

In GitLab by @KlausIllmayer on Feb 17, 2022, 13:31

Currently it seems that if an item is suggested by a contributor and then afterwards approved by a moderator, this item is not added into the Solr search index. At least the item does not show up in the autocomplete search but it shows up in the search result list on frontend. Try it out with "nakala": https://marketplace.sshopencloud.eu/search?q=nakala (will not show up in autocomplete).

@tparkola Can you have a look and add it to the search index.

(adding @egray523 as watcher as he discovered the issue)

[dpancic]

In GitLab by @tparkola on Feb 22, 2022, 22:39

In order for item show in autocomplete is whole autocomplete-reindex, but automatic autoreindex will work as e.g. items index, which will slow the app. I suggest autocomplete-index once in a period of time.

[dpancic]

In GitLab by @KlausIllmayer on Mar 10, 2022, 10:11

I now remember that there is a cronjob that runs once in the night (CET) that will do the autocomplete-reindex. But that means that depending on the time when an item is approved, it will take some hours until the item shows up in the autocomplete. This could be irritating for users but I guess we will have to live with it. Maybe we can give somewhere an information about this behaviour?

Comments? @egray523 @laureD19 @vronk

[dpancic]

In GitLab by @vronk on Mar 21, 2022, 13:46

So for me to understand:
autocomplete-index is a separate solr-index?
and you cannot add just the one new term to the index on the fly?

If that is the case, then we have to live with overnight-update. But I am surprised, because to my knowledge normally it it possible to add a new term anytime, without much overhead.

@tparkola, would you care to comment?

[dpancic]

In GitLab by @tparkola on Mar 21, 2022, 15:46

For suggestion special component with dedicated field are used. Our suggest param has been set to not build/indexed on start or perfomed commit of items:

<str name="buildOnStartup">false</str>
<str name="buildOnCommit">false</str>

therefore autocomplete reindex must be performed 'manually'.

[dpancic]

In GitLab by @tparkola on Apr 5, 2022, 10:45

The configuration of solr was changed to buildOnCommit - true, time did not increase exponentially. Fixes have been merged to develop.

[dpancic]

In GitLab by @KlausIllmayer on Jun 28, 2022, 16:49

Trying out if this works but the problem still occurs (I tried it out on the stage instance). If proposing an item as a contributor and approving it as a moderator, it still does not show up in the autocomplete endpoint. Only after manually starting the autocomplete-reindex it will show up (which happens every night so it is not the biggest issue, but as explained in the description, someone could expect to see it immediately - otherwise, it is more an issue for moderators, as contributors are not aware when an item is approved).

[dpancic]

In GitLab by @KlausIllmayer on Jun 30, 2022, 13:57

Problem was that it was necessary to restart the Solr container with a plain config, so that the changes are applied. After doing this the buildOnCommit was set and did work. But we then discovered, that it does not differ between the states of an item. Therefore a suggested item will also be indexed and show up for everyone as it is only expected for approved ones. Respecting the state of an item is a more extensive task. Therefore we decided to revert the changes and keep it, as it was before, as the favored behavior has only low priority. This due to the fact, that it anyway takes some time until a moderator approves such a proposed item and the time until the daily re-index is running in the night does hopefully not matter too much. But it is important that moderators are aware about this delay. @laureD19 should we add such information in a kind of MP primer document for moderators?

@tparkola can you revert the changes so that buildOnCommit is false again?

[dpancic]

In GitLab by @tparkola on Jun 30, 2022, 14:53

Done :) Please proceed with review, etc.

[dpancic]

In GitLab by @KlausIllmayer on Jul 4, 2022, 16:14

I had some problems with applying the change on our local server but in the meantime it works as before, the buildOnCommit is false. Remark: for applying the change the solr config files in the container (usually '/var/solr/data`) need to be deleted and the container to be restarted so that the new configuration will be deployed

[dpancic]

In GitLab by @KlausIllmayer on Sep 29, 2022, 17:21

We now found out, that an ingested item that is not approved, will show up in the autocomplete index. If not a moderator, clicking on this item in the autocomplete search will result in an empty search result page. @tparkola do you think, this is intended/a sideeffect of the current solution?

Notifying @laureD19

[dpancic]

In GitLab by @laureD19 on Sep 30, 2022, 12:49

the same behaviour also happens for "suggested" items not yet approved.

[dpancic]

In GitLab by @tparkola on Oct 5, 2022, 07:37

In my opinion it is not a side effect of the configuration. Explanation: after brief investigation I see that this is how the system works, i.e. when indexing an item its label goes to autocomplete index regardless of the item's state (approved or not, ingested or not, ...). And when autocomplete is requested there is no filtering based on state (only category is taken into consideration). In order to change that behaviour I would suggest to add a new issue, as this is not related to the current one.