Magento product search in admin product grid not working correctly. No OR functionality

cptX commented 5 months ago

Preconditions and environment

Magento version 2.4.5 and 2.4.6
Anything else that would help a developer reproduce the bug In admin product list let's try the following: Let's say we have a product called "Erika Running Short". If we search with full name the product appears in the results. If we search for "Erika Running" the product appears. But if we search for "Erika Short" no product appears.

In my everyday administering I search for product names or titles and I cannot remember all the words in between. So I need to put 2 words included in the title and expecting to get the corresponding products. Now I get 0 products in the product list search results when I don't put all words in between the wanted keywords.

Steps to reproduce

Open a magento instance (2.4.5 or 2.4.6) which has some products. Let's say it has the default sample products magento 2.4.5 and 2.4.6 have.
Go to Catalog > Products
Search for "Erika Running Short". There will be several results
Search for "Erika Short". There will be no result!

Expected result

When searching for "Erika Short" I want the system to return all the products that include these 2 words in their title independently of the position these 2 words are placed in the products titles. For example If a product has the title "Erika Running Short", it should appear when searching for "Erika Short"

Actual result

Now when searching for "Erika Short" no products appear, although there are many products available with the title "Erika Running Short"

Additional information

No response

Release note

No response

Triage and priority

[x] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
[ ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
[ ] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
[ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
[ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

m2-assistant[bot] commented 5 months ago

Hi @cptX. Thank you for your report. To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release
For more details, review the Magento Contributor Assistant documentation.
Add a comment to assign the issue: @magento I am working on this
To learn more about issue processing workflow, refer to the Code Contributions.

Join Magento Community Engineering Slack and ask your questions in #github channel. :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting. :clock10: You can find the schedule on the Magento Community Calendar page. :telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

m2-assistant[bot] commented 5 months ago

Hi @engcom-Dash. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
1. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
1. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
1. Verify that the issue is reproducible on 2.4-develop branch
  Details
  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

Join Magento Community Engineering Slack and ask your questions in #github channel.

drinkingsouls commented 5 months ago

I agree, the search functionality from the Admin Product Grid is annoyingly limited. We've had the same issue since launching Magento 2. The frontend search with Elasticsearch more helpful, I wonder why this was never integrated into the Admin side.

engcom-Dash commented 5 months ago

Hi @cptX

Thanks for reproting and collaboration.

Verified the issue in magento 2.4 dev instance and the issue is reproducible.

Steps to reproduce

Open a magento instance (2.4.5 or 2.4.6) which has some products.
Let's say it has the default sample products magento 2.4.5 and 2.4.6 have.
Go to Catalog > Products
Search for "Erika Running Short". There will be several results
Search for "Erika Short". There will be no result!

Magento product search in admin product grid not working correctly. We are not seeing results when searching with "Erika Short".

Hence, confirming the issue.

Please refer the attached screenshots.

github-jira-sync-bot commented 5 months ago

:white_check_mark: Jira issue https://jira.corp.adobe.com/browse/AC-11757 is successfully created for this GitHub issue.

m2-assistant[bot] commented 5 months ago

:white_check_mark: Confirmed by @engcom-Dash. Thank you for verifying the issue.
Issue Available: @engcom-Dash, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

digitalrisedorset commented 5 months ago

@magento I am working on this

digitalrisedorset commented 5 months ago

I have now committed some changes that makes the product 'Erika Running Short' appearing when using the full text search term 'Erika short'. I am yet to add tests, documentation. Although we are seeing the product as expected, I am still trying to understand how to handle the elasticsearch boost parameter to ensure the results are more sorted more intuitively

cptX commented 5 months ago

Hi @digitalrisedorset, can you please share the code you modified? Also, How is elasticsearch related to this search? Are you trying to use elasticsearch for the admin instead of the embedded mysql search? This would be excellent if true!

digitalrisedorset commented 5 months ago

@cptX https://github.com/magento/magento2/pull/38630 --> yes, there was a hook already in place but this hook was using Magento database, I swapped it to use elasticsearch now

cptX commented 4 months ago

Any progress on this? Are there any practical specific steps to follow in order to have some enhanced functionality, either by the standard mysql search or by using elasticsearch?

digitalrisedorset commented 4 months ago

@cptX is the question for me? Apologies, I have joined the project recently and I may not have performed the steps you'd expect by now. The situation with this work is: I have committed some changes and made a PR from them. This PR is now awaiting to be reviewed. The changes do consist in taking over the backend product full text feature and make use of elasticsearch instead.

digitalrisedorset commented 4 months ago

the PR is https://github.com/magento/magento2/pull/38634

cptX commented 4 months ago

Hi @digitalrisedorset, can there be any implications from your modification? Do you believe that this would be a better solution than just enabling some OR functionality? I'm afraid that with elasticsearch the results could be more broad than needed and products that are not needed could come to surface. If I want just to enable OR functionality in the existing mysql search what should I change?

digitalrisedorset commented 4 months ago

@cptX thanks for this feedback. Using elasticsearch has the benefits to reuse the elasticsearch built-in engine strategies that bring the products that are relevant to the keywords in the full text search: this is the same use experience as the frontend. Using mysql and making an OR search would imply to start all over again with different changes.

I was hoping we should first deploy and test what we have right now to make a judgement of UX, relevance and performance

digitalrisedorset commented 4 months ago

@cptX I have had more thoughts on how to handle this task. I will put a summary of these thoughts so that we can decide how we can move on before making any additional changes. Ideally, I’d like a confirmation of our next move by more than 2 people so that we are assured we perform some changes for which we understand the solution before its implementation

- the admin uses Mysql to retrieve the products for the Catalog/Product admin filters: This is what we currently have and I have made some comments below on this particular context:

the admin search currently uses mysql and no OR search is implemented
if we add an OR search, it may bring results that are less relevant than what the search results returns at this time unless we implement a relevance algorithm as well. Ideally, we'd need an OR search that also scores the matches with 3 words better than the matches with 2 words and so on (to illustrate with the example you provide, the search 'Erika Short' would need to score better the products that make use of both keywords than the other products that make use of either 'Erika' or 'Short' but not both

- the admin search uses Elasticsearch to enhance a search that has the relevance and the OR search built-in This is the solution I have come up with and I add the justifications for it below:

this method was chosen because it resolves 2 of the problems that are identified in this issue: 1. Erika Short returns results (meaning we are doing something similar to an OR search), 2. ElasticSearch will score the matches better if the products make use of more of the search keywords
the problem that ElasticSearch has is that is it originally built for the frontend. As a consequence, Elasticsearch does not return any invisible products. Yet, the admin user may want to search for these products. To address this issue, I have currently created a post mysql process that pulls the simple products for the matches that are configurable
The ElasticSearch implementation does not take over the mysql implementation fully. Instead, what the ElasticSearch implementation performs is a search for matches for the fulltext keywords and adds the simple products if relevant. This search does not disabled the search using the other filters (sku, product name, store, visibility..). For this reason, this implementation can return the invisible products in the catalog as long as it is not done whilst a fulltext search occurs

Conclusion: If we simply choose to add an OR search and do not have relevance, we will have some results with less relevance than now at the top of the product grid. If we use ElasticSearch, we may have the concern that we use a solution that is not designed to be used for the backend. Both solutions have pros and cons and we should decide on the next step once we have confirmed these pros and cons are all understood and once we have a final idea on how to finalise this solution.

digitalrisedorset commented 4 months ago

@magento run all tests

cptX commented 4 months ago

@digitalrisedorset thanks for spending time explaining the possibilities.

I'll describe my everyday needs as the eshop manager: Let's say I have many chargers for electric vehicles (check here https://electrons.gr/en/batteries/chargers) As you see the names are with this format:

Charger 54.6V 2A with 5.5x2.1 plug for lithium ion (Li-Ion) 13s 46.8V (48V) battery
Charger 54.6V 2A with GX16-3 plug for lithium ion (Li-Ion) 13s 46.8V (48V) battery
Charger 54.6V 2A with Τ-type / IEC13 plug for lithium ion (Li-Ion) 13s 46.8V (48V) battery

If we enable the OR functionality I can very easily type "charger 54.6V GX16" for example to find the second item. Without OR functionality I have to type ALL words up to GX16 until I have the correct item returned, in our case "Charger 54.6V 2A with GX16"! As you understand it's impossible to remember all the correct words that exist between the important keywords. So almost all my searches fail with no results. Also, if I know that the system works with OR I can use my search keywords wisely and put only the keywords that exist in the specific item I want to find, so I don't think any irrelevant items will come out. Anyway more irrelevant items are better than no returned items.

So in any case I'm pretty confident the OR functionality is absolutely necessary. Now regarding irrelevant items, of course taking advantage of the elasticsearch power would be great but then I'm afraid that this indeed will bring irrelevant items. Specifically I want you to clarify if the returned items will follow exactly the same search methodology as in frontend. For the frontend search I have in every product an extra field where I write keywords, and these keywords are searchable. If these keywords are also used in adminsearch then it will indeed return many more irrelevant products.

As a simpler and first step can you describe what can I do to enable just a simple OR functionality with the existing mysql implementation? At the moment I cannot do my required job. At least with the OR I will be able to do it.

digitalrisedorset commented 4 months ago

Hi @cptX , how about you give it a try to this solution that I have written. I suggest you download the file below, install it and enable it in your store and perform couple of tests. If it does not work, then we know right away that we can't go with this on Magento on a bigger scale.

If it works, we might have a temporary solution and we can mature the requirements for this issue so that it addresses your needs but in a way that is usable for all Magento owners and with an implementation that is not slowing down the backend either.

catalog-admin-search-main.zip

Disclaimer: the module is not ready for production. You may install it on a dev/test environment and perform a backup. If you believe the module is not performing well, you can disable the module, clear the cache and everything is back as normal: this module does not alter the database at all.

cptX commented 4 months ago

Hi @digitalrisedorset, can I install your module with composer? Do you have it in a repository or should I install it manually?

cptX commented 4 months ago

I have installed it and when I go to admin > products I get this error "Something went wrong with processing the default view and we have restored the filter to its original state."

digitalrisedorset commented 4 months ago

Ok thanks for the update and for trying the update. I would put the site in developer mode, then, clean the cache, clear the generated folder and then try bin/magento setup:upgrade before doing anything in the backend. If all the steps before are ok, then the backend should be fine too. Once the backend is ok, then you can also put the site in production mode or however it was before the module was installed

digitalrisedorset commented 4 months ago

@cptX I made a rehearsal of the deployment of the zip file I have sent you. Before I made the deployment of my module, I verified the admin catalog search does not return any match for the full text search with 'Erika short' (my site has the sample data installed)

I have created a folder under app/code called Mbs and then a folder CatalogAdminSearch under Mbs. Within this folder, I have added all the files from the extracted zip I then ran bin/magento setup:upgrade I verified the module was enabled by running bin/magento module:status

Finally, I tried the query again and verified some results appear. More importantly, I validated the product 'Erika short' was found was expected

magento / magento2