No result in autosuggest / and search result by name

[julienanquetil]

Preconditions

Magento Version : 2.4.3-p1

ElasticSuite Version :2.10.8.1

Environment : developer & production mode

Third party modules : try to disable all and fallback to luma theme

Steps to reproduce

1. add custom attribute (like supplier item number, text field) include in search
2. enrich a product with the supplier item number
3. reindex all
4. search for the product in frontend (by name and / or by attribute)

Expected result

1. be able to see the product in autosuggest

Actual result

1. I'm able to find the product by SKU, supplier item number but not by name (autosuggest empty by name)
2. if i put a generic term used in several products, no result on search page

we currently got a platform in 2.3.7 with the same configuration without any issue. We migrate to 2.4.3 and keep the same configuration for attribute (facet, %....) but 2.3 give result by name, not 2.4

I checked on the backend the indices are "green" (try to delete them from ES and full reindex) I checked the mapping and attribute seem to be fine

I also checked the logs (exception, debug, system...) and nothing wrong there

here i use the word "cire" in seach, the log return this query (with debug mode on)

[2021-12-16 18:55:28] main.INFO: curl -XPOST 'https://XXXXXXXXX/AAAqa24_YYY_fr_catalog_product/_search?pretty=true' -d '{"size":0,"query":{"bool":{"filter":{"terms":{"visibility":[3,4],"boost":1}},"must":{"bool":{"must":[],"must_not":[],"should":[{"bool":{"filter":{"multi_match":{"query":"cire","fields":["search^1","sku^1"],"minimum_should_match":"20%","tie_breaker":1,"boost":1,"type":"best_fields"}},"must":{"multi_match":{"query":"cire","fields":["search^1","name.standard^3","sku.reference^4","code_XXXstandard^5","code_XXXstandard^6","search.standard^10","search.whitespace^100","name.whitespace^30","sku.whitespace^40","code_XXXwhitespace^50","code_XXX.whitespace^60","name.sortable^60","sku.sortable^80"],"minimum_should_match":1,"tie_breaker":1.0,"boost":1,"type":"best_fields"}},"boost":1}}],"minimum_should_match":1,"boost":1}},"boost":1}},"aggregations":{"attribute_set_id":{"terms":{"field":"attribute_set_id","size":100000,"order":{"_count":"desc"}}},"indexed_attributes":{"terms":{"field":"indexed_attributes","size":100000,"order":{"_count":"desc"}}},"option_text_marque":{"terms":{"field":"option_text_marque.untouched","size":10,"order":{"_count":"desc"}}}},"track_total_hits":0}' [] []

I also checked and all needed plugins are here

[apache@instance1-m8-s50 log]$ curl -XGET 'https://XXX/_cat/plugins'
bb4aa8055682e973c26876490be9864b aes_cross_cluster                 7.7.0-aes.48
bb4aa8055682e973c26876490be9864b alerting                          7.7.0
bb4aa8055682e973c26876490be9864b analysis-icu                      7.7.0
bb4aa8055682e973c26876490be9864b analysis-ik                       7.7.0
bb4aa8055682e973c26876490be9864b analysis-kuromoji                 7.7.0
bb4aa8055682e973c26876490be9864b analysis-phonetic                 7.7.0
bb4aa8055682e973c26876490be9864b analysis-seunjeon                 x.x.x.x
bb4aa8055682e973c26876490be9864b analysis-smartcn                  7.7.0
bb4aa8055682e973c26876490be9864b analysis-stempel                  7.7.0
bb4aa8055682e973c26876490be9864b analysis-thaichub2                7.7.0
bb4aa8055682e973c26876490be9864b analysis-ukrainian                7.7.0
bb4aa8055682e973c26876490be9864b discovery-ec2                     7.7.0
bb4aa8055682e973c26876490be9864b elasticsearch-aes-iam             1.0.0
bb4aa8055682e973c26876490be9864b elasticsearch-aes-remote-reindex  1.0.0
bb4aa8055682e973c26876490be9864b elasticsearch-analysis-vietnamese 7.7.0
bb4aa8055682e973c26876490be9864b elasticsearch-jetty               2.2.0
bb4aa8055682e973c26876490be9864b ingest-attachment                 7.7.0
bb4aa8055682e973c26876490be9864b kraken                            unspecified
bb4aa8055682e973c26876490be9864b ltr                               1.2.1-es7.7.0
bb4aa8055682e973c26876490be9864b mapper-murmur3                    7.7.0
bb4aa8055682e973c26876490be9864b mapper-size                       7.7.0
bb4aa8055682e973c26876490be9864b opendistro-anomaly-detection      x.x.x.x
bb4aa8055682e973c26876490be9864b opendistro-job-scheduler          x.x.x.x
bb4aa8055682e973c26876490be9864b opendistro-knn                    x.x.x.x
bb4aa8055682e973c26876490be9864b opendistro_index_management       x.x.x.x
bb4aa8055682e973c26876490be9864b opendistro_security               x.x.x.x
bb4aa8055682e973c26876490be9864b opendistro_sql                    x.x.x.x
bb4aa8055682e973c26876490be9864b performance-analyzer              7.7.0
bb4aa8055682e973c26876490be9864b repository-s3                     7.7.0
bb4aa8055682e973c26876490be9864b sifi-remediation-plugin           7.7.0
9b11c60c7091d0e1affcd73e1ff6192c aes_cross_cluster                 7.7.0-aes.48
9b11c60c7091d0e1affcd73e1ff6192c alerting                          7.7.0
9b11c60c7091d0e1affcd73e1ff6192c analysis-icu                      7.7.0
9b11c60c7091d0e1affcd73e1ff6192c analysis-ik                       7.7.0
9b11c60c7091d0e1affcd73e1ff6192c analysis-kuromoji                 7.7.0
9b11c60c7091d0e1affcd73e1ff6192c analysis-phonetic                 7.7.0
9b11c60c7091d0e1affcd73e1ff6192c analysis-seunjeon                 x.x.x.x
9b11c60c7091d0e1affcd73e1ff6192c analysis-smartcn                  7.7.0
9b11c60c7091d0e1affcd73e1ff6192c analysis-stempel                  7.7.0
9b11c60c7091d0e1affcd73e1ff6192c analysis-thaichub2                7.7.0
9b11c60c7091d0e1affcd73e1ff6192c analysis-ukrainian                7.7.0
9b11c60c7091d0e1affcd73e1ff6192c discovery-ec2                     7.7.0
9b11c60c7091d0e1affcd73e1ff6192c elasticsearch-aes-iam             1.0.0
9b11c60c7091d0e1affcd73e1ff6192c elasticsearch-aes-remote-reindex  1.0.0
9b11c60c7091d0e1affcd73e1ff6192c elasticsearch-analysis-vietnamese 7.7.0
9b11c60c7091d0e1affcd73e1ff6192c elasticsearch-jetty               2.2.0
9b11c60c7091d0e1affcd73e1ff6192c ingest-attachment                 7.7.0
9b11c60c7091d0e1affcd73e1ff6192c kraken                            unspecified
9b11c60c7091d0e1affcd73e1ff6192c ltr                               1.2.1-es7.7.0
9b11c60c7091d0e1affcd73e1ff6192c mapper-murmur3                    7.7.0
9b11c60c7091d0e1affcd73e1ff6192c mapper-size                       7.7.0
9b11c60c7091d0e1affcd73e1ff6192c opendistro-anomaly-detection      x.x.x.x
9b11c60c7091d0e1affcd73e1ff6192c opendistro-job-scheduler          x.x.x.x
9b11c60c7091d0e1affcd73e1ff6192c opendistro-knn                    x.x.x.x
9b11c60c7091d0e1affcd73e1ff6192c opendistro_index_management       x.x.x.x
9b11c60c7091d0e1affcd73e1ff6192c opendistro_security               x.x.x.x
9b11c60c7091d0e1affcd73e1ff6192c opendistro_sql                    x.x.x.x
9b11c60c7091d0e1affcd73e1ff6192c performance-analyzer              7.7.0
9b11c60c7091d0e1affcd73e1ff6192c repository-s3                     7.7.0
9b11c60c7091d0e1affcd73e1ff6192c sifi-remediation-plugin           7.7.0

so now i'm a little bit lost on where to take a look :) thanks

[julienanquetil]

Additional note, i find that if i put the "exact" product name (like all characters, spaces....) i got a result If i downgrade the module to 2.10.5 everything work as expected.

[julienanquetil]

If i made a graphql query i got products, but if i use the search/ajax/suggest/?q=XXX (where XXX is the same search term used in graphql query) i didn't get any product results

I try to remove all custom modules from composer.json / setup upgrade / di compile / reindex without any luck...

@vahonc any idea / tips where to search to find my issue ?

Thanks

[vahonc]

@julienanquetil,

Have you noticed any errors in the Console/Dev tools of your browser when you try to perform a search?

BR, Vadym

[julienanquetil]

Hi, Nothing (except a favicon one but not related...)

[julienanquetil]

Also if i make a full reindex in 2.10.5, upgrade the module to 2.10.8.1 all search functionality (search result page + autocomplete) work as expected until i make a reindex.

[vahonc]

@julienanquetil,

I was not able to reproduce your issue on the fresh Magento EE 2.4.3 with default sample data and ES 2.10.8.1.

After adding a custom attribute and assigning it to the test product I was able to see my product in autosuggest during searching by name or attribute.

Could you provide the configuration of your custom attribute?

Also, are you sure that your test products do not have any problems with displaying on the front (visibility, qty, etc)?

BR, Vadym

[julienanquetil]

it's not a product attribute but the "regular" product name

For example if i take this items (visible on front)

If i search for "DM174" i got the product, but if i search for "Lampe A Photopolymeriser" i didn't get any result in autocomplete / search results page

but if i enter the whole product name (exact match like Lampe A Photopolymeriser O-Light II (WOODPECKER) 5397-178*) I got results

[julienanquetil]

Also if i run a manual query like

curl 'https://host:port/mega24_default_catalog_product/_search?q=lampe'

I end up with expected results results

[romainruaud]

Hi,

unfortunately, I'm not able to reproduce your issue. I created the exact same product as yours.

Here is my mapping for "name" :

"name": {
    "type": "text",
    "fields": {
        "shingle": {
            "type": "text",
            "analyzer": "shingle"
        },
        "sortable": {
            "type": "text",
            "analyzer": "sortable",
            "fielddata": true
        },
        "standard": {
            "type": "text",
            "analyzer": "standard"
        },
        "whitespace": {
            "type": "text",
            "analyzer": "whitespace"
        }
    },
    "copy_to": [
        "search",
        "spelling"
    ],
    "norms": false,
    "analyzer": "keyword"
},

And the query that is generated in the autocompletion box :

{
  "size": 20,
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    },
    {
      "entity_id": {
        "order": "desc",
        "missing": "_first",
        "unmapped_type": "keyword"
      }
    }
  ],
  "from": 0,
  "query": {
    "bool": {
      "filter": {
        "terms": {
          "visibility": [
            3,
            4
          ],
          "boost": 1
        }
      },
      "must": {
        "bool": {
          "filter": {
            "multi_match": {
              "query": "lampe",
              "fields": [
                "search^1",
                "sku^1"
              ],
              "minimum_should_match": "100%",
              "tie_breaker": 1,
              "boost": 1,
              "type": "best_fields"
            }
          },
          "must": {
            "multi_match": {
              "query": "lampe",
              "fields": [
                "search^1",
                "name.standard^5",
                "sku.reference^6",
                "search.whitespace^10",
                "name.whitespace^50",
                "sku.whitespace^60",
                "name.sortable^100",
                "sku.sortable^120"
              ],
              "minimum_should_match": 1,
              "tie_breaker": 1,
              "boost": 1,
              "type": "best_fields"
            }
          },
          "boost": 1
        }
      },
      "boost": 1
    }
  },
  "track_total_hits": 0
}

And the results :

"hits": { - 
    "total": { - 
      "value": 0,
      "relation": "gte"
    },
    "max_score": null,
    "hits": [ - 
      { - 
        "_index": "magento2_default_catalog_product_20220105_100002",
        "_type": "_doc",
        "_id": "1185",
        "_score": 159.47162,
        "_source": { - 
          "entity_id": "1185",
          "attribute_set_id": "4",
          "type_id": "simple",
          "sku": "DM174",
          "has_options": false,
          "required_options": false,
          "created_at": "2022-01-05 10:22:41",
          "updated_at": "2022-01-05 10:22:41",
          "row_id": "1185",
          "created_in": "1",
          "updated_in": "2147483647",
          "visibility": "4",
          "price": [ - 
            { - 
              "price": 100,
              "original_price": 100,
              "is_discount": false,
              "customer_group_id": 0,
              "tax_class_id": 2,
              "final_price": 100,
              "min_price": 100,
              "max_price": 100
            },
            { - 
              "price": 100,
              "original_price": 100,
              "is_discount": false,
              "customer_group_id": 1,
              "tax_class_id": 2,
              "final_price": 100,
              "min_price": 100,
              "max_price": 100
            },
            { - 
              "price": 100,
              "original_price": 100,
              "is_discount": false,
              "customer_group_id": 2,
              "tax_class_id": 2,
              "final_price": 100,
              "min_price": 100,
              "max_price": 100
            },
            { - 
              "price": 100,
              "original_price": 100,
              "is_discount": false,
              "customer_group_id": 3,
              "tax_class_id": 2,
              "final_price": 100,
              "min_price": 100,
              "max_price": 100
            }
          ],
          "indexed_attributes": [ - 
            "price",
            "name",
            "url_key",
            "status",
            "tax_class_id"
          ],
          "category": [ - 
            { - 
              "category_id": 2,
              "is_virtual": "false",
              "category_uid": "Mg=="
            },
            { - 
              "category_id": 3,
              "is_virtual": "false",
              "name": "Accessories",
              "category_uid": "Mw=="
            },
            { - 
              "category_id": 6,
              "is_parent": true,
              "is_virtual": "false",
              "name": "Scarves",
              "category_uid": "Ng=="
            }
          ],
          "name": [ - 
            "Lampe A Photopolymeriser O-Light II (WOODPECKER) 5397-178*"
          ],
          "url_key": [ - 
            "lampe-a-photopolymeriser-o-light-ii-woodpecker-5397-178"
          ],
          "status": [ - 
            1
          ],
          "option_text_status": [ - 
            "Enabled"
          ],
          "tax_class_id": [ - 
            2
          ],
          "option_text_tax_class_id": [ - 
            "Taxable Goods"
          ],
          "stock": { - 
            "is_in_stock": true,
            "qty": 100
          }
        },
        "sort": [ - 
          159.47162,
          1185
        ]
      }
    ]

[julienanquetil]

Thanks ! I try to simulate with your query but without any luck i didn't get any result. If i put the "DM174" instead of lampe i got the result

But i notice there is some differences in source element I didn't get this key tag :

"row_id": "1185",
"created_in": "1",
"updated_in": "2147483647",

and in indexed attribute we didn't have the tax_class_id as we didn't manage taxes.

so unfortunately no other hint from my side.

@romainruaud : can it be part of some "consulting" task you guys can achieve to debug ?

Thanks

[romainruaud]

Yes we could definitely provide some consulting support on this one.

row_id,created_in and updated_in are fields created by the Magento Commerce staging functionality, that's not relevant with your problem here.

But I'm thinking about some more things :

• can you show us the settings of the "name" attribute in back-office ? we have the same resulting mapping, but who knows ?
• what is the locale of this store ? is this configured to use french ?
• if you use Cerebro for debugging ES queries, do you know how to use the "analysis" tool that is embedded in Cerebro ?

Regards

[julienanquetil]

This is our config for name attribute

you're right the store locale is fr_FR

For the cerebro part i never used but will take a look. Right now i'm using postman to make curl query to ES like the "copy / paste" from the debug log (or your in the previous comment) to try several keywork, compare results..

thanks

[julienanquetil]

@romainruaud I try to digg into it with Cerebro on 2.3 / 2.4 and "compare" mapping / seeting / analysis... but can't find differences (maybe related as i never eard about this tools before) (I attached the settings / mappings in case it can help) Archive.zip

[julienanquetil]

I've made some progress (since of our meeting :) )

1st i discover there should be an issue in a "previous" upgrade as the version in setup_module table was fine, but we miss some fields in some table like sort_order_desc_missing in catalog_eav_attribute

So i made everything clean now.

Also played with query and end up with result (by hand...)

If i take the original query

{
    "size": 20,
    "sort": [
        {
            "_score": {
                "order": "desc"
            }
        },
        {
            "entity_id": {
                "order": "desc",
                "missing": "_first",
                "unmapped_type": "keyword"
            }
        }
    ],
    "from": 0,
    "query": {
        "bool": {
            "filter": {
                "terms": {
                    "visibility": [
                        3,
                        4
                    ],
                    "boost": 1
                }
            },
            "must": {
                "bool": {
                    "filter": {
                        "multi_match": {
                            "query": "lampe",
                            "fields": [
                                "search^1",
                                "sku^1"
                            ],
                            "minimum_should_match": "100%",
                            "tie_breaker": 1,
                            "boost": 1,
                            "type": "best_fields"
                        }
                    },
                    "must": {
                        "multi_match": {
                            "query": "lampe",
                            "fields": [
                                "search^1",
                                "name.standard^5",
                                "sku.reference^6",
                                "search.whitespace^10",
                                "name.whitespace^50",
                                "sku.whitespace^60",
                                "name.sortable^100",
                                "sku.sortable^120"
                            ],
                            "minimum_should_match": 1,
                            "tie_breaker": 1,
                            "boost": 1,
                            "type": "best_fields"
                        }
                    },
                    "boost": 1
                }
            },
            "boost": 1
        }
    },
    "track_total_hits": 0
}

and remove the filter part :

                    "filter": {
                        "multi_match": {
                            "query": "lampe",
                            "fields": [
                                "search^1",
                                "sku^1"
                            ],
                            "minimum_should_match": "100%",
                            "tie_breaker": 1,
                            "boost": 1,
                            "type": "best_fields"
                        }
                    },

I got the expected result.

Or also if i add the attribute in fields "name" it's work as expected

                    "filter": {
                        "multi_match": {
                            "query": "lampe",
                            "fields": [
                                "search^1",
                                "sku^1",
                                "name.standard^1"
                            ],
                            "minimum_should_match": "100%",
                            "tie_breaker": 1,
                            "boost": 1,
                            "type": "best_fields"
                        }
                    },

Also I find if i look for a 3 chars like here gan i got no result but some result (in EN : "No exact results found for: '%1'. The displayed items are the closest matches." )

[julienanquetil]

So after investigation (thanks to @romainruaud for the exchange) we find an "edge case" as we got a magento custom attribute named "search" and it's a "reserved" attribute.

After a deletion / update mapping / reindex all working as expected.