search

magento / magento2

Prior to making any Submission(s), you must sign an Adobe Contributor License Agreement, available here at: https://opensource.adobe.com/cla.html. All Submissions you make to Adobe Inc. and its affiliates, assigns and subsidiaries (collectively “Adobe”) are subject to the terms of the Adobe Contributor License Agreement.
http://www.magento.com
Open Software License 3.0
11.54k stars 9.32k forks source link

Elasticsearch issue "number_format_exception" - 2.4.1 #30540

Closed trunglv closed 3 years ago

trunglv commented 4 years ago

I found a bug in the Elasticsearch extension when I tried to use the search functionality.
Here is a log message I got

{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: For input string: \"xxx\"","index_uuid":"w5egRGlqQV6LbgrdYBqBYQ","index":"magento2_product_26_v157"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"magento2_product_26_v157","node":"UulsefOgTbiBKXUkTrwSPg","reason":{"type":"query_shard_exception","reason":"failed to create query: For input string: \"kawai\"","index_uuid":"w5egRGlqQV6LbgrdYBqBYQ","index":"magento2_product_26_v157","caused_by":{"type":"number_format_exception","reason":"For input string: \"xxx\""}}}]},"status":400}

What I found is: Magento\Elasticsearch\SearchAdapter\Query\ValueTransformerPool was missing an item for a double type attribute. So we should add a new one: <item name="double" xsi:type="object">Magento\Elasticsearch\SearchAdapter\Query\ValueTransformer\DoubleTransformer</item> or <item name="float" xsi:type="object">Magento\Elasticsearch\SearchAdapter\Query\ValueTransformer\FloatTransformer</item>

File: vendor/magento/module-elasticsearch/etc/di.xml

<type name="Magento\Elasticsearch\SearchAdapter\Query\ValueTransformerPool">
        <arguments>
            <argument name="valueTransformers" xsi:type="array">
                <item name="default" xsi:type="object">Magento\Elasticsearch\SearchAdapter\Query\ValueTransformer\TextTransformer</item>
                <item name="date" xsi:type="object">Magento\Elasticsearch\SearchAdapter\Query\ValueTransformer\DateTransformer</item>
                <item name="float" xsi:type="object">Magento\Elasticsearch\SearchAdapter\Query\ValueTransformer\FloatTransformer</item>
                <item name="integer" xsi:type="object">Magento\Elasticsearch\SearchAdapter\Query\ValueTransformer\IntegerTransformer</item>
            </argument>
        </arguments>
    </type>

Preconditions (*)

  1. Magento 2.4.1
  2. Elastic Search

Steps to reproduce (*)

  1. Try to search for a product on the frontend
  2. It doesn't work if there is a product attribute type is a double that is considered in the search query.

Expected result (*)

Search functionality should work properly.

Actual result (*)

It doesn't work if there is a product attribute type is a double that is considered in the search query.

m2-assistant[bot] commented 4 years ago

Hi @trunglv. Thank you for your report. To help us process this issue please make sure that you provided the following information:

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

m2-assistant[bot] commented 4 years ago

Hi @engcom-Bravo. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

engcom-Bravo commented 4 years ago

Hello @trunglv

Thank you for your report

Can you please provide more details about the steps to reproduce with, screenshots attached? It's not clear how is the double attribute created and searched for. In this way, we could faster reproduce the issue and provide a fix for it

Thank you in advance

trunglv commented 4 years ago

Hi @engcom-Bravo It seems a problem will happen if we set a decimal attribute to be searchable. Here is what I debugged:

Class : Magento\Elasticsearch\SearchAdapter\Query\Builder\Match => function buildQueries

$attributeAdapter = $this->attributeProvider->getByAttributeCode($resolvedField);
$fieldType = $this->fieldTypeResolver->getFieldType($attributeAdapter);

A variable $fieldType is returned a value is "double", should be "float"

class : Magento\Elasticsearch\Model\Adapter\FieldMapper\Product\FieldProvider\FieldType\Resolver\CompositeResolver class : Magento\Elasticsearch\Model\Adapter\FieldMapper\Product\FieldProvider\FieldType\Resolver\FloatType

if ($attribute->isFloatType()) {
            return $this->fieldTypeConverter->convert(ConverterInterface::INTERNAL_DATA_TYPE_FLOAT);
}

class Magento\Elasticsearch\Model\Adapter\FieldMapper\Product\FieldProvider\FieldType\Converter

private $mapping = [
        self::INTERNAL_DATA_TYPE_STRING => self::ES_DATA_TYPE_STRING,
        self::INTERNAL_DATA_TYPE_KEYWORD => self::ES_DATA_TYPE_STRING,
        self::INTERNAL_DATA_TYPE_FLOAT => self::ES_DATA_TYPE_DOUBLE,
        self::INTERNAL_DATA_TYPE_INT => self::ES_DATA_TYPE_INT,
        self::INTERNAL_DATA_TYPE_DATE => self::ES_DATA_TYPE_DATE,
    ];

You can see variable mappings: It mapped "float" => double. So it's a problem I think.

engcom-Bravo commented 4 years ago

Hello @trunglv

I have rechecked the issue on the latest 2.4-develop with the following settings Catalog Input Type for Store Owner - Text field Input Validation for Store Owner - Decimal Number dec_attr_sett Use in Search - Yes Visible in Advanced Search - Yes

I have created a Simple product and added the previously created attribute to it attr_in_product

I can perform search on Storefront via attribute value and via Title. No error appears in log

Can you please confirm that you are able to reproduce the issue on 2.4-develop?

Thank you

krish-vivek commented 4 years ago

Hello @trunglv Agree with @engcom-Bravo

After following steps mentioned by you, product is showing in search list, so can you provide more information

Please review screenshot.

1_search 2_attributr

m2-assistant[bot] commented 3 years ago

Hi @krishprakash. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

krishprakash commented 3 years ago

Hello @trunglv. Feel free to reopen/comment once you confirm that issue reproduce on 2.4-develop instance Thanks!!

siliconalchemy commented 3 years ago

I'm getting exactly the same problem. I wasn't actually aware I had so many attributes searchable (141!) so it will be difficult to track down which is the problem attribute, but this only occurred after upgrade from 2.3 to 2.4 and change from mysql to elasticsearch catalog search engine. The elasticsearch error is:

Caused by: java.lang.NumberFormatException: For input string: "BEKO"
    at jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054) ~[?:?]
    at jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110) ~[?:?]
    at java.lang.Double.parseDouble(Double.java:549) ~[?:?]
    at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType.objectToDouble(NumberFieldMapper.java:876) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType.access$200(NumberFieldMapper.java:173) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$3.parse(NumberFieldMapper.java:369) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$3.termQuery(NumberFieldMapper.java:388) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.mapper.NumberFieldMapper$NumberFieldType.termQuery(NumberFieldMapper.java:928) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.search.MatchQuery$MatchQueryBuilder.lambda$newTermQuery$1(MatchQuery.java:576) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.search.MatchQuery$MatchQueryBuilder.newTermQuery(MatchQuery.java:579) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.search.MatchQuery.parse(MatchQuery.java:277) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.query.MatchQueryBuilder.doToQuery(MatchQueryBuilder.java:426) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:99) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:394) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:380) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:99) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$1(QueryShardContext.java:334) ~[elasticsearch-7.6.2.jar:7.6.2]
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:346) ~[elasticsearch-7.6.2.jar:7.6.2]

And the corresponding entry in magento log:

}\",\"index_uuid\":\"rgbnzqW-SViJjs7WwTy-yg\",\"index\":\"magento2_product_1_v6\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"BEKO\\\"\"}}}]},\"status\":400} at /srv/www/dev.www/html/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:632)"} []

Magento: 2.4.1 opensource.

siliconalchemy commented 3 years ago

OK this was actually pretty easy to track down. Look for eav_attribute.backend_type ='decimal' and change 'Use in Search' to 'No' for each attribute:

Screenshot 2020-12-11 at 17 53 05 Screenshot 2020-12-11 at 17 52 03

After this, the search works as expected.

zhartaunik commented 3 years ago

I faced the same issue. In my case attribute wasn't created using UI. Decimal attributes there created as backend_type = varchar, frontend_input = text In my case I made the following: backend_type = decimal, frontend_input = text [optional input validation]

HarryHung-Shopstack commented 3 years ago

i got same issue. how about update it, hello @trunglv

trunglv commented 3 years ago

Pls check my quick fix - It may help https://github.com/betagento/magento2_search_fixing

HarryHung-Shopstack commented 3 years ago

Pls check my quick fix - It may help https://github.com/betagento/magento2_search_fixing

thanks @trunglv , i tried but it's not working for me. hot fix i have to set config "use in search" to No

trunglv commented 3 years ago

Pls check my quick fix - It may help https://github.com/betagento/magento2_search_fixing

thanks @trunglv , i tried but it's not working for me. hot fix i have to set config "use in search" to No

It worked on my customer website. Hope you have already done necessary commands for it : setup-upgrade, Di Compile -- Cache so on ... -- Anyway Thanks for trying. I will try it to another place

HarryHung-Shopstack commented 3 years ago

i am using magento 2.4.0 i create an attribute image i can typing the number to the search box and it's working but if i typing the string/text to the search box, it's not work already got the error Exception #0 (Elasticsearch\Common\Exceptions\BadRequest400Exception): {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: For input string: \"xxx\"","index_uuid":"4Z6fNLOJT9mTqJ6KP9Py2Q","index":"magento2_homepro_product_2_v2"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"magento2_homepro_product_2_v2","node":"TGeJyLN7QhWm3nsFo-bVxA","reason":{"type":"query_shard_exception","reason":"failed to create query: For input string: \"xxx\"","index_uuid":"4Z6fNLOJT9mTqJ6KP9Py2Q","index":"magento2_homepro_product_2_v2","caused_by":{"type":"number_format_exception","reason":"For input string: \"xxx\""}}}]},"status":400}

tuyennn commented 3 years ago

@krishprakash @engcom-Bravo this was still reproduced on M2.4.0-p1

trunglv commented 3 years ago

@krishprakash @engcom-Bravo this was still reproduced on M2.4.0-p1

Try to use my solution https://github.com/betagento/magento2_search_fixing

jamie-selesti commented 3 years ago

Still reproduced on Magento 2.4.2-p1

Using https://github.com/betagento/magento2_search_fixing fixed my issue

netjordanlee commented 2 years ago

Reproduced in 2.4.3-p1

As @HarryHung-Shopstack describes, we can search for numbers, but trying to do a search for any string results in the error above (number_format_exception). Haven't tested the fix from @trunglv yet, will edit here with results when tested.

We narrowed down the problematic attribute which was of type int not decimal - so this may affect other number types.

redati commented 1 year ago

reproduced in 2.4.5-p1 and https://github.com/betagento/magento2_search_fixing dont fix for me.

siliconalchemy commented 1 year ago

Magento will ignore any closed issue (and close any new one as quickly as possible) - no point commenting on this one. Raise a new issue and reference this one (see #30194). I've had to raise 4 or 5 issues for a problem before to get it fixed.

https://github.com/betagento/magento2_search_fixing doesn't work any longer for me either in 2.4.5-p1.

redati commented 1 year ago

my solution is fix in amasty shopby

app/code/Amasty/Shopby/Model/Layer/Filter/Category.php em apply $categoryId = is_numeric($categoryId) ? $categoryId : null;

where $categoryId is not validated and used for atack population log, requests contains ?cat=string and others sql injection

JonathanHelvey commented 1 year ago

https://github.com/betagento/magento2_search_fixing doesn't work any longer for us either. We are on Magento2.4.4-p2. This error is thrown constantly in our exception.log in Magento2. Also, we do not have Amasty/Shopby module on our store.