Poor performance on sales order update - string to integer

asim-vax commented 5 years ago

When running a sales order update, the where clause entity_id is a string. It is then comparing a string to an integer and the index is not used. same issue as https://github.com/magento/magento2/issues/20969 just for a different query. eg. WHERE (entity_id='99015') query below:

UPDATE `sales_order` 
SET `state` = 'processing', `status` = 'processing', `coupon_code` = NULL, `protect_code` = '623debed81d2068bf3e910a02e860066', `shipping_description` = 'Shipping Option - FREE Nominated Delivery', `is_virtual` = '0', `store_id` = '3', `customer_id` = NULL, `base_discount_amount` = '0', `base_discount_canceled` = NULL, `base_discount_invoiced` = '0', `base_discount_refunded` = NULL, `base_grand_total` = '179.96', `base_shipping_amount` = '0', `base_shipping_canceled` = NULL, `base_shipping_invoiced` = '0', `base_shipping_refunded` = NULL, `base_shipping_tax_amount` = '0',
`base_shipping_tax_refunded` = NULL, `base_subtotal` = '149.97', `base_subtotal_canceled` = NULL, `base_subtotal_invoiced` = '149.97', `base_subtotal_refunded` = NULL, `base_tax_amount` = '29.99', `base_tax_canceled` = NULL, `base_tax_invoiced` = '29.99', `base_tax_refunded` = NULL, `base_to_global_rate` = '1', `base_to_order_rate` = '1', `base_total_canceled` = NULL, `base_total_invoiced` = '179.96', `base_total_invoiced_cost` = '0', `base_total_offline_refunded` = NULL, `base_total_online_refunded` = NULL, `base_total_paid` = '179.96', `base_total_qty_ordered` = NULL, `base_total_refunded` = NULL, 
`discount_amount` = '0', `discount_canceled` = NULL, `discount_invoiced` = '0', `discount_refunded` = NULL, `grand_total` = '179.96', `shipping_amount` = '0', `shipping_canceled` = NULL, `shipping_invoiced` = '0', `shipping_refunded` = NULL, `shipping_tax_amount` = '0', `shipping_tax_refunded` = NULL, `store_to_base_rate` = '0', `store_to_order_rate` = '0', `subtotal` = '149.97', `subtotal_canceled` = NULL, `subtotal_invoiced` = '149.97', `subtotal_refunded` = NULL, `tax_amount` = '29.99', `tax_canceled` = NULL, `tax_invoiced` = '29.99', `tax_refunded` = NULL, `total_canceled` = NULL, `total_invoiced` = '179.96',
 `total_offline_refunded` = NULL, `total_online_refunded` = NULL, `total_paid` = '179.96', `total_qty_ordered` = '1', `total_refunded` = NULL, `can_ship_partially` = NULL, `can_ship_partially_item` = NULL, `customer_is_guest` = '1', `customer_note_notify` = '1', `billing_address_id` = '182939', `customer_group_id` = '0', `edit_increment` = NULL, `email_sent` = '1', `send_email` = '1', `forced_shipment_with_invoice` = NULL, `payment_auth_expiration` = NULL, `quote_address_id` = NULL, `quote_id` = '168962', `shipping_address_id` = '182938', `adjustment_negative` = NULL, `adjustment_positive` = NULL,
 `base_adjustment_negative` = NULL, `base_adjustment_positive` = NULL, `base_shipping_discount_amount` = '0', `base_subtotal_incl_tax` = '179.96', `base_total_due` = '0', `payment_authorization_amount` = NULL, `shipping_discount_amount` = '0', `subtotal_incl_tax` = '179.96', `total_due` = '0', `weight` = '3', `customer_dob` = NULL, `increment_id` = '', `applied_rule_ids` = NULL, `base_currency_code` = 'GBP', `customer_email` = '', `customer_firstname` = NULL, `customer_lastname` = NULL, `customer_middlename` = NULL, `customer_prefix` = NULL, `customer_suffix` = NULL, `customer_taxvat` = NULL, 
`discount_description` = NULL, `ext_customer_id` = NULL, `ext_order_id` = NULL, `global_currency_code` = 'GBP', `hold_before_state` = NULL, `hold_before_status` = NULL, `order_currency_code` = 'GBP', `original_increment_id` = NULL, `relation_child_id` = NULL, `relation_child_real_id` = NULL, `relation_parent_id` = NULL, `relation_parent_real_id` = NULL, `remote_ip` = '', `shipping_method` = 'productmatrix_Free_Nominated_Delivery', `store_currency_code` = 'GBP', `store_name` = 'Vax UK\nVax UK \nVax UK Store view', `x_forwarded_for` = '', `customer_note` = NULL, `created_at` = '2019-08-28 16:46:51', 
`total_item_count` = '1', `customer_gender` = NULL, `discount_tax_compensation_amount` = '0', `base_discount_tax_compensation_amount` = '0', `shipping_discount_tax_compensation_amount` = '0', `base_shipping_discount_tax_compensation_amnt` = '0', `discount_tax_compensation_invoiced` = '0', `base_discount_tax_compensation_invoiced` = '0', `discount_tax_compensation_refunded` = NULL, `base_discount_tax_compensation_refunded` = NULL, `shipping_incl_tax` = '0', `base_shipping_incl_tax` = '0', `coupon_rule_name` = NULL, `base_customer_balance_amount` = NULL,
 `customer_balance_amount` = NULL, `base_customer_balance_invoiced` = NULL, `customer_balance_invoiced` = NULL, `base_customer_balance_refunded` = NULL, `customer_balance_refunded` = NULL, `bs_customer_bal_total_refunded` = NULL, `customer_bal_total_refunded` = NULL, `gift_cards` = '[]', `base_gift_cards_amount` = '0', `gift_cards_amount` = '0', `base_gift_cards_invoiced` = NULL, `gift_cards_invoiced` = NULL, `base_gift_cards_refunded` = NULL, `gift_cards_refunded` = NULL, `gift_message_id` = NULL, 
`gw_id` = NULL, `gw_allow_gift_receipt` = NULL, `gw_add_card` = NULL, `gw_base_price` = '0', `gw_price` = '0', `gw_items_base_price` = '0', `gw_items_price` = '0', `gw_card_base_price` = '0', `gw_card_price` = '0', `gw_base_tax_amount` = NULL, `gw_tax_amount` = NULL, `gw_items_base_tax_amount` = NULL, `gw_items_tax_amount` = NULL, `gw_card_base_tax_amount` = NULL, `gw_card_tax_amount` = NULL, `gw_base_price_incl_tax` = NULL, `gw_price_incl_tax` = NULL, `gw_items_base_price_incl_tax` = NULL, `gw_items_price_incl_tax` = NULL, `gw_card_base_price_incl_tax` = NULL,
 `gw_card_price_incl_tax` = NULL, `gw_base_price_invoiced` = NULL, `gw_price_invoiced` = NULL, `gw_items_base_price_invoiced` = NULL, `gw_items_price_invoiced` = NULL, `gw_card_base_price_invoiced` = NULL, `gw_card_price_invoiced` = NULL, `gw_base_tax_amount_invoiced` = NULL, `gw_tax_amount_invoiced` = NULL, `gw_items_base_tax_invoiced` = NULL, `gw_items_tax_invoiced` = NULL, `gw_card_base_tax_invoiced` = NULL, `gw_card_tax_invoiced` = NULL, `gw_base_price_refunded` = NULL, `gw_price_refunded` = NULL, 
`gw_items_base_price_refunded` = NULL, `gw_items_price_refunded` = NULL, `gw_card_base_price_refunded` = NULL, `gw_card_price_refunded` = NULL, `gw_base_tax_amount_refunded` = NULL, `gw_tax_amount_refunded` = NULL, `gw_items_base_tax_refunded` = NULL, `gw_items_tax_refunded` = NULL, `gw_card_base_tax_refunded` = NULL, `gw_card_tax_refunded` = NULL, `reward_points_balance` = NULL, `base_reward_currency_amount` = NULL, `reward_currency_amount` = NULL, `base_rwrd_crrncy_amt_invoiced` = NULL, 
`rwrd_currency_amount_invoiced` = NULL, `base_rwrd_crrncy_amnt_refnded` = NULL, `rwrd_crrncy_amnt_refunded` = NULL, `reward_points_balance_refund` = NULL, `paypal_ipn_customer_notified` = '0', `account_id` = NULL, `case_id` = NULL, `agent_id` = NULL, `agent_name` = NULL, `nvm_agent_id` = NULL, `is_drtv` = NULL, `foc_code` = NULL, `foc_reason` = NULL, `sf_agent_id` = NULL, `call_product_sku` = NULL, `is_custom_upgrade` = NULL, `tcs_agreement_date` = NULL, `exported_to_gp` = '1', `gp_order_id` = '', `pentagon_order_id` = '', `pentagon_processed_order` = '0', `sent_in_bv_postpurchase_feed` = '0' 
WHERE (entity_id='99015');

should finish

WHERE (entity_id = 99015);

Column	Type	Default Value	Nullable	Character Set	Collation	Privileges	Extra	Comments
entity_id	int(10) unsigned		NO			select	auto_increment	Entity Id

Preconditions (*)

magento 2.3.2
ubuntu 18.04
php 7.2

Steps to reproduce (*)

turn on query logging, or view in mysql queries being ran
update an order
look at query
entity_id is wrapped in quotes - making it a string

Expected result (*)

entity_id is an integer
no performance loss

Actual result (*)

entity_id is a string
slow performance

m2-assistant[bot] commented 5 years ago

Hi @asim-vax. Thank you for your report. To help us process this issue please make sure that you provided the following information:

[ ] Summary of the issue
[ ] Information on your environment
[ ] Steps to reproduce
[ ] Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.3-develop instance - upcoming 2.3.x release

For more details, please, review the Magento Contributor Assistant documentation.

@asim-vax do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

[ ] yes
[ ] no

Join Magento Community Engineering Slack and ask your questions in #github channel.

irving0mar commented 5 years ago

@magento give me 2.6-develop instance

magento-engcom-team commented 5 years ago

Hi @irving0mar. Thank you for your request. I'm working on Magento 2.6-develop instance for you

irving0mar commented 5 years ago

@magento give me 2.2.6-develop instance

magento-engcom-team commented 5 years ago

Hi @irving0mar. Thank you for your request. Unfortunately, I can only deploy instances for 2.3 release line

csdougliss commented 5 years ago

This issue is on Magento 2.3.2

m2-assistant[bot] commented 5 years ago

Hi @engcom-Charlie. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

[ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
Details
If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.
[ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.
[ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.
[ ] 4. Verify that the issue is reproducible on 2.3-develop branch
Details
- Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
- If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
- If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
[ ] 5. Add label Issue: Confirmed once verification is complete.
[ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

magento-engcom-team commented 5 years ago

:white_check_mark: Confirmed by @engcom-Charlie Thank you for verifying the issue. Based on the provided information internal tickets MC-19791 were created

Issue Available: @engcom-Charlie, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

csdougliss commented 5 years ago

Further examples of this:

SELECT `sales_order`.`entity_id` FROM `sales_order` WHERE (entity_id = '105465') FOR UPDATE

and:

UPDATE `quote` SET `store_id` = ?, `created_at` = '2019-09-02 17:04:24', `converted_at` = ?, `is_active` = ?, `is_virtual` = ?, `is_multi_shipping` = ?, `items_count` = ?, `items_qty` = ?, `orig_order_id` = ?, `store_to_base_rate` = ?, `store_to_quote_rate` = ?, `base_currency_code` = ?, `store_currency_code` = ?, `quote_currency_code` = ?, `grand_total` = ?, `base_grand_total` = ?, `checkout_method` = ?, `customer_id` = ?, `customer_tax_class_id` = ?, `customer_group_id` = ?, `customer_email` = ?, `customer_prefix` = ?, `customer_firstname` = ?, `customer_middlename` = ?, `customer_lastname` = ?, `customer_suffix` = ?, `customer_dob` = ?, `customer_note` = ?, `customer_note_notify` = ?, `customer_is_guest` = ?, `remote_ip` = ?, `applied_rule_ids` = ?, `reserved_order_id` = ?, `password_hash` = ?, `coupon_code` = ?, `global_currency_code` = ?, `base_to_global_rate` = ?, `base_to_quote_rate` = ?, `customer_taxvat` = ?, `customer_gender` = ?, `subtotal` = ?, `base_subtotal` = ?, `subtotal_with_discount` = ?, `base_subtotal_with_discount` = ?, `is_changed` = ?, `trigger_recollect` = ?, `ext_shipping_info` = ?, `customer_balance_amount_used` = ?, `base_customer_bal_amount_used` = ?, `use_customer_balance` = ?, `gift_cards` = ?, `gift_cards_amount` = ?, `base_gift_cards_amount` = ?, `gift_cards_amount_used` = ?, `base_gift_cards_amount_used` = ?, `gift_message_id` = ?, `gw_id` = ?, `gw_allow_gift_receipt` = ?, `gw_add_card` = ?, `gw_base_price` = ?, `gw_price` = ?, `gw_items_base_price` = ?, `gw_items_price` = ?, `gw_card_base_price` = ?, `gw_card_price` = ?, `gw_base_tax_amount` = ?, `gw_tax_amount` = ?, `gw_items_base_tax_amount` = ?, `gw_items_tax_amount` = ?, `gw_card_base_tax_amount` = ?, `gw_card_tax_amount` = ?, `gw_base_price_incl_tax` = ?, `gw_price_incl_tax` = ?, `gw_items_base_price_incl_tax` = ?, `gw_items_price_incl_tax` = ?, `gw_card_base_price_incl_tax` = ?, `gw_card_price_incl_tax` = ?, `use_reward_points` = ?, `reward_points_balance` = ?, `base_reward_currency_amount` = ?, `reward_currency_amount` = ?, `is_persistent` = ?, `account_id` = ?, `case_id` = ?, `agent_name` = ?, `nvm_agent_id` = ?, `is_drtv` = ?, `foc_code` = ?, `foc_reason` = ?, `sf_agent_id` = ?, `call_product_sku` = ?, `is_custom_upgrade` = ?, `tcs_agreement_date` = ?, `delivery_date` = '2019-09-04 00:00:00', `marketing_opt_in` = ?, `marketing_post_opt_in` = ?, `instalments_first_date` = ? WHERE (entity_id='178265')

entity_id should not be enclosed in quotes.

pierzakp commented 5 years ago

"#mm19pl"

m2-assistant[bot] commented 5 years ago

Hi @pierzakp. Thank you for working on this issue. Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

[ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.
[ ] 2. Verify that the issue is reproducible on 2.3-develop branch
Details
- Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
- If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
- If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
[ ] 3. If the issue is not relevant or is not reproducible any more, feel free to close it.

Join Magento Community Engineering Slack and ask your questions in #github channel.

magento-engcom-team commented 5 years ago

Hi @asim-vax, @pierzakp.

Thank you for your report and collaboration!

The issue was fixed by Magento team.

The fix will be available with the upcoming 2.3.4 release.

csdougliss commented 5 years ago

@magento-engcom-team do you have a link to the commit or PR?

o-iegorov commented 5 years ago

This issue was fixed by internal team and PR currently in delivery queue

pierzakp commented 5 years ago

It's sad that there was no label / information that core team is handling it, waste of my time.

o-iegorov commented 5 years ago

Already merged into mainline

o-iegorov commented 5 years ago

@pierzakp there was information about internal ticket created, but i am agree, probably for such cases should be information about progress of internal ticket

hostep commented 5 years ago

Looks like this is the merge commit (for the people interested): https://github.com/magento/magento2/commit/0a390bbf1d0f27fc3fdcbf6844d7cf7c67efbf0c

pemann commented 5 years ago

There are lots of places in magento code where string is used instead of int when querying by id-fields. You can seen them by first enabling db query log php bin/magento dev:query-log:enable and then browse some time your magento store. After a while check db log with: grep "_id = '" var/debug/db.log and grep "_id IN ('" var/debug/db.log"

csdougliss commented 5 years ago

@pemann Is that with this commit applied or before?

pemann commented 5 years ago

@craigcarnell before. My comment was just a remark that there are similar problems all-over magento. Someone should look into this with wider perspective and try to fix them all.

pierzakp commented 5 years ago

@pemann my PR was fixing it in all cases have confirmed that after reviewing var/debug/db.log, for update and select statements.

Commit mentioned above is fixing it (still think that my PR was a cleaner solution) only for update statements, what about selects??

o-iegorov commented 5 years ago

@pierzakp The issue opened for update query, if you have proved measurement that for select queries this will increase performance please add new issue

hostep commented 4 years ago

@o-iegorov: it looks like a new issue was opened for select queries some days ago: https://github.com/magento/magento2/issues/25199

magento / magento2