2.4.4-p10-Set indexers invalid during next immediate deployment after the key rotation

[senthilengg]

Preconditions and environment

No response

Steps to reproduce

1. Apply Patch 2.4.4-P10
2. Run Key Rotation command
3. Try to deploy again
4. Indexers will set as invalid and if the store has more than a million customer then setup:upgrade will take a lot of time

Expected result

Setup Upgrade should complete it in few mins if there are no schema changes

Actual result

Setup:upgrade takes more than 30 min and increase based on the number of customers

Additional information

No response

Release note

No response

Triage and priority

• [ ] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• [X] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• [ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant[bot]]

Hi @senthilengg. Thank you for your report. To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce.

• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this

• To learn more about issue processing workflow, refer to the Code Contributions.

  Join Magento Community Engineering Slack and ask your questions in #github channel. :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting. :clock10: You can find the schedule on the Magento Community Calendar page. :telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[m2-assistant[bot]]

Hi @engcom-Hotel. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• [ ] 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• [ ] 5. Add label Issue: Confirmed once verification is complete.
• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Hotel]

Hello @senthilengg,

Thanks for the report and collaboration!

We have tried to reproduce the issue in the latest 2.4-develop branch with extra_large.xml performance toolkit.

We have 10 lakhs customers and same numbers of products. Please refer to the below screenshot:

We have followed the below steps to reproduce the issue:

1. Run below command for key rotation: bin/magento encryption:key:change

2. Deploy again: bin/magento setup:static-content:deploy -f

3. Then run bin/magento setup:upgrade, which took some seconds to complete.

Please let us know if we missed anything.

Thanks

[senthilengg]

@engcom-Hotel Thank you for working on this issue. It was taking 30-40+ mins for a 6 million customer DB. Fundamentally speaking there is no point in setting indexer invalid for this case.

More than reproducing this issue what i would suggest as part of the Key Rotation command, it should reupdate hash of indexer so the experience will be seamless given that we know no modification to the indexer schema.

Thank you

[engcom-Hotel]

Hello @senthilengg,

Thanks for the reply!

We have investigated this issue further. We ran bin/magento indexer:status, which does not show any index to reindex:

After running bin/magento encryption:key:change, we get the same index status:

After that, we ran time bin/magento setup:upgrade, which took almost 18 mins to complete:

But as the issue description, the bin/magento encryption:key:change command does not invalid the index status.

[senthilengg]

@engcom-Hotel Thank you for the update. The indexer will get invalidated and data get refreshed during the next deployment after the key rotation.

Step 1 : Deploy P10 patch Step 2 : Rotate Encryption Key Step 3: Deploy again with some changes (You can find all the indexer last updated date will be almost the same time)

Line of code that creates this issue : https://github.com/magento/magento2/blob/2.4.4-p10/app/code/Magento/Indexer/Setup/Recurring.php#L108

Hope this provide more details.

[engcom-Hotel]

Hello @senthilengg,

Thanks for the updates!

I can see several updates on this file in the latest development branch i.e. 2.4-develop: https://github.com/magento/magento2/blob/6acfd6aa744c905f7694f2cddd5b0add5bf202b7/app/code/Magento/Indexer/Setup/Recurring.php#L1

And as per the process, we need to reproduce the issue in the latest development branch. So we request you to please try to reproduce the issue in the latest development branch and let us know if the issue is reproducible for you or not.

And also we suggest you to please upgrade with the latest stable version.

Thanks

[senthilengg]

@engcom-Hotel Are you saying you are not able to reproduce with the above step after you update the encrypt key ?

The command bin/magento encryption:key:change command does not invalid the index status itself doesn't invalidate the index.

It happens during the next deployment after the command run.

For a small size of catalog & user base it may not very visible, since dropping the data in customer flat grid will not take time but for large customer base it does. You can find this easily after update check the indexer status/time then deploy again and check the time almost all indexer will have similar time.

You can also introduce additional logs patching the core while you are running in the local on the Recurring.php file you can check. Basically this condition will match $stateIndexers[$indexerId]->getHashConfig() != $expectedHashConfig and indexer will set as INVALID.

[engcom-Hotel]

Yes @senthilengg, we are not able to reproduce the issue by following the steps mentioned in the main description. Please refer to this https://github.com/magento/magento2/issues/39206#issuecomment-2446558963.

Please try to reproduce the issue in the latest stable version or the latest development branch. Actually, there were many changes that happened after the 2.4.4-P10 release in the said file.

Thanks

[engcom-Hotel]

Dear @senthilengg,

This issue is being closed since it has not been updated in a long time. Please feel free to reopen or raise a new ticket if the issue still exists.

Thanks