Too many 'clean cache by tags' actions, with same tags, when saving entities and/or running full reindex

[antoinekociuba]

Hi there,

After digging on clean cache by tags mechanism from the back office (save product, category, CMS block and CMS page) and from the CLI full reindex, I have notice that clean_cache_by_tags event is dispatched multiple times, at different places, which could makes sense, but also triggers many duplicated clean cache calls, with mostly same tag(s) to clean, which becomes pretty useless and overkill.

• When Built-in FPC is used, it means that we do multiple same clean calls to Redis when it could be avoided.
• When Varnish FPC is used, it means that we do multiple same purge requests to it, when it also could be avoided.
• When using something else, such as Cloudflare CDN as FPC engine, with a limitation of 30,000 purge API calls in a 24-hour period, it means that we may not be able to flush what is needed all the time, when it also could be much more possible without duplicates.

I enclosed below some logs with backtraces on different kind of entities/actions:

cache_clean_category.log cache_clean_cms_block.log cache_clean_cms_page.log cache_clean_configurable_product_index_save.log cache_clean_configurable_product_index_schedule.log cache_clean_full_reindex.log cache_clean_simple_product_index_save.log cache_clean_simple_product_index_schedule.log

Preconditions (*)

1. Magento Open Source 2.4.2

Steps to reproduce (*)

1. Add a log in Magento\CacheInvalidate\Observer\InvalidateVarnishObserver PHP class or in Magento\PageCache\Observer\FlushCacheByTags PHP class.
2. Save different types of entities from the back office (product, category, CMS block and CMS page), with indexers set on 'Update on Save' and/or 'Update on Schedule.'
3. And also run a full reindex from CLI.

Expected result (*)

Do not trigger duplicated cache clean actions with the exact same cache tags mostly, and reduce number of cache clean actions in the same execution thread (less connexions/traffic to Redis/Varnish/Anything else).

It could be nice to have a kind of 'Tags to clean' collector object, which will be responsible to collect, obviously, and to launch, at the very end of any save actions and/or full reindex CLI process, only one or needed cache clean calls, without any duplicated tags.

Actual result (*)

1. Multiple cache clean calls, with mostly same cache tags.

Thank you in advance for your help and your consideration.

[m2-assistant[bot]]

Hi @antoinekociuba. Thank you for your report. To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

  :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[mrtuvn]

cc: @Vinai may have solid knowledges on this area

[Vinai]

I've had issues in the past with Redis becoming unresponsive because of the number of cache clean requests generated by an import script. So in general I agree this issue exists when there are a large number of clean requests.

I heard from one colleague who also had issues and solved them by collecting cache clean requests instead of sending them to the backend immediately. Then a batch job (he used cron at the time) collected all pending tags and removed duplicates. It batched tags so they fit into a single request to Varnish (because of the header size limit).

[m2-assistant[bot]]

Hi @engcom-Bravo. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• [ ] 5. Add label Issue: Confirmed once verification is complete.

• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[m2-assistant[bot]]

Hi @engcom-Alfa. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• [ ] 5. Add label Issue: Confirmed once verification is complete.

• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[m2-assistant[bot]]

Hi @engcom-Hotel. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• [ ] 5. Add label Issue: Confirmed once verification is complete.

• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[mrtuvn]

Seem related with recently prob

[engcom-Hotel]

Hello @antoinekociuba,

We are able to reproduce the issue in the 2.4-develop branch with the following steps, hence we are confirming the issue:

1. Set up a fresh installation of Magento 2.4-develop.
2. Create a custom log file in the instance under var/log/ eg: var/log/clean_cahe.log
3. Add a logger in Magento\PageCache\Observer\FlushCacheByTags class to log the cache tags getting cleaned onto our custom log file.
4. Admin dashboard - System -> Tools -> Index Management -> Set the indexers to 'Update on Save'
5. Run a full reindex from CLI.
6. Try to save different types of entities from the back office (product, category, CMS block, and CMS page) - one can observe the same cache types getting cleaned multiple times for a single operation.

Thanks for the contribution.

[magento-engcom-team]

@engcom-Hotel Thank you for verifying the issue.

Unfortunately, not enough information was provided to acknowledge ticket. Please consider adding the following:

• [ ] Add "Reproduced on " label(s) to this ticket based on verification result

Once all required information is added, please add label "Issue: Confirmed" again. Thanks!

[magento-engcom-team]

:white_check_mark: Confirmed by @engcom-Hotel Thank you for verifying the issue. Based on the provided information internal tickets MC-43065 were created

Issue Available: @engcom-Hotel, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[mrtuvn]

I heard from one colleague who also had issues and solved them by collecting cache clean requests instead of sending them to the backend immediately. Then a batch job (he used cron at the time) collected all pending tags and removed duplicates. It batched tags so they fit into a single request to Varnish (because of the header size limit).

@Vinai seem this is private fix . Hope we can able deliver to repo public magento