m2-assistant[bot]
commented
1 week ago Hi @hostep. Thank you for your report. To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce.
- For more details, review the Magento Contributor Assistant documentation.
- Add a comment to assign the issue:
@magento I am working on this -
To learn more about issue processing workflow, refer to the Code Contributions.
Join Magento Community Engineering Slack and ask your questions in #github channel. :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting. :clock10: You can find the schedule on the Magento Community Calendar page. :telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.
hostep
engcom-Hotel
github-jira-sync-bot
Preconditions and environment
Steps to reproduce
var/locksdirectoryIdeally you should look at a shop running for many months and where file based locking has been enabled, and inspect the directory that contains those lock files.
Expected result
var/locksdirectory should not overflow with an ever increasing amount of filesActual result
var/lockscan contain up to millions of files after several months of running a Magento shopExample filenames:
We had one project that had 3.5 million files in that directory, up to a point where the filesystem started complaining with errors like
Failed to open stream: No space left on deviceevery time a lock file was needed (there was plenty of disk space, it's just that a directory can't handle that large amount of files in one directory). Another project had 1.9 million files and another had ~170.000 files.So that's really bad. We need some kind of bugfix or cleanup mechanism to keep this under control.
Additional information
The locking mechanism in Magento is used for various things:
Most of these things result in lock names that are predictable and remain the same over time:
However, some of these are pretty dynamic and those names can change over time:
getCacheKeyInfois called and that gets hashed and that hash is being used for the lock name, sometimes with a suffix for thefinal_priceblocks that contain ID's and dates and currencies and so onMore observations:
/var/www/html/release123/, the next one in/var/www/html/release124/, and it turns out that one of the parameters to generate a lock name contains the absolute path to template files, so for each deploy that path changes and the lock name changes as well, so we get an ever growing, never stopping, number of lock files after each deployPossible solutions:
For the last observeration, instead of using absolute paths, we could choose to use relative paths, which should result in more stable lock names, I already came up with this solution, but I have no idea if there will be regressions caused by this, because this data is not used solely for generating lock names:
$this->getTemplateFile(),
$this->getRelativeTemplateFile(), 'template' => $this->getTemplate() ]; } diff --git a/lib/internal/Magento/Framework/View/Element/Template.php b/lib/internal/Magento/Framework/View/Element/Template.php index e68d059c079..116452b4d8c 100644 --- a/lib/internal/Magento/Framework/View/Element/Template.php +++ b/lib/internal/Magento/Framework/View/Element/Template.php @@ -328,12 +328,17 @@ class Template extends AbstractBlock return [ 'BLOCK_TPL', $this->_storeManager->getStore()->getCode(),
$this->getTemplateFile(),
$this->getRelativeTemplateFile(), 'base_url' => $this->getBaseUrl(), 'template' => $this->getTemplate() ]; }
protected function getRelativeTemplateFile(): string
{
return $this->getRootDirectory()->getRelativePath($this->getTemplateFile());
}
/**
Maybe for the block generating locks through
LockGuardedCacheLoader, we can mark those as temporary (if file storage is used) and to be removed immediately after they have been unlocked. No idea if this is a good idea and won't lead to race conditions or other problemsMaybe a new configuration setting in
env.phpshould be added to determine what lock storage we should use for theLockGuardedCacheLoader, Adobe specifially disabled the database one in 2.4.7, for Galera clusters as they have issues with this for some reason, but if this would be configurable and we could choose ourselves what locking mechanism should be used for theLockGuardedCacheLoader, then we can already prevent a big deal of lock files being written to the filesystemAnd then the obvious one: write a cronjob that runs every hour or so that looks at the lock files (if file storage is used) and finds the ones that are not locked and haven't been touched in 24 hours or so and remove those
These are just some random ideas, in case other people have other suggestions, feel free to let me know in the comments below.
Temporary solution: Setup something like this in the
crontabon your server:Thanks!
Release note
No response
Triage and priority