Merged CSS/JS files require pub/static/_cache writeable, contrary to documentation

[scottsb]

Preconditions

1. Have Magento installed with two filesystem owners as documented, with all code files (source and CLI-generated) read-only for web server user. This includes the full pub/static directory.
2. Turn on the merging of CSS or JS files.

Steps to reproduce

Clear static file cache and load home page.

Expected result

No exception should be generated.

Actual result

Exception is logged:

[2018-01-16 16:18:56] main.CRITICAL: Directory "/var/www/html/___/pub/static/_cache/merged" cannot be created Warning!mkdir(): Permission denied {"exception":"[object] (Magento\\Framework\\Exception\\FileSystemException(code: 0): Directory \"/var/www/html/___/pub/static/_cache/merged\" cannot be created Warning!mkdir(): Permission denied at /var/www/html/___/vendor/magento/framework/Filesystem/Driver/File.php:225)"} []

Commentary

This could arguably be seen as a documentation issue in which case I can report it in the devdocs repo. However, as I understand it, it seems that the documentation does describe the intended state, meaning the code should be changed instead. I'm not sure if it's possible, but if it were, ideally the merged files would be generated by static-content:deploy as alluded to in #9542. This would allow multiple servers in a cluster to be certain to have the merged files present in response to requests without the directory being shared among the nodes.

[davidalger]

+1 here guys! Nothing should be written to pub/static when in production mode, and a setup:static-content:deploy should write out merged files to static/_cache. This causes problems in load-balanced environments where a snapshot of a pre-compiled build is deployed.

[dan-ding]

bump sort of related: https://github.com/magento/magento2/issues/9542

[ghost]

Hi @scottsb i cannot reproduce you scenario, which version of magento you've tested ?

[graemel]

I see this behaviour in 2.1.

[davidalger]

This is a problem with 2.2.x as well. It starts with the files in static/_cache/merged not being written during static content deploy, and then the issue is compounded (on either a read-only file system or ) when Magento attempts to write merged JS / CSS files to disk during a web request.

[ghost]

@scottsb , we are closing this issue due to inactivity. If you'd like to update it, please reopen the issue.

[ishakhsuvarov]

@davidalger @scottsb Sorry for the confusion. This ticket was closed mistakenly. We will take a look at it again and let you know.

[ghost]

Hi @scottsb , thank you for your report. We've acknowledged the issue and added to our backlog.

[kirkmadera]

We also build separate from deploy and ran into this issue. We are expecting pub/static to actually be static and read only. We have also made the workaround to make the cache dir writable. Would be cleaner if these merged files generated as part of static content deploy.

[scrivvles]

We are experiencing this same issue on our production Magento 2.2.6 server.

[gwharton]

Looking forward to an update on this. Running 2.2.7 on a cluster and had to disable merging of css, not because of non writeable /pub/static/_cache folder, as I have allowed it, but individual servers don't seem to be triggering the generation of the cache files. I seem to get many different min.css files generated, with different hash's, but not always the one that the browser is requesting. I'd love to see these generated in the static:content:deploy phase.

[kirkmadera]

I think I understand the reasoning behind this. It's because it has to create the merged file for every possible frontend page, which would be tough to generate statically, especially without a full running app present to crawl.

I honestly think documenting in Magento docs that pub/static/_cache needs to be writable and shared across all frontend servers would be enough.

[gwharton]

Why shared. Does magento keep track of which ones have been generated, and in effect only generates it on one of the cluster nodes and 404s it from the rest of the cluster as all cluster nodes think its already generated, not knowing it is only present on one node.

That would certainly explain what i am seeing.

[scottsb]

I was mulling over this last night and was thinking of how this could be solved. This is rough and may not be workable, but it might be a start:

1. Change the generation of the files to happen on-demand rather than as part of the triggering page request. This would allow whatever server receives the request to generate the file locally, even if it's a different server than triggered it originally. This file could then be cached using the general cache config (such as in Redis or at least in the expected var cache directory if using disk). In order for this to work the URL either has to include sufficient data or reference some cache key (which would be stored during the initial page request).

2. The first step alone would avoid the present filesystem permissions issue but at the significant cost of requiring a Magento boot to serve the merged files, even if actual processing is quick once cached. So the next step would be changing the default Varnish VCL config to include these merged static files. (Currently all non-HTML assets are excluded by default.) This only solves the problem for those using Varnish, but that's recommended config for prod in any case.

A different, more exotic option would be to keep the merging in the initial page request but to push the data into the full page cache inline. This wouldn't be hard for Magento's built-in FPC, but for Varnish it would require us to push data out to it rather than waiting on it to cache while proxying. This is possible but is a rather non-standard approach.

[scottsb]

@gwharton Yes, Magento merges the file as part of the page request and saves it the local disk. It's expected that the file is then served directly by the web server as a static file from then on (it never routes through PHP/Magento). But if the merged asset request goes to a different server in the cluster, then it won't be present locally if the directory isn't shared.

[gwharton]

@scottsb Yeah, I just deployed a shared writable pub/static/_cache folder across my cluster and re-enabled merging of css files and all seems ok.

[geerlingguy]

I've also had to mount pub/static/_cache as a shared folder in my Kubernetes environment since any time more than one Pod (container) is running, the file will be generated on one, but not another, and that kinda messes things up—plus I have to make sure it's writeable (which it wasn't earlier).

[geerlingguy]

To follow up to the previous commit, I can't actually mount pub/static/_cache into my containers because when Magento rebuilds static assets it tries to delete the _cache directory, causing a fatal error, because NFS mounts cannot be deleted, you get a "device is busy" exception.

So the next best compromise is mounting all of pub/static on all my containers. But if you do this, make sure the .htaccess file that ships with Magento core (which has rewrite rules for the 'version' directory for assets) is copied into the shared pub/static directory, otherwise all your static assets will be available, but 404'ing because Magento loads them at paths like pub/static/version1548342577/frontend/ (note that versionXYZ is not an actual directory).

[dooblem]

I'm having the same problem. I will put static/_cache in shared storage, writable by www-data. static appears to be not static...

[gwharton]

Depending on how many modules/extensions you have, you may find it easier to disable merging of css (and thus there is no static/_cache folder) and manually streamline your css instead.

There may be limited benefit to merging css with http2 anyway.

[Janagiramramakrishnan]

I have enabled the merge JS & CSS, minification JS & CSS and bundled JS options. I have set up multiple instances and turned on the cloud front caching in AWS. But whenever the deployment is initiated through AWS code pipeline and merged JS and CSS files are missing and the website page is broken.

Current Result:

• Checked the pub/static/_cache/merged/ folder from instance 1 and instance 2. But some merged JS and CSS files are missing.

Expected / Actual Result:

• Check the pub/static/_cache/merged/ folder from instance 1 and instance 2. Both instances should have the same merged JS and CSS files.

[csdougliss]

What is the current status of this? I share pub/media across servers, but not pub/static.

After a deploy, I have to clear the cache on all servers otherwise CSS is not loaded.

I could add back static content deploy again in capistrano if required.

[SeeniBorn]

I'm also facing the same issue. We have multiple instances with AWS. Post deploy my website broken due to 404's on css and js files. Will the issue be solved when we share the pub/static/_cache across all instances ?

[gwharton]

@SeeniBorn

Either turn off CSS merging, or pub/static/_cache needs to be writable by and shared across all instances.

[SeeniBorn]

@gwharton Turning off CSS merging will affect the site performance right? If so, I will go for shared across all instances.

[csdougliss]

You can't share pub/static/_cache without sharing pub/static.

Is that what's needed?

[gwharton]

Depends on how many modules you have. I just put up with the fact that i have about 10 css files instead of 1 merged and dont need shared static. Has little effect with http/2.0 anyway. What has far more of an effect is not being able to merge js effectively as there are far far more of them.

[gwharton]

My pub/static folder is uploaded to a read only cdn drive at deploy time so i cant have a shared writeable area under pub/static using that deploy method. My frontends dont even have the pub/static folder on them.

[SeeniBorn]

What is the cdn provider you are using?

[gwharton]

My servers are on google compute engine so i just upload static content to a google storage bucket configured for direct public access. Handles all the caching and scaling and offloads the majority from your server. Works quite well (given a few tweaks)

[csdougliss]

Is this the kind of error everyone is seeing:

[2019-06-06 12:40:28] report.ERROR: Requested path '_cache/merged/3be83f6cdb1c6b4e61ac4c5ee753c201.min.css' is wrong. [] []
[2019-06-06 12:40:28] report.CRITICAL: Requested path '_cache/merged/3be83f6cdb1c6b4e61ac4c5ee753c201.min.css' is wrong. [] []
[2019-06-06 12:40:28] report.ERROR: Requested path '_cache/merged/5bebc9bda012302869ab3ecd7b2a9006.min.js' is wrong. [] []
[2019-06-06 12:40:28] report.CRITICAL: Requested path '_cache/merged/5bebc9bda012302869ab3ecd7b2a9006.min.js' is wrong. [] []
[2019-06-06 12:41:58] report.ERROR: Requested path '_cache/merged/3be83f6cdb1c6b4e61ac4c5ee753c201.min.css' is wrong. [] []
[2019-06-06 12:41:58] report.CRITICAL: Requested path '_cache/merged/3be83f6cdb1c6b4e61ac4c5ee753c201.min.css' is wrong. [] []
[2019-06-06 12:41:58] report.ERROR: Requested path '_cache/merged/5bebc9bda012302869ab3ecd7b2a9006.min.js' is wrong. [] []
[2019-06-06 12:41:58] report.CRITICAL: Requested path '_cache/merged/5bebc9bda012302869ab3ecd7b2a9006.min.js' is wrong. [] []

[Thomas-Papiernik-Insign]

+1. Any workaround ?

[vasusundar]

@SeeniBorn / @craigcarnell, Solution for this one is that we need to share the same pub/static for the different instance in AWS. Is this solution will work?.

[LSERRE]

+1 , This is still an issue with Magento 2.3.1

[gjportegies]

I'm having this issue as wel on Magento 2.3.4 after enabling CSS merge. Because of this it looks like my images are dissapearing on PDP.

[erfanimani]

Same here — issue is happening on 2.3.4.

[hakro]

Having the issue on 2.3.4 as well. The static-content deployment happens on the CI, which will build a docker image. This image does not have the pub/static/_cache directory. If I deploy a first container, it works well, and the _cache directory is generated somehow. If I deploy a second container, from exactly the same image, the _cache directory is not there, so CSS and JS fails. I think sharing the _cache is a workaround, but is not a good solution. The whole purpose of it is to increase performance, if this cache lives on NFS, it would defeat the purpose, independently of whether you have a CDN on top or not. Also, even with HTTP2, downloading a single file is always better that downloading 15 files, so disabling bundling is not ideal either. I think every instance should have it's own _cache directory, just like it's own static assets.

[rioux602]

Having the issue on 2.3.4 as well. The static-content deployment happens on the CI, which will build a docker image. This image does not have the pub/static/_cache directory. If I deploy a first container, it works well, and the _cache directory is generated somehow. If I deploy a second container, from exactly the same image, the _cache directory is not there, so CSS and JS fails. I think sharing the _cache is a workaround, but is not a good solution. The whole purpose of it is to increase performance, if this cache lives on NFS, it would defeat the purpose, independently of whether you have a CDN on top or not. Also, even with HTTP2, downloading a single file is always better that downloading 15 files, so disabling bundling is not ideal either. I think every instance should have it's own _cache directory, just like it's own static assets.

+1

[dev-ajaysaini]

static-content:deploy command doesn't generates any file under pub/static/_cache/merged in Magento 2.4.0

[gwharton]

@webkul-ajaysaini You are correct. The frontend does it as and when requests come in.

[trinhpham]

@webkul-ajaysaini You are correct. The frontend does it as and when requests come in.

That should be a problem, which forces us to share the static folder between web nodes => the bottle-neck is becoming to the storage sharing system (we - like others - are using NFS). IMO, web nodes should have their own files system for its best performance.

[nemke82]

And if a folder does not exist (pub/static/_cache), Magento creates _cache folder in var/tmp directory which grows unlimited.

var/tmp $ ls _cache

Is fix only to create the folder manually or?

[harrigo]

Not sure if my issue is slightly different, im on 2.3.5-p1 Community and have locked my permissions down for those folders as per DevDocs and for just one module I have the following error sometimes generated in exception.log:

[2021-04-16 08:08:58] main.CRITICAL: The path "/home/sites/magento/public_html/var/view_preprocessed/pub/static/frontend/Buzz/buzzluma/en_GB/Harrigo_CallForPrice/css/style.min.css" is not writable. {"exception":"[object] (Magento\\Framework\\Exception\\FileSystemException(code: 0): The path \"/home/sites/magento/public_html/var/view_preprocessed/pub/static/frontend/Buzz/buzzluma/en_GB/Harrigo_CallForPrice/css/style.min.css\" is not writable. at /home/sites/magento/public_html/vendor/magento/framework/Filesystem/Directory/Write.php:57)"} []

Seems the file is present in pub static and should also be merged as referenced in default_head_blocks.xml file not sure why Magento trying to write it in there?

UPDATE

Yeah i also get the same error however for pub/static/_cache which i guess is same as above.

pub/static/_cache" cannot be created Warning!mkdir(): Permission denied {

Don't really wanna open up permissions on those directories but feel I might have to unless there is a workaround?

[github-jira-sync-bot]

:white_check_mark: Jira issue https://jira.corp.magento.com/browse/AC-1005 is successfully created for this GitHub issue.

[m2-assistant[bot]]

:white_check_mark: Confirmed by @engcom-Hotel. Thank you for verifying the issue.
Issue Available: @engcom-Hotel, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[onlinebizsoft]

Sometime we get empty minify JS cached on cloudfront (path is located at /pub/static/_cache) so ideally deploy-static process should build these ready

[lbajsarowicz]

2.4.3, 2022 issue is still there.

[bilalusean]

Any solution? For me, it is happening in Magento 2.4.2. I don't see any merged CSS added inside the HTML head tag. Also, it creates files in the var/tmp folder on every request. It grows to 300GB within a day and it down the server (No Space left). Right now, we schedule cron to clear the var/tmp folder to fix the "No space left" issue.

Load the home page first time

/var/tmp/_cache$ find . -type f ./merged/496119470a3f8db183209c81713b1f0e.min.css_OarcqoTd0W5G4yREWzaRbLgkzP9XwoVf **Load the same page scecond time** /var/tmp/_cache$ find . -type f ./merged/496119470a3f8db183209c81713b1f0e.min.css_OarcqoTd0W5G4yREWzaRbLgkzP9XwoVf ./merged/84ca4336623d8f0129099e328566962c.min.css_WvL2yQC0FIhpRDk8CGlSguKDe69VGWny

[Stoyvo]

Magento 2.4.4 - Issue still present.

CSS/JS should never be generated on web-request, these are static files, they do not change, having this part of the web request is a critical design flaw. It severely affects containerized applications and forces infrastructure work-arounds.

Merged JS / CSS should be built in static content deploy and not generated by frontend request.

[mattgrul]

With the current "PR Created" label being added I'm hoping we are close to a possible solution for this issue. However I can't seem to find any corresponding PR, does anyone have a link?