improve setup:static-content:deploy downtime period

[ioweb-gr]

Summary (*)

setup:static-content deploy is very annoying to use in production because it produces huge downtime for websites.

Examples (*)

Currently in order to force any changes in CSS no matter how small, change red to yellow for example requires the following process.

rm -rf var/view/preprocessed/* pub/static/frontend/*

Any other procedure will force obsolete files to be used. While everything is deleted there is downtime. While everything is recreated there is downtime. Depending on the server this can range from a few seconds to 10minutes+ and it's unacceptable for all users, customers and developers.

Proposed solution

Is it possible to change the way deployment is handled by using a basic way of temporary folders? For example everytime we issue the command setup:static-content:deploy a temporary folder is created to execute the procedure in and then the final contents are moved back to the locations we need.

This would mean that cached files and files that are currently being used in the website will be accessible during the whole deployment process without causing downtime. Afterwards, only renaming the top level directories and changing the version would be enough.

To depict better

Currently the pub/static folder structure in production mode looks like this

adminhtml _cache deployed_version.txt frontend

and in var/view/preprocessed we have the folder pub

So my recommendation would be, create the folder var/view/preprocessed/pub_version create the folders in pub/static folder

adminhtml_version
frontend_version

Do all the work in the temp folders, and then rename them accordingly.

Although it will not reduce the time for compilation it will still reduce the downtime.

[korostii]

Not sure what's your deploy flow, but there are means to reduce downtime to seconds by doing this from outside. The old-school approach goes like the this: you have your vhost folder replaced by a symlink to version_xxxxxxx folder and building the whole instance in such a new directory, then swapping the symlink to the new version's directory.

However, there's also a modern solution which would involve building this new version as an artifact on a separate machine and then pushing it to production host - this is the flow used by Magento Cloud out of the box. It's called "pipeline deployment" in Magento (available from 2.2 onwards) and there's quite a comprehensive guide in this regard at devdocs.

[ioweb-gr]

Hi @korostii I do understand your ideas and I've also read the various ways to "speed" it up. This doesn't mean that the original way should be left as is with no improvement to take such long times though as I see it. With that aside I'll mention why in some cases we're unable to have multiple systems for staging and building the files.

In order to use a build system on a separate environment you need to have enough resources for that. Some of our clients with Magento2 believe it or not are still in shared hosting environments or small VPS where they're limited by the number of inodes. Duplicating the whole environment pushes their limits so in order to avoid downtime they have to invest in a lot more money for external servers for building the files.

Also we have people hosted in cloud solutions where they are licensed to only host a single domain and not even staging sites / subdomains with limits on inodes as well.

For these use cases our only real option is to use the standard method magento 2 provides. Put the site in maintenance mode, deploy, exit maintenance mode.

This thread is meant to suggest improving a feature that's already there, that the command setup:static-content:deploy is slow and causing huge downtimes in limited resource systems.

Finally not all eshops have a lot of orders / day so they shouldn't need to pay that much for so many resources just because own a magento 2 eshop imho otherwise our first question in all customers will be, how much are you willing to pay for hosting before choosing magento 2 and suggest prestashop as alternative. Let me know what you think.

[hostep]

@ioweb-gr: you can also "build" on the webserver itself, in a separate directory and then switch the docroot symlink once that build is done. You don't necessarily need to build on a different server.

The upside of this is that the downtime is very small, whereas if you just have to rebuild in the same directory, you'll have to put the shop in a much longer maintenance window.

As for the inode problem, we remove the following directories after running composer install during a "build", since these contain many (> 18.000) files which aren't needed on a production environment:

• .github/
• dev/
• lib/web/css/docs/
• phpserver/
• vendor/magento/magento2-base/

Not sure if this is helpful, but this works in our case.

I understand what you are asking, but having temporary locations for static content deploy won't be enough, you need to handle temporary locations for SCD, for caches to be stored while building, for generating code (setup:di:compile), for running composer install, ... I'm assuming you aren't only changing frontend code but also some other code once in a while?

So what you want is one or multiple extra options in the setup:static-content:deploy command where you can specify some other directories to use while building. I think adding that to Magento's code probably wouldn't be that difficult and you can certainly try making a pull request (or a custom module) to have that behavior, but I have the feeling this change alone won't be enough in your case.