Deployment and Front End Development

[Khaleel]

Hi

Where do I begin to vent my frustrations with this platform? Firstly I will bring to your attention this Stackoverflow bounty which no one from the Magento Community gave an answer for that I a hoping will garner here: http://stackoverflow.com/questions/38589283/a-faster-route-to-deploying-static-content-in-magento-2-dev-to-live-etc?noredirect=1#comment65177020_38589283

1. Why is it that the Magento Team thought it was acceptable to turn a live website into Maintenance mode to carry out a deployment? Our company makes serious money and it is just - absurd - to consider ever placing our site in downtime (at any time or day).

We literally ship our code to git and then pull on the live server and then deploy static content and this takes an incredible amount of time. Just this week, our developers made a few mistakes and we had to re-deploy twice. 45 mins of no sales. Clearly this is not a solution. There must be something were not doing correctly; but your own documentation, recommends it.

1. @alankent who says 'I am not a frontend dev' Clearly not. Otherwise you would have chosen Sass and dropped the insane idea of Magento UI, which no one really knows what it is. And then further you would have scrapped RequireJS and all the insane (not SEO friendly) KnockoutJS bloat even on the cart. Please can you answer regarding this. It is been raised across the web on many threads on Stackoverflow which my bounty failed and even on this post: https://github.com/magento/magento2/issues/2104 other devs see the insane feature of this.

Why is it sooooo slow to make changes to Less and JS even in a developer mode node. You have to recompile files (no Livereload; which I get people not a fan of) but seriously? Make a padding or margin change; delete all var folders, re compile static files for all languages and then run grunt to generate less etc - seriously? ALL our Front End Engineers are strong with JavaScript and CSS and most are (including me) previously Back End Developers. It is insane.

Please can this be addressed? Gulp and Grunt is one argument but the process of static deploy on development mode is one thing - a live environment? Seriously?

[boldhedgehog]

Hi @Khaleel,

We've ended up with a 'build' server which does composer install, compile and deploy steps, then force commits vendor and static directories to git. This commit is then tagged and we switch the production environment to this tag. This way you don't have to compile or deploy the static files on the production server with maintenance. Git workflow described here is inaccurate, of course, but I hope you get the idea.

[Khaleel]

Hey @boldhedgehog

Can you explain a little further in detail?

What about the LESS and JS - some of my coders hate the speed of making things in M2

[boldhedgehog]

@Khaleel

I'm not sure what git workflow you have for deployments, still the main idea is to create a special build branch, install all composer packages, perform static content deploy and compilation, then force commit and push all generated files into this branch, then tag it. The tag is then fetched on the production and the file base is switched to this tag, thus we update all generated files (/vendor/, static, di, compiled files) in 1 'atomic' git operation. With this scheme it does not really matter how much time is spent to prepare the generated files, as git check to the tag usually takes seconds.

Our FEDs use grunt to compile less files: grunt watch, less, exec tasks, as the documentation describes; and this is not that terribly slow.

I hope this helps.

[choukalos]

CC'ing @xcomSteveJohnson into this thread to add to our documentation.

Hi @boldhedgehog - any suggestions on how to improve the build server flow? We've looked at it internally and are exploring some improvements but would love to get your insights. CC'ing @tanya-soroka and @piotrekkaminski.

[piotrekkaminski]

also cc @bwaters

[hostep]

Hi all

I kind of agree with @Khaleel but kind of also don't, because I think it all boils down to the fact that for newcomers in the Magento 2 ecosystem it's really hard to understand what is happening all over the place, and this is (I think) because some of the documentation is lacking a bit in certain areas.

So for the deployments. @Khaleel: it is possible to deploy on your production server without using a build server, but you'll need to reorganize your filestructure a bit to execute a bunch of commands in a new release directory (composer install, setup:di:compile, setup:static-content:deploy) and when that is done, create a symlink from that new releases directory to your docroot, put the shop in maintenance mode, run setup:upgrade --keep-generated and disable maintenance mode again. You also need a 'shared' directory for all the media files that get uploaded (and some other stuff) which needs to stay put in between deploys. We have this kind of deployment flow and it works (most of the time, not always), but takes a long time, about 8 minutes every deploy with a downtime of about 6 to 8 seconds, which are all not very ideal. If you don't want to write your own deploy scripts, you can take a look at the magento2 capistrano scripts from @davidalger over here: https://github.com/davidalger/capistrano-magento2 We use Fabric ourselves and have written all these scripts by hand.

Now, it would be great if Magento could write documentation about this to see what they recommend we do for deployments. Because they have more inside information about how everything works, they might have a better knowledge of how we need to do this. There is already a pending issue over at the devdocs for this. For the part where the above deployment flow doesn't always work in our case: if you create a new theme, you'll need to run setup:upgrade before setup:static-content:deploy otherwise the latter one crashes. So in this case I think the Magento guys hopefully have more insights into when and in what order you'll need to run all these deployment steps, because we just have to keep guessing...

@choukalos: for an improvement in the deployment flow: I would really really really like you to get rid of locale specific css/less compilation. It makes no sense in my opinion. I understand you need it for javascript translations and images in different locales with text written on them. But I don't understand why we need it for css/less files. If you could remove this (using a configuration option, to maintain backwards compatible) it would improve deployment speeds a lot I think. It would also help frontend development, if they switch to a different locale not everything has to be recreated. Also this PR will help a lot: https://github.com/magento/magento2/pull/4294 And I've written a small module myself which switches the less.php compiler into using less.js which also improves deployment speeds (but you'll need to be able to install nodejs on your server).

Then for the frontend development part.

I think it makes sense if somebody wrote some documentation about what happens with all the frontend assets when you refresh your browser when you are working in developer mode. Symlinks, var/view_processed, pub/static, ... Also what happens if a file already exists, how does Magento determine it should be recreated or not, stuff like that.

And if you could somehow improve the speed of compiling less into css (which now takes about 8 seconds if I remember correctly) that would also be very helpful. 8 seconds is a lot, it completely breaks your concentration if you have to sit and wait for so long. In Magento 1 we also did less precompiling, and it only took 2 or 3 seconds, which is a lot better.

And what I think will also help out a lot, is if you let javascript bundling work in developer mode. For example: if you now open the cart or checkout page in your browser, it downloads (and generates if they don't exist yet) like 200 or 300 javascript files, which takes a looooooooooong time (more then a minute). If we could get javascript bundling work in developer mode, it would only download/generate 3 javascript files (not exactly sure about the number) and the page refresh would be much faster. I do understand this won't be an easy feature to implement.

And we also need documentation of all the caches, what every cache is responsible for, what caches should we disable for certain scenario's. For example, for frontend development (css/less, changing phtml's, changing xml's, translations), I think it makes sense to disable these caches: layout, block_html, full_page, translate and leave the others enabled. This has some impact on the performance when refreshing the page every couple of minutes. And then you'll probably have another list of caches for people working on modules who don't have to work a lot on css/less, translations for example. So an overview of what each cache type actually caches and what caches you recommend we disable in certain scenario's would be helpful. I see some of my frontend colleagues disabling all caches so they don't have to manually refresh every time they change an xml file for example, and then they refresh, and sit and wait more time than actually needed, because just disabling some specific caches would be enough.

@xcomSteveJohnson: And then one last request: maybe create a weekly changelog of the devdocs somewhere. A lot of new content is added on a daily basis, but we can't just open devdocs every week and take a look at all the pages to see if something new has popped up somewhere. I think a weekly changelog with a short overview of what was done on the devdocs would help out a lot. I'm talking about mentioning big new topics being added. Important changes being made. But please don't add 'fixing typo's' to the changelog, that's not really needed :)

I hope this helps somewhat.

[Khaleel]

@hostep

Thank you so much for your messages and to all the above. What we are doing at the moment.

Development:

All developers have Magento 2 installed on their local machines - database releases are done weekly and images and media are already on Cloudinary as a CDN. We considered Rackspace/Akamai and may change to Ampliance in future but for now that is how it works. The developers have pretty powerful machines so they develop there code locally. Once done, they push and commit there changes to the staging repo. The developers have ALL got M2 configured in developer mode and still have to delete files and re build the cache libraries. This is what they do. None of the developers have less compilation working. Even in developer mode the LESS files do not compile automatically even with Magento 2 admin setup - is this is a bug. Even the JS. So they have to delete all files and re-compile. I am really happy that many other people here on Github are so open and share these same concerns. The FE code, KnockoutJS, requireJS, less is sooooo bloated. @alankent is pretty ignorant that he does not consider Front End Development the core component system to the platform. WordPress and many others are shipping to FE and JavaScript only servers with only API and JSON talking between. @alankent and many other Magento engineers (European and American based) who built the Front End system I do not think give much thought nor have the years of experience in ecommerce. Most of the developers are locked in the "Magento mindset"

bin/magento maintenance:enable
rm -rf var/cache var/page_cache var/view_preprocessed pub/static
mkdir var/cache var/page_cache var/view_preprocessed pub/static
bin/magento setup:static-content:deploy
bin/magento setup:static-content:deploy ar_SA
grunt less
bin/magento maintenance:disable

This is every time they want to see a change to less or JS. So most of the senior developers work directly in static files and then ship it back to the structure files in app for modules and themes. They hate and loathe this process and are already voting to our board to leave Magento. This takes 15mins max every single time.

Staging:

The staging environment gets a few "deploys" as we call them daily. A release manager goes onto the staging box (we use Digital Ocean) and have wildcard SSL, nginx, varnish and fpm and a few features like redis etc. Release manager executes the command below. Only difference is the upgrade on the CLI.

bin/magento maintenance:enable
rm -rf var/cache var/page_cache var/view_preprocessed pub/static
mkdir var/cache var/page_cache var/view_preprocessed pub/static
bin/magento setup:upgrade
bin/magento setup:static-content:deploy
bin/magento setup:static-content:deploy ar_SA
grunt less
bin/magento maintenance:disable

This takes around 15-20mins. Then we have QA and testing done on the staging server. We have around 14 stores in English and Arabic so various testing takes place; some automated some manual. Once it is good to go we go to the live nodes.

Node 2 and Node 1 are r-sync servers and load balanced. (We may switch off Node 1 and do a deployment on the Node 1 and once done enable it and allow Node 2 to download the new files) At the moment we have not done that so we have to do:

bin/magento maintenance:enable
rm -rf var/cache var/page_cache var/view_preprocessed pub/static
mkdir var/cache var/page_cache var/view_preprocessed pub/static
bin/magento setup:upgrade
bin/magento setup:static-content:deploy
bin/magento setup:static-content:deploy ar_SA
grunt less
bin/magento maintenance:disable

This is on our live machine. Keep in mind that any errors, changes even if the FE or BE makes errors we have to re-deploy from the staging. 25mins this process takes to execute one staging to live deployment.

CTOs from some big Magento 1.9 and EE versions asking reference:

Take your existing system offline by enabling maintenance mode using magento maintenance:enable. (DOWNTIME STARTS) Install the Magento software in a clean database (optionally using the --cleanup-database option). Import database data from your development system. Move your existing file system to an archive location. Move the file system you copied from your production system to the Magento docroot. Change to production mode and set file system permissions and ownership for production. Test everything thoroughly and carefully. Switch DNS to the new server. (DOWNTIME ENDS) Source: http://devdocs.magento.com/guides/v2.1/config-guide/prod/prod_deploy.html

We have to turn our live website offline and switch in maintenance mode and run a deployment and then switch live. Absurd. Who thought of this? Even in your documentation you mention this? What is the rational that this is considered even acceptable? Imagine turning eBay offline and telling customers we will be right back once we deploy some three new lines of CSS on our website.

[hostep]

Hi @Khaleel: I'm not going to try to help your specific case (because that's not what github issues are for), but let me give you this link, it more or less explains what you need to do to keep the downtime during deploys low enough (max 10 seconds mostly): http://www.classyllama.com/blog/deploying-magento-2-using-capistrano Don't copy/paste everything without thinking about it, try to understand what is happening and then make your own version of this. This might take you a couple of days to fully figure out, so be prepared for that. Also be aware that the article is tailored for Magento 2.0, not 2.1, so you might need to change some things here and there (like bin/magento setup:di:compile instead of bin/magento setup:di:compile-multi-tenant).

Btw: the article you refer to is only for the initial setup, not for continuous deployments.

Hope this helps.

Looking forward to see any kind of activity from Magento about these issues (I know it's weekend, don't worry :))

[Khaleel]

Hi @hostep

This seems the best way for us:

Node1 Node2 Node3 (all load balanced and sync files)

1. All Developers have their own nodes locally.
2. All Developers push to Git and The Release Manager merges and controls a staging site.
3. Testing on staging.
4. Release manager turns off Node 3. He pulls repo and runs static files and deployments. Done.
5. Release manager turns Node 1 and Node 2 off and turns Node 3 off. (11 seconds possible downtime)
6. Release manager runs sync for Node 1 and Node 2 which get the latest from Node 3. Once done switches these boxes back online.

Weekly task. Still awaiting feed back from Magento.

[stamster]

Have you guys tried to disable certain cache types in M2, at least during development? Our FE devs were happy after we disabled the following cache types:

• Layouts
• Blocks HTML output
• Collections Data
• Page Cache

Apart from that, I agree with @Khaleel regarding levels of insanity M2 introduces. :-1:

[Khaleel]

@stamster another thing I have noticed is that Magento are outsourcing a lot of European developers where English is not their main language. You often find with these developers that they are primarily solid Back End driven and have a focus of over engineering a product. Not sure why less and less American eBay developers are on the table any more.

If they had the years of experience working with actual ecommerce companies and working within commercial environments they would have foreseen more commercial based objectives. Again "closed mindset"

Still not official response. I have seen plugins and capistro and all the other tools and solutions but this still does not answer why or who placed the idea to turn a live website offline to apply changes.

[Zaylril]

@Khaleel In terms of the CSS, we have just taken to creating our own base theme fully in SASS and written our own tooling in Gulp. I'd also take a look here: https://github.com/SnowdogApps/magento2-frontools

They are working with Magento to convert M2 back into SASS and gulp.

[Khaleel]

@Zaylril thanks for the reply @andimov thanks for tagging.

@Zaylril the tools sound and look good. Is that a rumour or is it official that Magento 2 will go to into SASS and Gulp?

[Zaylril]

@Khaleel Yes it was confirmed a couple of months ago - please see https://github.com/magento/magento2/issues/1337

However I'm not sure about the Gulp side of things - i'm pretty sure the new tooling will be in Gulp though.

[stamster]

That is probably for 3.0 version or similar major version. 2.0.x and 2.1.x are too late for such a change IMHO.

[hostep]

I don't see why switching from less to sass or grunt to gulp will improve the frontend developer performance problems. Since they are basically the same things. It's not the tooling which is the problem, but how Magento is using the tools I think.

[Khaleel]

@stamster @Zaylril @hostep Aside from the development issues the deployment one is the biggest one.

A real life example of loosing clients. A UK ecommerce company we know currently 1.9 EE and are turning over $2-$3 million dollars per day.

When discussing a consultation on M2 and discussing deployment - just by the mention of downtime the client showed no interest in upgrading nor using the services. They actually have made extensive changes to core Magento and are very, very well known both in the retail and the development and Magento world; they could not fathom any downtime. Whatsoever. Ever. Period. They have no plans to upgrade.

Clients this big face commercial issues: M2 or DemandWare and things platforms like IBM WebSphere Commerce, Hybris, ATG etc

[Igloczek]

IMO there is one way to deploy M2 to production - have a middle step (CI/CD) in deployment pipeline, to run every necessary compilation and test steps, then save results as ZIP for future use.

Then you can take this ready to use code on production server (i.e. via Capistrano) and enable 1-2 min maintenance mode just for database updates (probably setup:upgrade will be enough)

@Zaylril @Khaleel Officially we as a Snowdog are leading community project moving M2 to SASS, but... It's kinda hard to push execution of this idea forward. Everybody have different needs, no one can really write down their point of view and possible solution (including myself), that's why this project is stuck in middle of nowhere 👎

I was talking about this project on #mm16de - YouTube - Slideshare

If you have any solid ideas to rebuild some parts related to styling / tooling and they may fit almost everybody (solutions providers and extension developers - I leave non-tech clients out of the target) just let me know, maybe together we can build something useful.

BTW. To be honest I'm even thinking about moving front-end out of Magento, build everything on top of bigger MV* JS framework and thread M2 just as an API provider.

[hostep]

Regarding the deployment issues, @piotrekkaminski & @xcomSteveJohnson & @choukalos: maybe you can ask your colleagues of the 'Magento Enterprise Cloud Edition' how they go about and do this? Are they for example able to upgrade the Magento version without any downtime? If yes: it would be great if this can be shared with the community about how we can accomplish this. Just an idea...

[Khaleel]

cc @alankent

[ghost]

@hostep Regarding this comment:

• What are all the caches for?

  Does that make it clear which ones to disable?

• Which directories do you clear and when?

[hostep]

Hi @xcomSteveJohnson

Those pages looks really useful! I have to admit, I didn't properly research to see if the topics I mentioned were actually missing. I wrongly thought that particular information wasn't in the devdocs yet, because it has been a few months since I went through them in depth (hence my request for a changelog of the devdocs).

Anyway, the caches explanation is very useful, especially with those recommendations about when and if you should flush a particular cache type (although some types - reflection, translate, config_integration_api and config_webservice) don't have that recommendation right now). Is there btw a difference when working in development mode and working in production mode about certain caches being automatically flushed (or not used)? If that is the case, that would also help if this was documented.

I would still like some kind of recommendation for different types of developers. So frontend developers (in my opinion) should probably best disable these types of cache while doing development: block_html, layout, full_page, translate. If you guys think this is useful, you should probably add it somewhere in the Frontend Developer Guide section in a new Setting up your local environment topic or something like that. This is just an idea, please consult with other people if this is useful or not.

While that other page you link to about clearing certain directories is more geared towards PHP developers. And that page is also really useful, nice to find this page, first time I see this, was definitely an interesting read!

Keep up the good work! :)

[piotrekkaminski]

Thank you for your submission.

We recently made some changes to the way we process GitHub submissions to more quickly identify and respond to core code issues.

Feature Requests and Improvements should now be submitted to the new Magento 2 Feature Requests and Improvements forum (see details here).

We are closing this GitHub ticket and have moved your request to the new forum.

[antonkril]

@Khaleel, can you please clarify this statement:

None of the developers have less compilation working. Even in developer mode the LESS files do not compile automatically even with Magento 2 admin setup

LESS compilation should work in developer mode. Actually you should not run setup:static-content:deploy in developer mode at all. This command is there for production.

With Grunt workflow, LESS files are automatically recompiled.

Also, why do you run grunt less on production? It's not needed after setup:static-content:deploy is run.

[Khaleel]

LESS compilation does not work in developer mode. Do I need to run a Grunt command or an additional setting - I edit various .less files in both app/code and app/design - they do not automatically merge and appear on the Front end staging server.

We do not always run grunt less - we use that as it breaks the minification on the server and easy for us to see.

[davidalger]

Please take additional discussion to the forums per the link provided by Piotr: https://community.magento.com/t5/Magento-2-Feature-Requests-and/Deployment-and-Front-End-Development/idi-p/44979