Closed Eddcapone closed 4 years ago
Hi @Eddcapone. Thank you for your report. To help us process this issue please make sure that you provided the following information:
Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:
@magento give me 2.4-develop instance
- upcoming 2.4.x release
For more details, please, review the Magento Contributor Assistant documentation.
Please, add a comment to assign the issue: @magento I am working on this
Join Magento Community Engineering Slack and ask your questions in #github channel.
:warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.
:clock10: You can find the schedule on the Magento Community Calendar page.
:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.
:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel
:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel
IMO this is more related with infastructure how to delivery code than related with magento problem In real case process more complicated than normal deploy zero-downtime = you can not run these commands that you posted on live system. So basically you need to have pre-production system deploy on that and apply to real live production
"So basically you need to have pre-production system deploy on that and apply to real live production"
Like described in my workflow example?
Hi @Eddcapone, You have to separate build preparation and production itself. More details here: https://devdocs.magento.com/guides/v2.4/config-guide/deployment/ In general it should look like this:
For example you can look at https://github.com/jalogut/magento2-deployer-plus that supports zero downtime deployments
Are you using magento cloud, some magento compatible hosting or developing infra yourself?
—- Technically you could run all the commands on your development env, but you still need to execute some commands on your production env as shown on image above, but it’s not recommended solution
Hi @ihor-sviziev. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:
[ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).Details
If the issue has a valid description, the label Issue: Format is valid
will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid
appears.
[ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description
label to the issue by yourself.
[ ] 3. Add Component: XXXXX
label(s) to the ticket, indicating the components it may be related to.
[ ] 4. Verify that the issue is reproducible on 2.4-develop
branchDetails
- Add the comment @magento give me 2.4-develop instance
to deploy test instance on Magento infrastructure.
- If the issue is reproducible on 2.4-develop
branch, please, add the label Reproduced on 2.4.x
.
- If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
[ ] 5. Add label Issue: Confirmed
once verification is complete.
[ ] 6. Make sure that automatic system confirms that report has been added to the backlog.
@ihor-sviziev Thank you. We are hosted on Rackspeed.
I have a few questions:
Where did you read about this, it is not described in the documentation entry which you have posted. How is it done? How do I know if the queue workers are even running atm?
I guess this is done by calling php bin/magento setup:upgrade --keep-generated
?
If I execute this command, then my site stops loading until the command run through, so I will have downtime.
Can you please explain what this means in this context?
About message queue in general described in https://devdocs.magento.com/guides/v2.4/config-guide/mq/manage-message-queues.html It is available not only in EE since magento 2.3. By default magento runs message queue consumers automatically with a cron job, but you can customize this behavior if needed. Usually we’re killing all message queue processes just before pulling new code to production infra.
Yes, it’s correct, you need to run php bin/magento setup:upgrade --keep-generated
.
If you have any issues - please provide any logs and I’ll be able to provide typical reason for such behavior
As a build system don’t have any db - we need to provide it some basic values, like websites, locales, used themes, is is generated by php bin/magento app:config:dump
and could be adjusted if needed.
I found quite good example here: https://blog.e-zest.com/magento-2-pipeline-deployment-with-minimum-downtime
Hi @Eddcapone, As I understood - my responses fixed your issues? Can we close the issue?
Not yet. I have some questions left.
How to kill/stop/pause queue workers. It is not described in the documentation entry.
If I understand it correctly, then we should not make any changes in production only in development.
We are supposed to dump the config with php bin/magento app:config:dump
and push the resulting file app/etc/config.php to the repository (e.g. git) and then pull this file in production. But this file contains sensitive data like password for Elasticsearch, this data should not be pushed to git. How are we supposed to handle sensitive data?
About killing the cron jobs and message queues - you can see how magento does it in their deployment tool named ece tools Here is how they detect which processes need to be killed https://github.com/magento/ece-tools/blob/9d201d782d976012ab4a90d9dbb6a4a21999da8c/src/Util/BackgroundProcess.php#L51 And here is how they actually killing them: https://github.com/magento/ece-tools/blob/9d201d782d976012ab4a90d9dbb6a4a21999da8c/src/Util/BackgroundProcess.php#L77
About dumping sensitive data - you could remove whole section “system”, it should work without it
Production system Your live store. You should make minimal configuration changes here and no changes to: websites stores store views products product view settings catalog categories category view settings. You should make all those types of changes in your development system.
@Eddcapone
I think these config params should be marked as "system specific", if they're not - seems like it's a bug. https://devdocs.magento.com/guides/v2.4/config-guide/prod/config-reference-sens.html
"we're doing the same way on Magento OS, I believe for EE it should be the same." I still don't understand how you are doing it. Do you call a command or a script to stop queue workers? How is it done?
We just using custom bash script that does the same commands as I listed above
On Fri, 4 Sep 2020 at 15:21 Eddcapone notifications@github.com wrote:
"we're doing the same way on Magento OS, I believe for EE it should be the same."
I still don't understand how you are doing it. Do you call a command or a script to stop queue workers? How is it done?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/magento/magento2/issues/29819#issuecomment-687109215, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOJOUKEGVDDWYYKCICQAQLSEDLVLANCNFSM4QQVEWQA .
Oh... so there is not even something official from magento? What if I just not stop the queue workers?
I think it’s just not documented. If you’ll not stop it - potentially you could have multiple workers running in parallel that will eat your server's memory, I don’t think there might be some other issues
One last question. In the diagram you posted, it says "Stop Queue Workers". But are we supposed to start them again at some point or does this happen automatically? And do you mind to share your bash script for stopping queue workers?
@Eddcapone, there is a feature request pending to introduce an official way to stop queue consumers, but it's not implemented yet. See "Problem 2" (A solution to "Problem 1" will make its introduction in Magento 2.4.1 btw, but it's not 100% implemented like I wanted it too, but it should be sufficient for most people, I haven't found time yet to properly report what's wrong with it yet, I'll try to do this one day, but I'm really busy)
For now, if you haven't changed the default configuration, after you killed the consumers using some script, they will spawn again by the cron system.
See here for some script we've been using to kill them: https://github.com/davidalger/capistrano-magento2/issues/133#issuecomment-537158716
@ihor-sviziev 1. In your "deploy" phase, you have enabled the maintenance mode then disable it at the end of the process so I understand that "maintenance" mode is equal as downtime
@onlinebizsoft, basically setup upgrade (with keep generated flag) is running on the add/update configuration phase on production environment. And yes, when the maintenance mode enabled - you will have downtime. As I said before, there is a way to skip setup upgrade when no changes in configuration and not data/schema upgrades. You can see example of such implementation in the deployer (I posted link to it in past)
@ihor-sviziev in my opinion, we can't call that as zero downtime which misleads to readers. I think we should have something like this installed in the system https://github.com/zepgram/module-zero-downtime-deployment
Then we don't need enable/disable maintenance during "setup:upgrade".
@onlinebizsoft in case when we don't have any config changes as data/schema upgrades - we shouldn't enable/disable maintenance as well as running setup upgrade. That means that your visitors will immediately see a new version w/o any interruptions.
This mechanism was introduced in order to prevent having not expected issues. I'm not recommend disabling it without really knowing what you're doing.
Summary (*)
Lets say you have two environments. A dev and a production. Now on dev make a change in your theme or in a module, and push the theme/module into production and execute the commands:
Now reload the frontend a second after firing up the command and you will get:
There has been an error processing your request
How are you supposed to bring the change to your production environment, without any downtime?
Examples (*)
Proposed solution
Is my workflow below good or bad and how should it be done instead?
WORKFLOW
I setup my project like this:
On Dev I am developing. If I am done with my development, then I push it via git to the branch
master
.Then I switch to Staging where the branch
master
is also checked out, and executegit pull
.Now I execute my commands to deploy static files. I am executing
grunt refresh
but you can also use(de_DE for german shops...) if you don't use grunt.
Important: Make sure that your
production
branch is not ignoring the folderspub/static/
andgenerated/
.Now commit the generated files to the branch
production
and push the branch. Then switch to master branch again.production
is checked out, executegit pull
and then clear the cache withphp bin/magento c:c
. Your new changes should be live now.EDIT: It won't work this way, since you also have to execute
php bin/magento setup:upgrade
in production, which will cause downtime.Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.