spatie / mailcoach-support

Questions and support for Mailcoach
https://mailcoach.app
31 stars 2 forks source link

Entire server freezes #233

Closed leveredge closed 3 years ago

leveredge commented 3 years ago

I installed the package version of mailcoach. I've been playing around with it. Seemingly at random, it causes the entire server to freeze. This has occured twice in the last few hours.

I'm hosting it on a t3.xlarge instance setup by Laravel Forge on AWS. Fairly standard setup: redis, horizon, mailcoach, mysql database on the server itself.

The freeze occurs as I'm setting up the campaigns i.e. nothing intensive is expected to be happening like sending thousands of emails etc. I'm simply editing the email content. I'm using the UnLayerEditor.

It seems to work fine for a bit and then all of a sudden it will freeze the entire server. Other sites on the server are also impacted and stop responding. The only solution is to restart the server and then everything starts working again. I don't see any messages in laravel.log

I'm not sure what might be causing the issue as it's not repeatable. Once I restart the server, I can do the same actions I was doing when it froze and it continues to work just fine.

This is on a relatively fresh install of Laravel. I have 3 routes on it so quite surprised to see this issue. Just a landing page, a form and a success page. I used the laravel/ui package for initial auth setup. It would be hard to imagine my application is somehow unique and causing this issue given we haven't really started building the application yet.

Version Information: PHP 7.4 mailcoach 2.23.2 (just purchased today)

Edit: Looking at CPU Utilization, I see this: Screen Shot 2020-08-04 at 9 00 23 PM

Makes me wonder if the "$schedule->command('mailcoach:send-campaign-summary-mail')->hourly();" is possibly causing the issue.

Issue continues. Third freeze. Same spike in CPU usage: Screen Shot 2020-08-04 at 9 45 13 PM

Digging into it a little further, I can see that the CPU is being used by kswapd0 process. This is what I get when I try to look at memory usage: Screen Shot 2020-08-04 at 10 42 19 PM

Edit 3: Sorry about opening this issue. I realized this was a mistake I made installing horizon. I added a scheduled job that kept spawning new workers every minute. 14 workers every minute results in the server crashing every hour or so... Closing the issue now

freekmurze commented 3 years ago

Could you share the solution? Might be helpfull for others.

On Wed, 5 Aug 2020 at 05:17, LeverEdge notifications@github.com wrote:

Closed #233 https://github.com/spatie/mailcoach-support/issues/233.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/spatie/mailcoach-support/issues/233#event-3622827338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADWEDMIMHX6J6MP2O2DM4LR7DFLJANCNFSM4PU6IA2Q .

-- Freek Van der Herten https://spatie.be +32 495 84 27 91

leveredge commented 3 years ago

Yes, of course.

If you are setting up Horizon using Forge, I recommend following Taylor's medium article. I was following the instructions on Laravel's site. While it's correct, I incorrectly interpreted that I should be running the php artisan horizon command every minute so I set it up as a CRON job rather than a supervisord process where just one process is running and keeps running.

If you run a CRON job, then, it keeps creating new horizon processes, and launching new workers. Each worker uses some amount of memory. As the minutes pass, you end up with hundreds / thousands of workers and the virtual machines memory is entirely consumed. This results in kswapd0 kicking in and trying to save the processes from crashing. However, the performance seriously suffers to the point where it is no longer possible to get a response from the server.