What is going on?

[Anoesj]

Our servers are currently going down one by one. Can you please let us know what to do?

[leevh]

Same here, self hosted, different datacenters, I'm freaking out

[Anoesj]

Some of our servers are back up again, but mysql won't boot on some servers. service mysql start does the trick on some servers, on others, this results in a big red failed! message.

[leevh]

Out of curiosity, my problems started with servers timing out when trying to visit sites. I then tried rebooting servers and ended up with 502 - bad gateway for all sites. Does that sound the same?

I've started restoring server backups in a panic

[Anoesj]

We had 502's as well. We then manually restarted php where it wasn't automatically restarted yet

[Anoesj]

FYI rebooting the servers works wonders

[reswild]

The same thing happened on 4 of my servers, and on two of them, it took a while to get mysql to start up again after the reboot.

[omega8cc]

That shouldn't happen, and it's entirely our fault, we are sorry about that. No need to panic, though, you just need to reboot your Linode or Droplet to get it back and fully working. We have fixed critical issues with Redis memory reported by many hosts earlier today, and inadvertently pushed the memory fix to our public mirrors, which resulted with this set for everyone, even if for most of the BOA servers it didn't break anything:

vm.overcommit_memory = 2
vm.overcommit_ratio = 100

It has been reverted to BOA default already, but you may need to power cycle your droplet or Linode.

vm.overcommit_memory = 0
vm.overcommit_ratio = 0

[Juc1]

Both my droplets went down with 502 bad gateway and "remote side unexpectedly closed network connection" in putty. I have done power cycle in Digital Ocean UI. I can now connect with putty but not yet loading in browser. Should I do something else or does it just take a while like reswild said?

[Juc1]

sites still not loading (HTTP ERROR 500) one hour after I did power cycle

$ cat /var/log/php/error_log_72 

[04-Oct-2020 00:44:19 UTC] PDOException: SQLSTATE[HY000] [2002] No such file or directory in /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Component/DependencyInjection/Container.php on line 255 #0 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/Database/Driver/mysql/Connection.php(416): PDO->__construct('mysql:host=loca...', 'example.com...', 'cnXExuvFj4', Array)
#1 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/Database/Database.php(374): Drupal\Core\Database\Driver\mysql\Connection::open(Array)
#2 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/Database/Database.php(169): Drupal\Core\Database\Database::openConnection('default', 'default')
#3 [internal function]: Drupal\Core\Database\Database::getConnection('default')
#4 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Component/DependencyInjection/Container.php(255): call_user_func_array(Array, Array)
#5 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Component/DependencyInjection/Container.php(173): Drupal\Component\DependencyInjection\Container->createService(Array, 'database')
#6 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/DrupalKernel.php(743): Drupal\Component\DependencyInjection\Container->get('database')
#7 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/DrupalKernel.php(716): Drupal\Core\DrupalKernel->handleException(Object(PDOException), Object(Symfony\Component\HttpFoundation\Request), 1)
#8 /data/disk/o1/static/platforms/892-250720/index.php(19): Drupal\Core\DrupalKernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#9 {main}
$

[tijmenve]

sites still not loading (HTTP ERROR 500) one hour after I did power cycle

$ cat /var/log/php/error_log_72 

[04-Oct-2020 00:44:19 UTC] PDOException: SQLSTATE[HY000] [2002] No such file or directory in /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Component/DependencyInjection/Container.php on line 255 #0 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/Database/Driver/mysql/Connection.php(416): PDO->__construct('mysql:host=loca...', 'example.com...', 'cnXExuvFj4', Array)
#1 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/Database/Database.php(374): Drupal\Core\Database\Driver\mysql\Connection::open(Array)
#2 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/Database/Database.php(169): Drupal\Core\Database\Database::openConnection('default', 'default')
#3 [internal function]: Drupal\Core\Database\Database::getConnection('default')
#4 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Component/DependencyInjection/Container.php(255): call_user_func_array(Array, Array)
#5 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Component/DependencyInjection/Container.php(173): Drupal\Component\DependencyInjection\Container->createService(Array, 'database')
#6 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/DrupalKernel.php(743): Drupal\Component\DependencyInjection\Container->get('database')
#7 /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/DrupalKernel.php(716): Drupal\Core\DrupalKernel->handleException(Object(PDOException), Object(Symfony\Component\HttpFoundation\Request), 1)
#8 /data/disk/o1/static/platforms/892-250720/index.php(19): Drupal\Core\DrupalKernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#9 {main}
$

Make sure mysql is running. I had to run service mysql start on a few droplets before everything was back to normal.

[Juc1]

@tijmenve thanks a lot both droplets now working - the first droplet started working with just the power cycle - so I thought the second would be the same but the second one needed power cycle + $ service mysql start

[Juc1]

PS I spoke too soon - when a few sites loaded I thought everything was ok but actually about half the sites (on the same platforms) are still not loading ("unexpected error"). Any ideas please on these errors...

$ cat /var/log/php/error_log_72 [04-Oct-2020 07:25:11 UTC] Uncaught PHP Exception Drupal\Component\Plugin\Exception\PluginNotFoundException: "The "node" entity type does not exist." at /data/disk/o1/static/platforms/892-250720/core/lib/Drupal/Core/Entity/EntityTypeManager.php line 150 waterloo:/var/log/php# [04-Oct-2020 07:25:03 UTC] Uncaught PHP Exception InvalidArgumentException: "Class "\Drupal\system\Controller\Http4xxController" does not exist." at /data/disk/o1/static/platforms/896-011020/core/lib/Drupal/Core/DependencyInjection/ClassResolver.php line 24

[Juc1]

I have fixed this with the hack of clone then delete the bad sites - all sites ok now.