search

librecores / librecores-web

LibreCores Web Site
https://www.librecores.org/
Other
35 stars 30 forks source link

Refresh database connections in long-running consumers #406

Open imphil opened 5 years ago

imphil commented 5 years ago

We repeatedly get errors about dropped MySQL connections. They are the result of long-running connections coming from the RabbitMQ consumers. Example:

Exception(most recent call first)

ErrorException: Error while sending QUERY packet. PID=30007
#26 vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(117): handleError
#25 vendor/sentry/sentry/lib/Raven/ErrorHandler.php(0): execute
#24 vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(117): execute
#23 vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(903): executeQuery
#22 vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(733): load
#21 vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/Entity/BasicEntityPersister.php(751): loadById
#20 vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(460): find
#19 var/cache/prod/ContainerClgqxbg/EntityManager_9a5be93.php(88): find
#18 vendor/doctrine/orm/lib/Doctrine/ORM/EntityRepository.php(154): find
#17 src/Consumer/AbstractProjectUpdateConsumer.php(58): execute
#16 src/Consumer/AbstractProjectUpdateConsumer.php(0): call_user_func
#15 vendor/php-amqplib/rabbitmq-bundle/RabbitMq/Consumer.php(133): processMessageQueueCallback
#14 vendor/php-amqplib/rabbitmq-bundle/RabbitMq/Consumer.php(179): processMessage
#13 vendor/php-amqplib/rabbitmq-bundle/RabbitMq/Consumer.php(0): call_user_func
#12 vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AMQPChannel.php(1040): basic_deliver
#11 vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AMQPChannel.php(0): call_user_func
#10 vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AbstractChannel.php(218): dispatch
#9 vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AbstractChannel.php(373): wait
#8 vendor/php-amqplib/rabbitmq-bundle/RabbitMq/Consumer.php(88): consume
#7 vendor/php-amqplib/rabbitmq-bundle/Command/BaseConsumerCommand.php(90): execute
#6 vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php(255): run
#5 vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(987): doRunCommand
#4 vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php(86): doRunCommand
#3 vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(255): doRun
#2 vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php(74): doRun
#1 vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(148): run
#0 bin/console(26): null

The problem is explained in more detail at https://blog.vandenbrand.org/2015/01/09/symfony2-and-rabbitmq-lessons-learned/. A first workaround could be processing only one message a time, increase wait_timeout in MySQL, or disconnect/reconnect when starting to process a message.

imphil commented 5 years ago

Sentry issue: LIBRECORES-WEB-Z

imphil commented 5 years ago

Sentry issue: LIBRECORES-WEB-Q