Check why we see occasional timeouts with AMQP in the workers

[mrchrisadams]

We're seeing timeouts along the lines of when some domains are being checked:

PhpAmqpLib\Exception\AMQPTimeoutException: The connection timed out after 3 sec while awaiting incoming data

It doesn't seem to be tied to the domain. Maybe we need to check the max connections in rabbitmq?

[arendjantetteroo]

Looking at rollbar we see that this is not on the consumer end but on the producer end. It happens only on /greencheckmulti urls, meaning that possibly we take longer to respond than the timeout that is on the amqp connection that is waiting for the result.

This doesn't happen for every request, so possibly related to long taking dns/as lookups for a domain that was not cached before.

This means that a second request for this would work fine, but it's annoying.

[mrchrisadams]

Okay cool, so enqueue seems to have this idea of a heartbeat that it listens for from a consumer, for longer running tasks, and we'd need to pass in a set number of 'ticks' for a task to check for, before it considers a worker timed out.

I think this would give us more than 3 seconds if needed:

https://php-enqueue.github.io/transport/amqp_lib/#long-running-task-and-heartbeat-and-timeouts

$context = (new AmqpConnectionFactory('amqp:?heartbeat_on_tick=1'))->createContext();

$queue = $context->createQueue('a_queue');
$consumer = $context->createConsumer($queue);

$subscriptionConsumer = $context->createSubscriptionConsumer();
$subscriptionConsumer->subscribe($consumer, function(AmqpMessage $message, AmqpConsumer $consumer) {
    // ticks number should be adjusted.
    declare(ticks=1) {
        foreach (fetchHugeSet() as $item) {
            // cycle does something for a long time, much longer than amqp heartbeat.
        }
    }

    $consumer->acknowledge($message);

    return true;
});

$subscriptionConsumer->consume(10000);

function fetchHugeSet(): array {};

[arendjantetteroo]

Solved by #22 , we now use the amqp pecl extension which doesn't seem to exhibit this problem.