maxbanton / cwh

Amazon Web Services CloudWatch Logs Handler for Monolog library
MIT License
417 stars 83 forks source link

[BUG] Logs are not sent in Laravel queues #95

Open antoinelame opened 3 years ago

antoinelame commented 3 years ago

Description

Logs from an asynchronous job in a Laravel queue are not sent to CloudWatch until you shutdown the queue process.

This is because php artisan queue:work does not close the process. Queue workers do not "reboot" the framework before processing each job.

How to reproduce

  1. Create a new job:
<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;

class TestJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function handle()
    {
        Log::info('The job has been executed');
    }
}
  1. Create a new route:
use App\Jobs\TestJob;

Route::get('dispatch-job', function () {
    TestJob::dispatch();
});
  1. Run the queue:
php artisan queue:work

Note: make sure that in your .env you do not use the sync queue driver.

  1. Call GET /dispatch-job

You will receive the logs in CloudWatch only when you'll interrupt the queue:work command. Not before.

Related GitHub issues

mugisham commented 3 years ago

Running php artisan queue:listen. seems to work but this is a less than ideal solution. This closes after each job and sends the logs, is there a way it can keep working without closing after each job as done by php artisan queue:work

ninjacoder96 commented 3 years ago

any update on this? we are encountering this as well.

olivier1208 commented 3 years ago

Finally found THE trick Change the buffer size to 1 !! That's it !

mugisham commented 3 years ago

Finally found THE trick

Change the buffer size to 1 !!

That's it !

What do you mean?

olivier1208 commented 3 years ago

$handler = new CloudWatch($client, $groupName, $streamName, $retentionDays, 10000, $tags); to $handler = new CloudWatch($client, $groupName, $streamName, $retentionDays, 1, $tags);

If you look at the protected function write in Cloudwatch.php in the package you'll see the following

if (count($this->buffer) >= $this->batchSize) { $this->flushBuffer(); }

That's why

mugisham commented 2 years ago

Does this affect performance sending a log at a time

mugisham commented 2 years ago

https://stackoverflow.com/questions/46931742/cloudwatch-agent-batch-size-equal-to-1-is-it-a-bad-idea