laravel / horizon

Dashboard and code-driven configuration for Laravel queues.
https://laravel.com/docs/horizon
MIT License
3.87k stars 657 forks source link

Pending & completed list empty while workload count is correct #1035

Closed mrmonat closed 3 years ago

mrmonat commented 3 years ago

Description:

Pending / completed jobs don't show up in the list view. The counting of pending jobs does work on the dashboard. All jobs finish execution without problems / failures but are not visible in the pending / completed list view. Each Job only takes a few seconds so "Horizon Trimming" should not be the problem. Interestingly all jobs are visible in Telescope while pending and when completed. Strangely this only happens on the Debian 10 Production Server. While local dev on MacOS everything works as expected.

Dashboard Workload Bildschirmfoto 2021-06-15 um 10 21 59

Workload JSON Response Bildschirmfoto 2021-06-15 um 10 23 13

Pending / Completed List View Bildschirmfoto 2021-06-15 um 10 22 14 Bildschirmfoto 2021-06-15 um 10 53 06

Pending / Completed JSON Response Bildschirmfoto 2021-06-15 um 10 23 21 Bildschirmfoto 2021-06-15 um 10 53 28

Steps To Reproduce:

Inside a dispatched Job a new Job Batch is created:

$batch = Bus::batch([])->dispatch();

$jobs = $this->getJobs(); // Returns Array of Dispatchable, Batchable Jobs

$batch->add($jobs);

Horizon Config (Trim):

'trim' => [
        'recent' => 60,
        'pending' => 60,
        'completed' => 60,
        'recent_failed' => 10080,
        'failed' => 10080,
        'monitored' => 10080,
],
driesvints commented 3 years ago

I'm not sure what the problem here is. The jobs are handled almost immediately like you say so they shouldn't be showing up in pending?

mrmonat commented 3 years ago

The problem is not only happening in the pending list. The completed list is empty as well as the "completed" JSON response.

driesvints commented 3 years ago

There's really very few info here to debug all this. Can you first please try one of the support channels below? If you can actually identify this as a bug, feel free to report back and I'll gladly help you out and re-open this issue.

Thanks!

Jon78 commented 3 years ago

Could it be you're using setlocale? See https://github.com/laravel/horizon/issues/1003

mrmonat commented 3 years ago

@Jon78 no I don't set setlocale. I left everything related to localization on default. There seems to be a problem with the redis "zsets" "failed_jobs", "completed_jobs" and "pending_jobs" because they are always empty / don't get created...

Jon78 commented 3 years ago

@Jon78 no I don't set setlocale. I left everything related to localization on default. There seems to be a problem with the redis "zsets" "failed_jobs", "completed_jobs" and "pending_jobs" because they are always empty / don't get created...

Ah, because the problem you're describing sounded similar to mine and your screenshots are in German :smiley: Maybe your server config sets a non-English locale then? I would investigate what gets passed to Redis to see if floats are properly formatted as per https://github.com/laravel/horizon/issues/788#issuecomment-601255010

mrmonat commented 3 years ago

@Jon78 You were quite right...

For everyone having the same issue in the future:

The Server had a default Language of de_DE.UTF-8. This caused Problems in Laravels RedisJobRepository because the microtime passed to the Redis ZADD method was formatted like 1234556,12345 instead of 1234556.12345. Because of that, not a single job got added to the sets failed_jobs, completed_jobs, pending_jobs.

The solution is to set the Locale in AppServiceProvider:

setlocale(LC_ALL, 'en_US.UTF-8');
Jon78 commented 3 years ago

@driesvints this keeps popping up. Will you accept a PR to permanently fix it?

driesvints commented 3 years ago

@Jon78 think previous attempts got shot down. You're always free to give it another go but it seems to be a tough nut to crack.