search

owen-it / laravel-auditing

Record the change log from models in Laravel
https://laravel-auditing.com
MIT License
3.05k stars 390 forks source link

Excessive Queue job logged when use with batch import of large amount of data (#846) #884

Closed kenfai closed 5 months ago

kenfai commented 11 months ago
Q A
Bug? no(?)
New Feature? yes
Framework Laravel
Framework version 8.x
Package version 13.6.3
PHP version 7.4.x

Actual Behaviour

When using Supervisor to manage process queues and job logging, and Laravel Excel package to import large amount of data to the database, a high amount of logging been produced from the Queue Auditing:

[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit

I believe this issue happened due to a recently added Queue Auditing feature from PR:#846.

Expected Behaviour

I understand this may not be related to Laravel-Auditing library, but is there any way to disable logging for the above audit listener class by default?

This excessive logging is polluting the actual Queue job processes in the Supervisor's stdout_logfile file, making it hard to read.

Steps to Reproduce

  1. Setup Supervisor to manage Queue processes in Laravel: https://laravel.com/docs/8.x/queues#supervisor-configuration

  2. Create an Laravel-Excel Import class: https://docs.laravel-excel.com/3.1/imports/

  3. Upload an excel file with data to be imported to the database.

  4. Monitor or read the Supervisor stdout_logfile file to see the logs.

Possible Solutions

Allow audit Listener to be set to 'silent' mode from config/audit.php?

parallels999 commented 11 months ago

Did you try audit.queue.enable => false on config/audit.php??

https://github.com/owen-it/laravel-auditing/blob/66498a9cfa23dc1b34601dfb74ef691772eca800/config/audit.php#L169

kenfai commented 11 months ago

Did you try audit.queue.enable => false on config/audit.php??

https://github.com/owen-it/laravel-auditing/blob/66498a9cfa23dc1b34601dfb74ef691772eca800/config/audit.php#L169

Okay I have just applied the settings. Didn't realise when upgrading.

The feature is also causing Out of Memory issue as such:

[2023-12-22 11:33:07] production.ERROR: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 90181632 bytes) {"userId":500,"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Allowed memory size of 1073741824 bytes exhausted (tried to allocate 90181632 bytes) at /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Queue.php:158)

Will see if this fixes the issues.

Thanks!

angeljqv commented 11 months ago

Log Levels: Laravel supports various log levels such as emergency, alert, critical, error, warning, notice, info, and debug. Use appropriate log levels to differentiate between different types of log entries. Avoid excessive logging to prevent performance issues.