Document new jobs queue in main documentation, admin guide, and 3.4 release notebook

The new jobs queue in 3.4 will need to be documented. There are three kinds of documentation we want:

In the main developer documentation, we should document how to create queued jobs (and scheduled jobs, if we support that for 3.4).
In the admin guide, we should document how a sysadmin can configure their server to process queued jobs. This should include running service workers, using cron, or relying on the register_shutdown method.
In the 3.4 Release Notebook (not created yet), we should include a notice about the new queue, and point them to the admin guide for any configuration changes they should make.

These guides should not try to duplicate the Laravel documentation as much as possible. Where best, they can link on to the Laravel documentation.

@henriqueramos has begun work on a PR:

https://github.com/pkp/pkp-docs/pull/755/files

A comment from @touhidurabir:

I faced this similar type of issue few time working on Laravel’s Queue Jobs . I usually follow 2 approaches which work for me most of the cases using a failed method in the job class and log through that

   /**
     * Handle a job failure.
     *
     * @param  \Throwable  $exception
     * @return void
     */
    public function failed(Throwable $exception)
    {
        // Send user notification of failure or log etc ...
    }

using the global queue failed event to log it through some service provider

    Queue::failing(function (JobFailed $event) {
            // $event->connectionName
            // $event->job
            // $event->exception
        });

if it’s using the database , the failed table will have that in a column like exception . Most of the time I used Laravel Horizon as my queue monitoring dashboard but that a laravel first package and required redis driver so may be it will require a ton of modification if we want to use in our app where laravel itself is a dependency of the system .

More details about running jobs in a worker here:

https://github.com/pkp/pkp-lib/issues/8044#issuecomment-1213059149

After this get merged with main, to run a worker daemon, run command
php lib/pkp/tools/jobs.php work
use the flag --help with the work command to view all available options that can be passed as it's support most of laravel queue:work command options .

To set up the worker daemon monitor via Supervisor

install it in linux(for example ubuntu) system as
sudo apt-get install supervisor
set up a conf file(for example ojs-queue-monitor.conf) with following content
[program:ojs-queue-monitor]
process_name=%(program_name)s
command=/PHP_PATH/php /OJS_INSTALLATION_PATH/lib/pkp/tools/jobs.php work
directory=/OJS_INSTALLATION_PATH
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/OJS_INSTALLATION_PATH/supervisor-database.log
usually path of conf files are at /etc/supervisor/conf.d. Make sure to update few setting like command, directory, program stdout_logfile etc as per need .

Restart the supervisor sudo service supervisor restart and go into the supervisor control shell via sudo supervisorctl. A reload command may need to reload the .conf file .

Installation in MAC is bit different as it use .ini file extension but the file configuration remains same . To install in MAC
brew install supervisor
install path usually /usr/local/etc/ but need to create directory supervisor.d and create .ini file (for example ojs-queue-monitor.ini) in there .

Rest of the process is same as Linux system .

More information about options for running the jobs queue:

We just got the PR for #7105 merged . This allows us to run queue jobs via the cron. But this PR add some brand new queue jobs execution functionality with a some breaking changes .

Breaking Changes :

previously we use a failed method in the PKP\Support\Jobs\BaseJob and in any job class we used to register the failure through that . It's been removed and moved to a global queue event listener (new implementation at https://github.com/pkp/pkp-lib/blob/main/classes/core/PKPQueueProvider.php#L115-L125) as that was not the right way to register failure. I have fixed all the existing jobs but if new jobs are going to introduce, just throw an exception and if I missed any, we need to update those jobs .
Previously jobs used to run at each request life cycle end via register_shutdown_function. We still have this feature but the configs has changed . previous config was

; Do not run jobs on shutdown
; By default, jobs in the queue will be run during PHP's shutdown
; function. Disable this if you want to run jobs through a separate
; cron job or workers.
disable_jobs_run_at_shutdown = Off

we have now introduced 4 new configs, changes at https://github.com/pkp/ojs/blob/main/config.TEMPLATE.inc.php#L531-L570 . This changes applies for OJS, OMP and OPS. If still wish to run jobs as previous way, that is one job at each request life cycle end , need to update the config.inc.php and set job_runner to On.

Running Jobs

We now have multiple ways to run jobs such as

Run one job at each request. This can be achieved by setting the configuration job_runner to On. This will also run jobs via Acron which will run once in every minutes . As this run via register_shutdown_function at the end of request life cycle, we have some config constrains like How many jobs, Max job run time and Max memory consumption. see the config details at https://github.com/pkp/ojs/blob/main/config.TEMPLATE.inc.php#L531-L570 . Note that those configs will only apply when jobs run via Acron, as when jobs process at each request is set to 1(we don't want to put a big load on each request) . But this approach NOT RECOMMENDED at all and should be used by installation that have no other ways to process jobs .
Run jobs via cron. This can be achieved via command php tools/runScheduledTasks.php. Set up the cron as per need.
Run jobs via command line tool via php lib/pkp/tools/jobs.php run. This will run all jobs in the default queue till it become empty.
Run jobs via a worker daemon. This is the best and MOST RECOMMENDED way to run jobs . run the command php lib/pkp/tools/jobs.php work and leave that shell/bash window as it is . Every time a job get pushed to queue , it will start processing jobs immediately. the work command take a lot of options(almost all that laravel support and bit more), to see the list of available options, run command php lib/pkp/tools/jobs.php work --help.
Another way is to use a daemon monitor tool such as Supervisor to run the worker deamon via the work command . see that details at https://github.com/pkp/pkp-lib/issues/8044#issuecomment-1213059149 . This may be bit over kill for the local development but probably the best approach to run in production .

Finally the job command line tool have some good enhancement and feature added, to see the available options and commands , run php lib/pkp/tools/jobs.php usage .

pkp / pkp-docs