SimpleBackgroundJobs Support

[JoePJisc]

As of MISP version 2.4.151 we introduced a simpler way to handle background jobs without relying in CakeResque as this library is no longer mantained.

For the time being both background jobs backends will be supported, but we plan to phase out the CakeResque one in a near future.

The new backend requires Supervisor and some extra PHP packages.

https://www.circl.lu/doc/misp/appendices/#appendix-g-simplebackgroundjobs-migration-guide

It would be good if the RPM either installed with the new module enabled or provided the required packages to enable it.

[amuehlem]

Hi JoeP-oss

I've create misp-2.4.153-1 where the required packages are included and the supervisord is set as a requirement. Unfortunately the migration does not work, I get an error when trying to start the workers

[root@misp7 supervisord.d]# /var/www/MISP/app/Console/cake start_worker email
2022-02-07 14:56:38 Info: [WORKER PID: 6488][email] - starting to process background jobs...
Error: Call to a member function blpop() on null
#0 /var/www/MISP/app/Console/Command/StartWorkerShell.php(65): BackgroundJobsTool->dequeue()
#1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(462): StartWorkerShell->main()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(222): Shell->runCommand()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(66): ShellDispatcher->dispatch()
#4 /var/www/MISP/app/Console/cake.php(45): ShellDispatcher::run()
#5 {main}

I'm checking with the MISP developers if they have any idea what's not working properly to start the workers.

[JoePJisc]

Thanks for the update 👍

[amuehlem]

Same behavior on RHEL7 and RHEL8, opened an issue with MISP

https://github.com/MISP/MISP/issues/8195

[JoePJisc]

I'm in the process of trying out the RHEL8 support, and did the following post install to try and set up Supervisor per https://github.com/MISP/MISP/blob/2.4/docs/background-jobs-migration-guide.md

• Created the Supervisor Listener: sudo vi /etc/supervisord.d/misp-server.ini

[inet_http_server]
port=127.0.0.1:9001
username=supervisor
password=securePasswordHere

• Enable and start the MISP workers:
  • sudo systemctl enable supervisord
  • sudo systemctl start supervisord

Running sudo supervisorctl status gives me:

misp-workers:cache_00            FATAL     Exited too quickly (process log may have details)
misp-workers:cache_01            FATAL     Exited too quickly (process log may have details)
misp-workers:cache_02            FATAL     Exited too quickly (process log may have details)
misp-workers:cache_03            FATAL     Exited too quickly (process log may have details)
misp-workers:cache_04            FATAL     Exited too quickly (process log may have details)
misp-workers:default_00          FATAL     Exited too quickly (process log may have details)
misp-workers:default_01          FATAL     Exited too quickly (process log may have details)
misp-workers:default_02          FATAL     Exited too quickly (process log may have details)
misp-workers:default_03          FATAL     Exited too quickly (process log may have details)
misp-workers:default_04          FATAL     Exited too quickly (process log may have details)
misp-workers:email_00            FATAL     Exited too quickly (process log may have details)
misp-workers:email_01            FATAL     Exited too quickly (process log may have details)
misp-workers:email_02            FATAL     Exited too quickly (process log may have details)
misp-workers:email_03            FATAL     Exited too quickly (process log may have details)
misp-workers:email_04            FATAL     Exited too quickly (process log may have details)
misp-workers:prio_00             FATAL     Exited too quickly (process log may have details)
misp-workers:prio_01             BACKOFF   Exited too quickly (process log may have details)
misp-workers:prio_02             FATAL     Exited too quickly (process log may have details)
misp-workers:prio_03             FATAL     Exited too quickly (process log may have details)
misp-workers:prio_04             FATAL     Exited too quickly (process log may have details)
misp-workers:update_00           FATAL     Exited too quickly (process log may have details)

/var/log/supervisor/supervisord.log shows these three lines over and over for each worker:

INFO spawned: 'prio_01' with pid 11617
INFO success: prio_01 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
INFO exited: prio_01 (exit status 1; not expected)

and there are some lines like this one:

INFO gave up: default_00 entered FATAL state, too many start retries too quickly

[JoePJisc]

/var/www/MISP/app/tmp/logs/misp-workers-errors.log is also full of

Error: Call to a member function blpop() on null
#0 /var/www/MISP/app/Console/Command/StartWorkerShell.php(65): BackgroundJobsTool->dequeue()
#1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(462): StartWorkerShell->main()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(222): Shell->runCommand()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(66): ShellDispatcher->dispatch()
#4 /var/www/MISP/app/Console/cake.php(45): ShellDispatcher::run()

but not sure if that is related or not in all honesty.

[JoePJisc]

Ignoring the FATAL statuses and setting SimpleBackgroundJobs.enabled to True, has fixed the issue.

sudo supervisorctl status now shows:

misp-workers:cache_01            RUNNING   pid 27597, uptime 0:00:07
misp-workers:cache_02            RUNNING   pid 27598, uptime 0:00:07
misp-workers:cache_03            RUNNING   pid 27599, uptime 0:00:07
misp-workers:cache_04            RUNNING   pid 27614, uptime 0:00:07
misp-workers:default_00          RUNNING   pid 27586, uptime 0:00:07
misp-workers:default_01          RUNNING   pid 27587, uptime 0:00:07
misp-workers:default_02          RUNNING   pid 27588, uptime 0:00:07
misp-workers:default_03          RUNNING   pid 27589, uptime 0:00:07
misp-workers:default_04          RUNNING   pid 27590, uptime 0:00:07
misp-workers:email_00            RUNNING   pid 27591, uptime 0:00:07
misp-workers:email_01            RUNNING   pid 27592, uptime 0:00:07
misp-workers:email_02            RUNNING   pid 27593, uptime 0:00:07
misp-workers:email_03            RUNNING   pid 27594, uptime 0:00:07
misp-workers:email_04            RUNNING   pid 27595, uptime 0:00:07
misp-workers:prio_00             RUNNING   pid 27615, uptime 0:00:07
misp-workers:prio_01             RUNNING   pid 27616, uptime 0:00:07
misp-workers:prio_02             RUNNING   pid 27617, uptime 0:00:07
misp-workers:prio_03             RUNNING   pid 27619, uptime 0:00:07
misp-workers:prio_04             RUNNING   pid 27620, uptime 0:00:07
misp-workers:update_00           RUNNING   pid 27626, uptime 0:00:07

[amuehlem]

Is the inet_http_listener needed in the misp-workers.ini file? I did not find it in the documentation and all redis settings for the background jobs are set in the 'SimpleBackgroundJobs' section of MISPs config.php

[JoePJisc]

Yes, I beleive so, abstracted from step 3 of https://github.com/MISP/MISP/blob/2.4/docs/background-jobs-migration-guide.md.

The listener is configured in SimpleBackgroundJobs.supervisor_host, SimpleBackgroundJobs.supervisor_port, SimpleBackgroundJobs.supervisor_user, and SimpleBackgroundJobs.supervisor_password.

[amuehlem]

Ah, yes it's added to /etc/supervisor/supervisord.conf

[JoePJisc]

Yes, on RedHat the base file is /etc/supervisord.conf, which ends with an incldue for /etc/supervidsord.d/*.ini, so, for clarity I put it in its own INI file - same effect just bit easier to maintain IMHO.

Don't know if you wanted to make the file as part of the RPM and tell people to go change the password, or just tell people to make the file in RHEL8.md; I think either would work.

[amuehlem]

I've added it to the README on the start page, it's the same for RHEL7 and RHEL8 and I think there most people will see it