Katalogus caching in the scheduler

[jpbruinsslot]

The sophisticated caching and filtering in the scheduler can perhaps be optimized by:

• Adding more API filtering options in the Katalogus that would resolve creating 3 separate caches in the scheduler
• Adding ETag (Entity Tag) or Last-Modified headers to katalogus to invalidate internal katalogus state in the scheduler instead of relying on the ExpiringDict functionality
• Potentially resolve https://github.com/minvws/nl-kat-coordination/issues/856 since the katalogus state in the scheduler will always be updated to the latest modified timestamp
• Retrieval of new boefjes is now a call per organisation to diff internal cache of present boefjes to enabled boefjes. Could be optimized by a signal to the scheduler when a boefje gets enabled/added

Related: https://github.com/minvws/nl-kat-coordination/issues/856

[underdarknl]

In general, I want to move in the direction of storing the enabled/configured plugins in XTDB, instead of the katalogus. For the following point:

* Retrieval of new boefjes is now a call per organisation to diff internal cache of present boefjes to enabled boefjes. Could be optimized by a signal to the scheduler when a boefje gets enabled/added

When we move the plugin config to XTDB, octopoes should already send out signals to the Scheduler for changes to OOI's. We can use the same mechanism to pick up changes on plugins.