[Idea] Remote Workers & Queue Management

[cshyjak]

Have you read the Code of Conduct? By filing an Issue, you are expected to comply with it, including treating everyone with respect: Yes

Remember that these are some guidelines, use your best judgement.

Description

I have a couple of ideas around how remote workers work that would be great to get some input on. They fall into 2 main interconnected areas:

1. Re-envisioning of the linked installation concept to more of a coordinator/ui combo with pools of workers either local or remote
2. Utilize a distributed queueing mechanism (like zeromq) to support multiple different worker pool topologies

The linked installation concept works atm but I’ve ran into some limitations:

• Hard to get a single pane of glass view into what all is processing
• Some features don’t work well in a linked install setup (file size data panel results)
• Library configuration synchronization can be tedious/buggy

What I’m thinking about is allowing for a worker only configuration of the app. It would subscribe to the main install and show up as a remote worker in the worker pools (maybe just a separate pool). By utilizing a messaging framework like zeromq we’d be able o push tasks to a remote worker including all of the information required to perform it and the workers could message back status updates/results. This would centralize the library management configuration into the main install. Local workers and remote workers would subscribe in the same fashion to keep things consistent.

[cshyjak]

Issues this could help with:

• https://github.com/Unmanic/unmanic/issues/346
• https://github.com/Unmanic/unmanic/issues/336
• https://github.com/Unmanic/unmanic/issues/329
• https://github.com/Unmanic/unmanic/issues/261

[Josh5]

Sounds good :+1:

My next scoped piece of work was to add the ability to duplicate plugins within a library config. Something like making each plugin entry in the flow a pointer to the plugin and give that pointer a unique ID. This way we can add multiple jobs in our plugin flow that use the same plugin. This may take me a little while to figure out and implement (especially with home life the way it is at the moment - I'm not getting a lot of open-source development time). If you wanted to take on Linking improvements, I'm all for that man. DM me on Discord if you want to open up a more direct line of coms to discuss things. Your contributions this week have been on point and much appreciated.