Offline runners are not removed but are selected for task execution

[enpaul]

Last week I started the process of migrating my Semaphore instance to using remote runners. In the process of testing, I spun up a few temporary runners using Docker and didn't bother setting up a persistent storage volume for them. After doing this a few times I could no longer launch a task, as every task would hang indefinitely on "starting..." while waiting to be assigned to a runner. A quick investigation of the database showed that all the runners I launched and exited were still registered with Semaphore and presumably still being assigned tasks. I was able to resolve this by connecting to the database and manually deleting the runner registrations for the now-deactivated runners.

I know runners are still fairly new to the ecosystem, but I wanted to raise this as both an operational and administration issue.

Some potential solutions I thought of quick:

• Have Semaphore check whether a runner is online before assigning a task to it
• Have Semaphore periodically check whether all runners are online and remove any that aren't
  • Runners should also periodically re-register to account for runners that may occasionally lose connectivity to the server
• Add an administration panel in the UI that shows the list of registered runners and allow admins to delete them

This is an amazing project and I'm really excited to work with it more!

[fiftin]

Hi @enpaul

When designing, I had a different concept in mind than that of GitLab.

Offline runner in ok. Because it can be started by demand. We have webhooks for this.

So. When user starts a task, Semaphore find suitable runner and call webhook if it is provided. Currently result of webhook not handled, but I think we should handle it.

Yes, we can add logic: if webhook not provided we shouldn't use offline runner. I think it is right way.

[bernd-mueller-el]

Hello, I get the same at my test environment. Is there a plan when this will be implemented? I can see here Milestone with 2.9, is this the version of semaphoreUI? I ask because my semaphore container for server and runner shows me 2.10. Kind Regards