UnicornTranscoder / UnicornLoadBalancer

MIT License
128 stars 16 forks source link

Unicorn LB not aware of node status without service restart #44

Open g-vdn opened 3 years ago

g-vdn commented 3 years ago

Hi,

First I want to say thanks for this great piece(s) of software!

Here is my set-up: PMS, Unicorn LB and Transcoder running on rpi4, and I have set-up a VM on my desktop for when I need help on transcodes, with Unicorn Transcoder and PMS running on it. On both machines there is systemd service running for unicorn transcoder and only on rpi4 systemd service for load balancer. I managed to install all pieces of software correctly (or at least I think I did), as all works fine. When the max number of transcodes is reached on VM transcoder, the next one goes on pi. The problem is when the VM goes down for any reason, LB doesn't know it, unless a restart is issued, and will keep redirecting to the offline node. After restart, it will see that it has only 1 node. Same vice-versa, if LB knows about 1 node, when 2nd one is online, without a restart it wont see it.

Here is the part of config files I think it's relevant to the issue:

rpi4 Unicorn LB config:

timeout: env.int('CUSTOM_SCORES_TIMEOUT', 10) list: env.array('CUSTOM_SERVERS_LIST', 'string', ['https://transcoder-2.domain.com/','https://transcoder-1.domain.com/'])

rpi4 Unicorn Transcoder config:

transcoder_decay_time: env.int ('TRANSCODER_DECAY_TIME', 120), loadbalancer_address: env.string('LOADBALANCER_ADDRESS', 'https://plex.domain.com'), ping_frequency: env.int ('PING_FREQUENCY', 10), instance_address: env.string('INSTANCE_ADDRESS', 'https://transcoder-1.domain.com'), maxSessions: env.int('MAX_SESSIONS', 0),

VM Unicorn Transcoder config:

transcoder_decay_time: env.int ('TRANSCODER_DECAY_TIME', 120), loadbalancer_address: env.string('LOADBALANCER_ADDRESS', 'https://plex.domain.com'), ping_frequency: env.int ('PING_FREQUENCY', 10), instance_address: env.string('INSTANCE_ADDRESS', 'https://transcoder-2.domain.com'), maxSessions: env.int('MAX_SESSIONS', 2),

g-vdn commented 3 years ago

As a workaround, I create 2 systemd services on the VM which will run at boot and at shutdown, restarting the LB on rpi4. It works, but would be great to have the LB figure it out on its own. Not sure what CUSTOM_SCORES_TIMEOUT from the LB config is doing, but the explanation I understand is that it should consider a node transcoder not available after x amount of seconds. Also the config on Transcoder, has a line ping frequency. Any reply would be greatly appreciated

Maxou44 commented 3 years ago

Interesting... Normaly when the transcoder don't ping the LB during 30 sec, the LB flag it as "unavailable" and don't redirect streams on it. When your transcoder is down, maybe the transcoder continue to ping the load balancer ?

g-vdn commented 3 years ago

Here I launched the shutdown command on the VM (transcoder-2)

Feb 12 10:51:16 XXXX npm[5549]: 2021-02-12T09:51:16.938Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com]

it went like this for some time...

Feb 12 11:01:49 XXXX npm[22497]: 2021-02-12T10:01:49.560Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:02:00 XXXX npm[22497]: 2021-02-12T10:02:00.156Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:02:07 XXXX npm[22497]: 2021-02-12T10:02:07.521Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:02:10 XXXX npm[22497]: 2021-02-12T10:02:10.841Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:02:17 XXXX npm[22497]: 2021-02-12T10:02:17.591Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:02:21 XXXX npm[22497]: 2021-02-12T10:02:21.685Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:02:27 XXXX npm[22497]: 2021-02-12T10:02:27.649Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:02:32 XXXX npm[22497]: 2021-02-12T10:02:32.381Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:02:37 XXXX npm[22497]: 2021-02-12T10:02:37.702Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:02:45 XXXX npm[22497]: 2021-02-12T10:02:45.380Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:02:47 XXXX npm[22497]: 2021-02-12T10:02:47.729Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:02:57 XXXX npm[22497]: 2021-02-12T10:02:57.692Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:03:00 XXXX npm[22497]: 2021-02-12T10:03:00.412Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:03:07 XXXX npm[22497]: 2021-02-12T10:03:07.741Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:03:09 XXXX npm[22497]: 2021-02-12T10:03:09.142Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:03:17 XXXX npm[22497]: 2021-02-12T10:03:17.845Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:03:19 XXXX npm[22497]: 2021-02-12T10:03:19.961Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:03:27 XXXX npm[22497]: 2021-02-12T10:03:27.820Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:03:32 XXXX npm[22497]: 2021-02-12T10:03:32.932Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:03:37 XXXX npm[22497]: 2021-02-12T10:03:37.917Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24g70d [PING] Feb 12 11:03:43 XXXX npm[22497]: 2021-02-12T10:03:43.488Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:03:48 XXXX npm[22497]: 2021-02-12T10:03:48.418Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24icv2 [PING] Feb 12 11:03:58 XXXX npm[22497]: 2021-02-12T10:03:58.426Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24icv2 [PING] Feb 12 11:03:59 XXXX npm[22497]: 2021-02-12T10:03:59.252Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:04:04 XXXX npm[22497]: 2021-02-12T10:04:04.996Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:04:08 XXXX npm[22497]: 2021-02-12T10:04:08.521Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24icv2 [PING] Feb 12 11:04:15 XXXX npm[22497]: 2021-02-12T10:04:15.809Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:04:18 XXXX npm[22497]: 2021-02-12T10:04:18.499Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24icv2 [PING] Feb 12 11:04:26 XXXX npm[22497]: 2021-02-12T10:04:26.422Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:04:28 XXXX npm[22497]: 2021-02-12T10:04:28.608Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24icv2 [PING] Feb 12 11:04:37 XXXX npm[22497]: 2021-02-12T10:04:37.013Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:04:38 XXXX npm[22497]: 2021-02-12T10:04:38.709Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24icv2 [PING] Feb 12 11:04:47 XXXX npm[22497]: 2021-02-12T10:04:47.741Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:04:48 XXXX npm[22497]: 2021-02-12T10:04:48.722Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24icv2 [PING] Feb 12 11:04:58 XXXX npm[22497]: 2021-02-12T10:04:58.776Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24icv2 [PING] Feb 12 11:05:00 XXXX npm[22497]: 2021-02-12T10:05:00.666Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:05:09 XXXX npm[22497]: 2021-02-12T10:05:09.363Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24k3b0 [PING] Feb 12 11:05:13 XXXX npm[22497]: 2021-02-12T10:05:13.662Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:05:19 XXXX npm[22497]: 2021-02-12T10:05:19.390Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24k3b0 [PING] Feb 12 11:05:24 XXXX npm[22497]: 2021-02-12T10:05:24.477Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:05:29 XXXX npm[22497]: 2021-02-12T10:05:29.371Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24k3b0 [PING] Feb 12 11:05:35 XXXX npm[22497]: 2021-02-12T10:05:35.201Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:05:39 XXXX npm[22497]: 2021-02-12T10:05:39.476Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24k3b0 [PING] Feb 12 11:05:45 XXXX npm[22497]: 2021-02-12T10:05:45.966Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:05:49 XXXX npm[22497]: 2021-02-12T10:05:49.504Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24k3b0 [PING] Feb 12 11:05:56 XXXX npm[22497]: 2021-02-12T10:05:56.800Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:05:59 XXXX npm[22497]: 2021-02-12T10:05:59.525Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24k3b0 [PING] Feb 12 11:06:09 XXXX npm[22497]: 2021-02-12T10:06:09.610Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24k3b0 [PING] Feb 12 11:06:09 XXXX npm[22497]: 2021-02-12T10:06:09.857Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com] Feb 12 11:06:19 XXXX npm[22497]: 2021-02-12T10:06:19.599Z UnicornLoadBalancer FFMPEG 3rl8shdhkl24k3b0 [PING] Feb 12 11:06:20 XXXX npm[22497]: 2021-02-12T10:06:20.541Z UnicornLoadBalancer PING 9bdc5c59b76960b8-com-plexapp-android [https://transcoder-2.domain.com]