Add support for an external signature server

[SamantazFox]

Closes #4649

This PR adds support for inv_sig_helper, which offloads the player fetching, function extraction and signature parsing, which in turn allows to use the web client to watch videos.

Side note: I had to update the crystal overrides because the stdlib changed quite a while ago (See https://github.com/crystal-lang/crystal/pull/11049) and it was required to properly use TCP/unix sockets.

---

TODO:

• [x] Only activate if something like inv_sig_helper_url: XX is filled. This way, this functionality is only enabled when the user needs it. And the user can specify a TCP endpoint in case of Docker, for example.
• [ ] Additionally, if this is not activated, ANDROID_TEST_SUITE will still be used. (As long as it still works fine for "unblocked IP addresses") This will be addressed when this and #4789 will be merged

[SamantazFox]

Also, what do you think about splitting the signature communication logic into its own shard?

Why though? Sound harder to maintain and I don't see anyone using this code except us. The connection multiplexing code might actually be a good candidate for a shard, but I don't think the code is versatile enough in its current state.

[syeopite]

Ah right. I did not think about the additional maintenance burden.

I requested it to potentially open the opportunity for other YouTube projects within the Crystal ecosystem though the possibility of that is pretty low...

Let's just keep it within Invidious.

[SamantazFox]

Important note: with the current state of the PR, invidious can't start if the socket is not present. I will add some logic to fallback to another client in that case.

[unixfox]

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight?

I'm thinking about docker, where each service has its separate private IP address.

[techmetx11]

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight?

I'm thinking about docker, where each service has its separate private IP address.

Currently, it's listening on a UNIX socket (aka. on a path, by default: /tmp/inv_sig_helper.sock) I could change it to let it listen on UNIX or TCP

[unixfox]

@techmetx11

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight? I'm thinking about docker, where each service has its separate private IP address.

Currently, it's listening on a UNIX socket (aka. on a path, by default: /tmp/inv_sig_helper.sock) I could change it to let it listen on UNIX or TCP

Ah, yes that would be a great improvement because we have people in the community that run Invidious over multiple servers, and you can only communicate through TCP if you have multiple servers, a UNIX socket won't work.

I'm not sure if the sig parameter can be shared between multiple public IP addresses or not, though.

[techmetx11]

@techmetx11

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight? I'm thinking about docker, where each service has its separate private IP address.

Currently, it's listening on a UNIX socket (aka. on a path, by default: /tmp/inv_sig_helper.sock) I could change it to let it listen on UNIX or TCP

Ah, yes that would be a great improvement because we have people in the community that run Invidious over multiple servers, and you can only communicate through TCP if you have multiple servers, a UNIX socket won't work.

That would make the current design, of Invidious sending a FORCE_UPDATE opcode to the server, a bit network-intensive. I would have to make the signature server have its own scheduler

I'm not sure if the sig parameter can be shared between multiple public IP addresses or not, though.

The JS code doesn't require network access, so I would hope so

[unixfox]

Continuing the discussion about signature helper server here: https://github.com/iv-org/inv_sig_helper/issues/1

[unixfox]

There will be a need to add the support for TCP/IP, as such that it can be used from multiple invidious processes.

Issue opened on inv_sig_helper side: https://github.com/iv-org/inv_sig_helper/issues/2

[999eagle]

i've been running this and #4789 on iv.catgirl.cloud for a while now to circumvent the login requirements, but occasionally invidious workers fail to start correctly after their hourly restart due to waiting indefinitely for the sig helper to force update the player. i've opened https://github.com/iv-org/inv_sig_helper/issues/4 for the underlying issue here

[unixfox]

Missing feature:

• Only activate if something like inv_sig_helper_url: XX is filled. This way, this functionality is only enabled when the user needs it. And the user can specify a TCP endpoint in case of Docker, for example.
• Additionally, if this is not activated, ANDROID_TEST_SUITE will still be used. (As long as it still works fine for "unblocked IP addresses")

[MMaster]

To also write it here: Video caching in invidious also caches and reuses data from player so if the player changes the video that is pulled from cache will cause 403s to be returned for videoplayback requests.

[SamantazFox]

To also write it here: Video caching in invidious also caches and reuses data from player so if the player changes the video that is pulled from cache will cause 403s to be returned for videoplayback requests.

Yeah, I should rewrite the URLs during parsing (i.e before caching) not during use.