Add ability to set po_token and visitor data

unixfox commented 2 months ago

Helps with https://github.com/iv-org/invidious/issues/4734

This adds the ability to set po_token and visitordata ID statically in the config.yaml

Also, if po_token parameter is passed, then WEB client is used, if not Android client is used. TvHtml5ScreenEmbed will still be used as a fallback.

Script for generating po_token and visitor_data: https://github.com/iv-org/youtube-trusted-session-generator

There are plans to automate the tokens creation: https://github.com/iv-org/youtube-trusted-session-generator/issues/1 but not yet implemented. At least this unlocks people for keep using invidious.

MrSparkle23 commented 2 months ago

Tried unixfox:potoken-config branch and got this error. Generated both values and added to config. Am I missing something?

`Title: Missing hash key: "videoDetails" (KeyError) Date: 2024-07-18T21:30:55Z Route: /watch?v=jNQXAC9IVRw Version: 2024.07.11-9e11d59a @ potoken-config

Backtrace

Missing hash key: "videoDetails" (KeyError) from /usr/share/crystal/src/hash.cr:1077:11 in '[]' from /usr/share/crystal/src/hash.cr:1128:17 in 'dig' from src/invidious/videos/parser.cr:147:12 in 'try_fetch_streaming_data' from src/invidious/videos/parser.cr:111:5 in 'extract_video_info' from src/invidious/videos.cr:395:3 in 'fetch_video' from src/invidious/videos.cr:383:13 in 'get_video' from src/invidious/videos.cr:365:1 in 'get_video:region' from src/invidious/routes/watch.cr:55:15 in 'handle' from src/invidious/routing.cr:159:19 in '->' from lib/kemal/src/kemal/route.cr:12:26 in '->' from src/invidious/helpers/handlers.cr:30:37 in 'process_request' from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/handlers.cr:157:5 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/handlers.cr:94:12 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/handlers.cr:146:5 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/handlers.cr:70:5 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/ext/kemal_static_file_handler.cr:162:16 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/logger.cr:17:35 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call' from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process' from /usr/share/crystal/src/http/server.cr:517:5 in 'handle_client' from /usr/share/crystal/src/http/server.cr:470:13 in '->' from /usr/share/crystal/src/fiber.cr:146:11 in 'run' from /usr/share/crystal/src/fiber.cr:98:34 in '->' from ???

`

syeopite commented 2 months ago

You'll also need to setup #4772

MMaster commented 2 months ago

My instance "got hit by" sign in requirement yesterday. Since this mentions it should help with it I thought I will test it.

I've used unixfox:potoken-config, merged #4772 into it and built inv_sig_helper.

inv_sig_helper starts successfully, invidious connects to it and successfully starts. Trace logs show that it is using my visitor_data and po_token in video request (generated using docker). I'm still getting the "This helps protect our community" error. I've tried generating 3 different visitor_data and po_token combos even from different IPs.

2024-07-19 00:42:28 UTC [trace] SigHelper: Send transaction 0x2507ee1d / opcode GET_SIGNATURE_TIMESTAMP
Decoder length: 5
Received job: GetSignatureTimestamp
Decoder length: 0
2024-07-19 00:42:28 UTC [trace] SigHelper: Send transaction 0x2507ee1d - Done
2024-07-19 00:42:28 UTC [trace] SigHelper: Recv transaction 0x2507ee1d / length 8
2024-07-19 00:42:28 UTC [trace] SigHelper: payload = Bytes[0, 0, 0, 0, 0, 0, 77, 209]
2024-07-19 00:42:28 UTC [trace] SigHelper: Recv transaction 0x2507ee1d - Done
2024-07-19 00:42:28 UTC [trace] SigHelper: Transaction unqueued and data sent to channel
2024-07-19 00:42:28 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/player"
2024-07-19 00:42:28 UTC [trace] YoutubeAPI: ClientConfig: YoutubeAPI::ClientConfig(@client_type=YoutubeAPI::ClientType::Web, @region="US")
2024-07-19 00:42:28 UTC [trace] YoutubeAPI: POST data: {"contentCheckOk" => true, "videoId" => "jNQXAC9IVRw", "context" => {"client" => {"hl" => "en", "gl" => "US", "clientName" => "WEB", "clientVersion" => "2.20240304.00.00", "clientScreen" => "WATCH_FULL_SCREEN", "osName" => "Windows", "osVersion" => "10.0", "platform" => "DESKTOP", "visitorData" => "XXX"}}, "racyCheckOk" => true, "user" => {"lockedSafetyMode" => false}, "playbackContext" => {"contentPlaybackContext" => {"html5Preference" => "HTML5_PREF_WANTS", "referer" => "https://www.youtube.com/watch?v=jNQXAC9IVRw", "signatureTimestamp" => 19921}}, "serviceIntegrityDimensions" => {"poToken" => "XXX"}}
2024-07-19 00:42:29 UTC [error] get_video: jNQXAC9IVRw : This helps protect our community. Learn more
2024-07-19 00:42:29 UTC [warn] i18n: Missing translation key "This helps protect our community. Learn more"
2024-07-19 00:42:29 UTC [info] 500 GET /watch?v=jNQXAC9IVRw 90.44ms

(I've replaced visitorData & poToken with XXX, but they matched the generated ones.)

Maybe I'm missing something, but I just thought I will let you know.

unixfox commented 2 months ago

@MMaster thanks for the feedback. I personally think that something changed since this PR was created with how youtube treat the poToken validity on a youtube embed page.

For the moment on yewtu.be, I'm using a poToken generated from www.google.com search and it works fine. I'll try to adapt youtube-trusted-session-generator for this other kind of potoken generation. Will get back to you once it's done!

MMaster commented 2 months ago

@MMaster thanks for the feedback. I personally think that something changed since this PR was created with how youtube treat the poToken validity on a youtube embed page.

For the moment on yewtu.be, I'm using a poToken generated from www.google.com search and it works fine. I'll try to adapt youtube-trusted-session-generator for this other kind of potoken generation. Will get back to you once it's done!

Strangely I just did exactly what the trusted session generator does manually in private window and copied visitorData and poToken from the play button POST request and it looks like this time it got further, but based on the logs it still looks like it requires login which in turn errors out on Missing hash key: "videoDetails" (KeyError)

Decoder length: 5
Received job: GetSignatureTimestamp
Decoder length: 0
2024-07-19 18:33:48 UTC [trace] SigHelper: Recv transaction 0x65aa7ba1 / length 8
2024-07-19 18:33:48 UTC [trace] SigHelper: payload = Bytes[0, 0, 0, 0, 0, 0, 77, 209]
2024-07-19 18:33:48 UTC [trace] SigHelper: Recv transaction 0x65aa7ba1 - Done
2024-07-19 18:33:48 UTC [trace] SigHelper: Transaction unqueued and data sent to channel
2024-07-19 18:33:48 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/player"
2024-07-19 18:33:48 UTC [trace] YoutubeAPI: ClientConfig: YoutubeAPI::ClientConfig(@client_type=YoutubeAPI::ClientType::Web, @region="US")
2024-07-19 18:33:48 UTC [trace] YoutubeAPI: POST data: {"contentCheckOk" => true, "videoId" => "jNQXAC9IVRw", "context" => {"client" => {"hl" => "en", "gl" => "US", "clientName" => "WEB", "clientVersion" => "2.20240304.00.00", "clientScreen" => "WATCH_FULL_SCREEN", "osName" => "Windows", "osVersion" => "10.0", "platform" => "DESKTOP", "visitorData" => "XXX"}}, "racyCheckOk" => true, "user" => {"lockedSafetyMode" => false}, "playbackContext" => {"contentPlaybackContext" => {"html5Preference" => "HTML5_PREF_WANTS", "referer" => "https://www.youtube.com/watch?v=jNQXAC9IVRw", "signatureTimestamp" => 19921}}, "serviceIntegrityDimensions" => {"poToken" => "XXX"}}
2024-07-19 18:33:48 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/next"
2024-07-19 18:33:48 UTC [trace] YoutubeAPI: ClientConfig: YoutubeAPI::ClientConfig(@client_type=YoutubeAPI::ClientType::Web, @region="US")
2024-07-19 18:33:48 UTC [trace] YoutubeAPI: POST data: {:videoId => "jNQXAC9IVRw", :params => "", "context" => {"client" => {"hl" => "en", "gl" => "US", "clientName" => "WEB", "clientVersion" => "2.20240304.00.00", "clientScreen" => "WATCH_FULL_SCREEN", "osName" => "Windows", "osVersion" => "10.0", "platform" => "DESKTOP", "visitorData" => "XXX"}}}
2024-07-19 18:33:49 UTC [debug] extract_video_info: parsing related videos...
2024-07-19 18:33:49 UTC [trace] parse_related_video: Found "watchNextEndScreenRenderer" container
...
2024-07-19 18:33:49 UTC [trace] parse_related_video: Found "watchNextEndScreenRenderer" container
2024-07-19 18:33:49 UTC [trace] extract_video_info: Found "likes" button. Button text is "like this video along with 16,594,063 other people"
2024-07-19 18:33:49 UTC [debug] extract_video_info: Likes count is 16594063
2024-07-19 18:33:49 UTC [debug] try_fetch_streaming_data: [jNQXAC9IVRw] Using AndroidTestSuite client.
2024-07-19 18:33:49 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/player"
2024-07-19 18:33:49 UTC [trace] YoutubeAPI: ClientConfig: YoutubeAPI::ClientConfig(@client_type=YoutubeAPI::ClientType::AndroidTestSuite, @region="US")
2024-07-19 18:33:49 UTC [trace] YoutubeAPI: POST data: {"contentCheckOk" => true, "videoId" => "jNQXAC9IVRw", "context" => {"client" => {"hl" => "en", "gl" => "US", "clientName" => "ANDROID_TESTSUITE", "clientVersion" => "1.9", "androidSdkVersion" => 31, "osName" => "Android", "osVersion" => "12", "platform" => "MOBILE", "visitorData" => "XXX"}}, "racyCheckOk" => true, "user" => {"lockedSafetyMode" => false}, "playbackContext" => {"contentPlaybackContext" => {"html5Preference" => "HTML5_PREF_WANTS", "referer" => "https://www.youtube.com/watch?v=jNQXAC9IVRw"}}, "serviceIntegrityDimensions" => {"poToken" => "XXX"}, "params" => "2AMB"}
2024-07-19 18:33:49 UTC [debug] try_fetch_streaming_data: [jNQXAC9IVRw] Got playabilityStatus == LOGIN_REQUIRED.
2024-07-19 18:33:49 UTC [error] get_video: jNQXAC9IVRw : Missing hash key: "videoDetails"
2024-07-19 18:33:49 UTC [info] 500 GET /watch?v=jNQXAC9IVRw 1825.3ms

I haven't changed anything except for po_token and visitor_data in config - its still the same docker image I've built before. Anyway - I don't want to pollute this pull request. Just wanted to let you know that maybe its not just about poToken.

EDIT: Just noticed that the android request is missing "signatureTimestamp", I'm not sure if that's expected.

MMaster commented 2 months ago

Last message, I promise :) I just got it to work by completely removing the AndroidTestSuite fallback and instead using the TvHtml5ScreenEmbed if reason is nil.

EDIT: I figured out that youtube-trusted-session-generator returns poToken and visitorData even if youtube expects you to login there. Those are shorter than the proper ones. (visitorData 48 vs 80 chars, poToken 164 vs 204 chars)

PS: Thank you so much for maintaining this project. Much appreciated! I hope my findings will help at least a little bit :)

unixfox commented 2 months ago

hello @MMaster, please try again the script https://github.com/iv-org/youtube-trusted-session-generator with the latest commit

Without your latest patch in order to confirm that it works great with the PR #4772 thank you.

I just got it to work by completely removing the AndroidTestSuite fallback and instead using the TvHtml5ScreenEmbed if reason is nil.

That's not a good idea if it only works with TvHtml5ScreenEmbed because it means that ONLY videos that can be embedded will work. My new commit in the script should fix this issue.

MMaster commented 2 months ago

@unixfox Thank you for getting back to me! I've tested it, but unfortunately I'm still getting the "This helps protect our community." with visitor_data and po_token generated using that script in docker.

2024-07-20 01:21:15 UTC [trace] SigHelper: Recv transaction 0x3d561ef1 - Done
2024-07-20 01:21:15 UTC [trace] SigHelper: Transaction unqueued and data sent to channel
2024-07-20 01:21:15 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/player"
2024-07-20 01:21:15 UTC [trace] YoutubeAPI: ClientConfig: YoutubeAPI::ClientConfig(@client_type=YoutubeAPI::ClientType::Web, @region="US")
2024-07-20 01:21:15 UTC [trace] YoutubeAPI: POST data: {"contentCheckOk" => true, "videoId" => "jNQXAC9IVRw", "context" => {"client" => {"hl" => "en", "gl" => "US", "clientName" => "WEB", "clientVersion" => "2.20240304.00.00", "clientScreen" => "WATCH_FULL_SCREEN", "osName" => "Windows", "osVersion" => "10.0", "platform" => "DESKTOP", "visitorData" => "stringwithnospaces%3D%3D"}}, "racyCheckOk" => true, "user" => {"lockedSafetyMode" => false}, "playbackContext" => {"contentPlaybackContext" => {"html5Preference" => "HTML5_PREF_WANTS", "referer" => "https://www.youtube.com/watch?v=jNQXAC9IVRw", "signatureTimestamp" => 19921}}, "serviceIntegrityDimensions" => {"poToken" => "stringwithnospaces=="}}
2024-07-20 01:21:15 UTC [error] get_video: jNQXAC9IVRw : This helps protect our community. Learn more
2024-07-20 01:21:15 UTC [warn] i18n: Missing translation key "This helps protect our community. Learn more"
2024-07-20 01:21:15 UTC [info] 500 GET /watch?v=jNQXAC9IVRw 77.88ms

It doesn't matter if I run it from the instance or other machine where the generated tokens are longer.

Unfortunately I currently don't have machine where I would be willing to install all the dependencies of chromium without docker, but I can get one if you think it would be worth it.

Now even if I open the original embed video url using private browser from instance IP it doesn't show the sign in dialog and I can play it directly (previously I couldn't play the embed video without signing in). The tokens I get manually from play button submit POST request work as long as I skip Android. I understand this is not good - just saying.

unixfox commented 2 months ago

@MMaster Please try yet again, this time it should work! The reason why it didn't work the previous time is that google is doing some checks whenever you are in "headfull" mode or not in Chromium. I was testing in headfull, whereas with Docker it's in headless.

But in Docker for headfull, this requires a graphical environment (X11) so now the image is a bit bigger but at least it works.

MMaster commented 2 months ago

@unixfox Bingo. This time the tokens generated from the script work just as the ones I got manually. The AndroidTestSuite "/youtubei/v1/player" requests still return Got playabilityStatus == LOGIN_REQUIRED. followed by Missing hash key: "videoDetails". But my suspicion would be this is related to signature stuff not the poToken itself since it looks like the AndroidTestSuite doesn't use the inv_sig_helper at all.

unixfox commented 2 months ago

@unixfox Bingo. This time the tokens generated from the script work just as the ones I got manually. The AndroidTestSuite "/youtubei/v1/player" requests still return Got playabilityStatus == LOGIN_REQUIRED. followed by Missing hash key: "videoDetails". But my suspicion would be this is related to signature stuff not the poToken itself since it looks like the AndroidTestSuite doesn't use the inv_sig_helper at all.

the po_token generated only works with the WEB client. It will never work for ANDROID client since this token has been generated for a webpage, not on the youtube app for android.

Hence you need https://github.com/iv-org/invidious/pull/4772 too.

MMaster commented 2 months ago

@unixfox Bingo. This time the tokens generated from the script work just as the ones I got manually. The AndroidTestSuite "/youtubei/v1/player" requests still return Got playabilityStatus == LOGIN_REQUIRED. followed by Missing hash key: "videoDetails". But my suspicion would be this is related to signature stuff not the poToken itself since it looks like the AndroidTestSuite doesn't use the inv_sig_helper at all.

the po_token generated only works with the WEB client. It will never work for ANDROID client since this token has been generated for a webpage, not on the youtube app for android.

Hence you need #4772 too.

I got #4772 merged and invidious is using it and adding the signatureTimestamp to WEB requests. But when you look into parser.cr extract_video_info:

it uses WEB to fetch data from the player endpoint
if the returned playabilityStatus is "OK" and if videoId matches the videoId returned in videoDetails it sets reason to nil
if the reason is nil it uses AndroidTestSuite client to call try_fetch_streaming_data

As is visible in traces here: https://github.com/iv-org/invidious/pull/4789#issuecomment-2239915836

MMaster commented 2 months ago

I have pushed the merged code (unixfox:potoken-config with merged #4772) with quick and dirty Dockerfile to compile and then run inv_sig_helper in background in docker here: https://github.com/MMaster/mm-invidious/tree/potoken-config-with-sig (it doesn't contain the latest wording changes in config, but those don't matter for the functionality anyway)

I've added inv_sig_helper directly to it (I said it's quick and dirty :) ) I hope people won't mind - I will remove the branch when it won't be needed anymore.

https://github.com/MMaster/mm-invidious/tree/potoken-config-with-sig

unixfox commented 2 months ago

@MMaster I'm so so sorry, to be transparent with you, I didn't try #4772 yet, and I thought that in this PR the client would be switched from Android to WEB. But I was wrong. In fact it makes perfect sense because some IP addresses are not yet blocked but sig decipher is becoming a must for almost all youtube clients.
Anyway, I'm diverging, you should contact us in private if you want more details.

So in this PR, #4789, I just pushed a new commit. It's quite simple. If po_token parameter is passed, then WEB client is exclusively used, if not Android client is used. TvHtml5ScreenEmbed will still be the fallback.

I've added inv_sig_helper directly to it (I said it's quick and dirty :) ) I hope people won't mind - I will remove the branch when it won't be needed anymore.

I wouldn't recommend doing that because we specifically created a separate process for isolating the invidious process from possible remote code executions from Google itself! So the final plan is to offer inv_sig_helper as a separate docker container for an ideal isolation.

Thank you for the feedback, though! Without you, I think we wouldn't have found this issue!

@SamantazFox @syeopite please re-review the PR.

MMaster commented 2 months ago

@unixfox No worries. I already thought it's because the token doesn't work on Android yesterday, but I just got steered by your comment that Android is required and thought that you got it working so I suspected it's the signature. I also think that the old method of youtube-trusted-session-generator would work just fine since that's the approach how I got the tokens manually that worked for me (except for the Android client).

I've tested the changes and it works properly now. I've also merged the changes to that temporary branch in my repo if anyone else wants to try it in docker.

I'm happy I could help!

I wouldn't recommend doing that because we specifically created a separate process for isolating the invidious process from possible remote code executions from Google itself! So the final plan is to offer inv_sig_helper as a separate docker container for an ideal isolation.

Yeah, I know. I will definitely do that when the inv_sig_helper communication gets finalized. But inv_sig_helper still created just unix domain socket and invidious is automatically using it. So running it in the same container was the easiest and fastest way to get it working for testing since sharing UDS between containers can be tricky and I wasn't going for permanent solution.

EDIT: I also plan to automate the potoken generation to get and configure new token with each invidious restart.

unixfox commented 2 months ago

EDIT: I also plan to automate the potoken generation to get and configure new token with each invidious restart.

Please consult https://github.com/iv-org/youtube-trusted-session-generator/issues/1 if you want to help us.

techmetx11 commented 2 months ago

I wouldn't recommend doing that because we specifically created a separate process for isolating the invidious process from possible remote code executions from Google itself! So the final plan is to offer inv_sig_helper as a separate docker container for an ideal isolation.

Yeah, I know. I will definitely do that when the inv_sig_helper communication gets finalized. But inv_sig_helper still created just unix domain socket and invidious is automatically using it. So running it in the same container was the easiest and fastest way to get it working for testing since sharing UDS between containers can be tricky and I wasn't going for permanent solution.

At the very least, you can run inv_sig_helper in a seperate user, with really strict rules set on it (in fact, this is how i designed it originally without thinking about docker). But seperating inv_sig_helper in its own container should be easier now with TCP/IP support

unixfox commented 1 month ago

Seems like videoplayback requests will still fail (403) if invidious does not provide the po_token as pot parameter: https://github.com/LuanRT/YouTube.js/pull/708#issuecomment-2272217364

Can anyone replicate the issue?

MMaster commented 1 month ago

Seems like videoplayback requests will still fail (403) if invidious does not provide the po_token as pot parameter: LuanRT/YouTube.js#708 (comment)

Can anyone replicate the issue?

From my testing over last 3 weeks:

HD720 playback works fine without any additional changes whole 3 weeks
DASH playback returns 403 without any additional changes
DASH playback worked for a while when I updated youtube client version and changed invidious user-agent to firefox
DASH playback stopped working and started returning 403 2 days after that
It started working again when I changed the logic of inv_sig_helper to always fetch player of video that is being played
later videoplayback started returning 200, 206 for first few requests and 403s for all other requests (HD720 still works)
that's the latest state I'm in and I haven't had time to test more changes, but so far all the changes that stopped 403s were with inv_sig_helper (but maybe its just google doing changes and its all unrelated to it)

I can try to test if adding pot to videoplayback request stops the 403s later.

MMaster commented 1 month ago

I can try to test if adding pot to videoplayback request stops the 403s later.

After quick hack to add pot to the videoplayback url in invidious video_playback route it seems DASH playback works again.

I tested right before adding it and I was getting 403s to all DASH videoplayback requests, as soon as I added pot it started working again. Not sure if it's just temporary like before or not, but I will continue testing it and let you know if anything changes.

I checked with regular youtube and that one is also adding pot to videoplayback requests so it's probably good idea to do it.

unixfox commented 1 month ago

I have just added pot parameter required by YouTube.

iv-org / invidious

Add ability to set po_token and visitor data #4789