[Bug] "Please sign in" from residential IP, youtube.com works

markozajc commented 1 month ago

Describe the bug The following is shown after attempting to visit any /watch url:

I am not using a proxy and visiting youtube.com lets me play the video without logging in. yt-dlp 2024.09.27 and mpv also work. This appears to be the same issue as #4972, but I don't believe it's the same as #4734.

Steps to Reproduce

Open invidious
Try playing any video, for example /watch?v=aqz-KE-bpKQ

Logs

2024-10-03 10:37:46 UTC [trace] SigHelper: Send transaction 0x6b9d453e / opcode GET_SIGNATURE_TIMESTAMP
2024-10-03 10:37:46 UTC [trace] SigHelper: Send transaction 0x6b9d453e - Done
2024-10-03 10:37:46 UTC [trace] SigHelper: Recv transaction 0x6b9d453e / length 8
2024-10-03 10:37:46 UTC [trace] SigHelper: payload = Bytes[0, 0, 0, 0, 0, 0, 78, 29]
2024-10-03 10:37:46 UTC [trace] SigHelper: Recv transaction 0x6b9d453e - Done
2024-10-03 10:37:46 UTC [trace] SigHelper: Transaction unqueued and data sent to channel
2024-10-03 10:37:46 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/player"
2024-10-03 10:37:46 UTC [trace] YoutubeAPI: ClientConfig: YoutubeAPI::ClientConfig(@client_type=YoutubeAPI::ClientType::WebCreator, @region="US")
2024-10-03 10:37:46 UTC [trace] YoutubeAPI: POST data: {"contentCheckOk" => true, "videoId" => "aqz-KE-bpKQ", "context" => {"client" => {"hl" => "en", "gl" => "US", "clientName" => "WEB_CREATOR", "clientVersion" => "1.20240918.03.00", "osName" => "Windows", "osVersion" => "10.0", "platform" => "DESKTOP", "visitorData" => "SNIP"}}, "racyCheckOk" => true, "user" => {"lockedSafetyMode" => false}, "playbackContext" => {"contentPlaybackContext" => {"html5Preference" => "HTML5_PREF_WANTS", "referer" => "https://www.youtube.com/watch?v=aqz-KE-bpKQ", "signatureTimestamp" => 19997}}, "serviceIntegrityDimensions" => {"poToken" => "SNIP"}, "params" => "2AMB"}
2024-10-03 10:37:46 UTC [error] get_video: aqz-KE-bpKQ : Please sign in
2024-10-03 10:37:46 UTC [warn] i18n: Missing translation key "Please sign in"
2024-10-03 10:37:46 UTC [info] 500 GET /watch?v=aqz-KE-bpKQ 146.11ms

Additional context

Browser: Firefox
OS: Debian trixie
invidious: 53e8a5d62d4d7d66e8819f221a1c7b168102995c
invidious_sig_helper: https://github.com/iv-org/inv_sig_helper/commit/5025e49e6106f93ec06d0e3fd542a51e1c44c25a

svpi11 commented 1 month ago

Same here...

msinfo32github commented 1 month ago

Was encountering the same problem - ran the inv_sig_helper and generated a new visitor_data and po_token. This seems to have solved the issue. Running on a residential IP, with same setup as described by @markozajc

gr3m1in commented 1 month ago

I can confirm that regenerating visitor_data/po_token helps

markozajc commented 1 month ago

I just did the same, and the issue has gone away, but doing so initially didn't help. This sounds like A/B testing, so there's probably more to come. I'll close the issue for now.

unixfox commented 1 month ago

https://github.com/iv-org/invidious/issues/4734

Please don't create duplicated issues thank you. If you have a constant error please logging, it's expected: https://github.com/iv-org/invidious/issues/4734#issuecomment-2365205990

markozajc commented 1 month ago

@unixfox apologies if this has been researched already, but this does not seem like the same bug as #4734:

The error message appears to be different: #4734 uses "Sign in to confirm you’re not a bot", while this issue uses "Please sign in"
As far as I understand, owners of instances affected by #4734 are unable to use yt-dl(p) on their servers, as the ban applies to the IP address. This is not the case here - only invidious is unable to play videos, but yt-dlp and NewPipe work fine on the same IP address.

As I suspected, the issue has reappeared and I'm once again unable to fix it by regenerating po_token and visitor_data. If the problems are indeed separate, please reopen the issue.

If they are the same issue, it might be worth adding the "Please sign in" message and any other details to #4734 to prevent confusion.

unixfox commented 1 month ago

The issue lies with the fact that we have already done all the research possible with the possible solutions that Invidious can handle. yt-dlp is another tool, which means it may differ from how invidious works.

I said to not create duplicated issue because we already know about these issues, and for the moment we can't do anything more than what we have already done. Something that was explained in https://github.com/iv-org/invidious/issues/4734#issuecomment-2365205990

Right now, there are many factors that contribute to the "please sign in" errors or to the fact that it works for some people. This makes it very difficult to diagnose the root cause of everyone.

That's why for the moment we have offering what generally works for most people with a residential IP address: https://docs.invidious.io/installation/. If it doesn't, I'm sorry but you can't use Invidious for the moment. I know this is sad but it's the current state of YouTube against 3rd party clients.

retrocaster commented 1 month ago

I know this is sad but it's the current state of YouTube against 3rd party clients.

@unixfox Can you explain why Android apps such as SmartTube continue to work without any issues? It's also a 3rd party client that is open source with ad skipping and Sponsorblock integration: https://github.com/yuliskov/SmartTube

Caian commented 1 month ago

Sadly I'm facing this issue as well at home and no token generation managed to fix it.

markozajc commented 1 month ago

I've managed to temporarily fix the issue by applying the following patch from https://forgejo.jing.rocks/jing/invidious/:

diff --git a/src/invidious/videos/parser.cr b/src/invidious/videos/parser.cr
index fb8935d9..4683058b 100644
--- a/src/invidious/videos/parser.cr
+++ b/src/invidious/videos/parser.cr
@@ -53,10 +53,6 @@ end
 def extract_video_info(video_id : String)
   # Init client config for the API
   client_config = YoutubeAPI::ClientConfig.new
-  # Use the WEB_CREATOR when po_token is configured because it fully only works on this client
-  if CONFIG.po_token
-    client_config.client_type = YoutubeAPI::ClientType::WebCreator
-  end

   # Fetch data from the player endpoint
   player_response = YoutubeAPI.player(video_id: video_id, params: "2AMB", client_config: client_config)
@@ -106,13 +102,6 @@ def extract_video_info(video_id : String)

   new_player_response = nil

-  # Second try in case WEB_CREATOR doesn't work with po_token.
-  # Only trigger if reason found and po_token configured.
-  if reason && CONFIG.po_token
-    client_config.client_type = YoutubeAPI::ClientType::WebEmbeddedPlayer
-    new_player_response = try_fetch_streaming_data(video_id, client_config)
-  end
-
   # Don't use Android client if po_token is passed because po_token doesn't
   # work for Android client.
   if reason.nil? && CONFIG.po_token.nil?
@@ -125,9 +114,10 @@ def extract_video_info(video_id : String)
   end

   # Last hope
-  # Only trigger if reason found or didn't work wth Android client.
-  # TvHtml5ScreenEmbed now requires sig helper for it to work but doesn't work with po_token.
-  if reason && CONFIG.po_token.nil?
+  # Only trigger if reason found and po_token or didn't work wth Android client.
+  # TvHtml5ScreenEmbed now requires sig helper for it to work but po_token is not required
+  # if the IP address is not blocked.
+  if CONFIG.po_token && reason || CONFIG.po_token.nil? && new_player_response.nil?
     client_config.client_type = YoutubeAPI::ClientType::TvHtml5ScreenEmbed
     new_player_response = try_fetch_streaming_data(video_id, client_config)
   end
@@ -206,11 +196,10 @@ def parse_video_info(video_id : String, player_response : Hash(String, JSON::Any
   end

   video_details = player_response.dig?("videoDetails")
-  if !(microformat = player_response.dig?("microformat", "playerMicroformatRenderer"))
-    microformat = {} of String => JSON::Any
-  end
+  microformat = player_response.dig?("microformat", "playerMicroformatRenderer")

   raise BrokenTubeException.new("videoDetails") if !video_details
+  raise BrokenTubeException.new("microformat") if !microformat

   # Basic video infos

@@ -247,7 +236,7 @@ def parse_video_info(video_id : String, player_response : Hash(String, JSON::Any
     .try &.as_a.map &.as_s || [] of String

   allow_ratings = video_details["allowRatings"]?.try &.as_bool
-  family_friendly = microformat["isFamilySafe"]?.try &.as_bool
+  family_friendly = microformat["isFamilySafe"].try &.as_bool
   is_listed = video_details["isCrawlable"]?.try &.as_bool
   is_upcoming = video_details["isUpcoming"]?.try &.as_bool

diff --git a/src/invidious/yt_backend/youtube_api.cr b/src/invidious/yt_backend/youtube_api.cr
index baa3cd92..d66bf7aa 100644
--- a/src/invidious/yt_backend/youtube_api.cr
+++ b/src/invidious/yt_backend/youtube_api.cr
@@ -29,7 +29,6 @@ module YoutubeAPI
     WebEmbeddedPlayer
     WebMobile
     WebScreenEmbed
-    WebCreator

     Android
     AndroidEmbeddedPlayer
@@ -81,14 +80,6 @@ module YoutubeAPI
       os_version: WINDOWS_VERSION,
       platform:   "DESKTOP",
     },
-    ClientType::WebCreator => {
-      name:       "WEB_CREATOR",
-      name_proto: "62",
-      version:    "1.20240918.03.00",
-      os_name:    "Windows",
-      os_version: WINDOWS_VERSION,
-      platform:   "DESKTOP",
-    },

     # Android

@@ -300,9 +291,8 @@ module YoutubeAPI
     end

     if client_config.screen == "EMBED"
-      # embedUrl https://www.google.com allow loading almost all video that are configured not embeddable
       client_context["thirdParty"] = {
-        "embedUrl" => "https://www.google.com/",
+        "embedUrl" => "https://www.youtube.com/embed/#{video_id}",
       } of String => String | Int64
     end

dakabali commented 1 month ago

Is there eventually a working docker image with the above patch? Thx

matzzze commented 1 month ago

I've managed to temporarily fix the issue by applying the following patch from https://forgejo.jing.rocks/jing/invidious/:
diff --git a/src/invidious/videos/parser.cr b/src/invidious/videos/parser.cr

Hmm, for me the issue with "Please sign in" persists

used the parser.cr file from that fork and replaced the one in master
rebuilt invidious
got a new po_token and userdata via youtube_trusted_session_generator
restarted inv_sig_helper
started invidious

invidious raises this error: get_video: AmRlHwUPQtc : Please sign in [warn] i18n: Missing translation key "Please sign in"

anything specific i need to consider to apply that fix? Might there be other diffs that play a role?

markozajc commented 1 month ago

used the parser.cr file from that fork and replaced the one in master

Make sure you've also replaced src/invidious/yt_backend/youtube_api.cr.

bbsixzz commented 1 month ago

Is there eventually a working docker image with the above patch? Thx

Yes please can someone provide this?

matzzze commented 1 month ago

used the parser.cr file from that fork and replaced the one in master

Make sure you've also replaced src/invidious/yt_backend/youtube_api.cr.

Thanks markozajc!

I have used Master and replaced both files from https://forgejo.jing.rocks/jing/invidious git status On branch master Changes not staged for commit: modified: src/invidious/videos/parser.cr modified: src/invidious/yt_backend/youtube_api.cr

Then rebuild invidious, generated new token and voila I can confirm it works for my private instance.

Anyone knows the reason this is not in master?

retrocaster commented 1 month ago

Anyone knows the reason this is not in master?

Perhaps @unixfox can answer that.

If it's just a temporary fix then maybe that's the answer. We've had a lot of temporary fixes, but they're ultimately kind of pointless if they quickly stop working.

Also I haven't seen an explanation for why this fixes anything. Given that the issues are inherently intermittent, a fix without an explanation doesn't inspire much confidence.

unixfox commented 1 month ago

We are busy on offering more permanent solutions. Things that get shared here may not work after a few weeks or so.

We prefer to tell the public that they can use whatever solution fits to them instead of saying that we fixed the software but 2 weeks after it's broken again.

Anyway, if people here finds a good solution that works for the moment please create a pull request! By good I mean something that doesn't revert a whole PR just "because it works".

Caian commented 1 month ago

We are busy on offering more permanent solutions. Things that get shared here may not work after a few weeks or so.

We prefer to tell the public that they can use whatever solution fits to them instead of saying that we fixed the software but 2 weeks after it's broken again.

Anyway, if people here finds a good solution that works for the moment please create a pull request! By good I mean something that doesn't revert a whole PR just "because it works".

Perhaps making the Web Creator API optional via environment variable, and making it enabled by default so it won't affect the majority of the users? Unfortunately this solution may be more robust, but clearly broke things for several people.

unixfox commented 1 month ago

We are busy on offering more permanent solutions. Things that get shared here may not work after a few weeks or so.

We prefer to tell the public that they can use whatever solution fits to them instead of saying that we fixed the software but 2 weeks after it's broken again.

Anyway, if people here finds a good solution that works for the moment please create a pull request! By good I mean something that doesn't revert a whole PR just "because it works".

Perhaps making the Web Creator API optional via environment variable, and making it enabled by default so it won't affect the majority of the users? Unfortunately this solution may be more robust, but clearly broke things for several people.

We prefer to avoid this scenario. Because this would make the diagnostic of someone else installation difficult.

What if they configured a client that no longer works, we would have to think about it in the diagnostic.

We prefer that everyone is on the same implementation. This way we can more easily understand why it doesn't work for someone for example.

UlyssesZh commented 2 weeks ago

I just found that, for the time being, latest does not have this problem, but master has this problem.

I think the issue should be kept open because it is still affecting.

If it doesn't, I'm sorry but you can't use Invidious for the moment.

It is OK, but it's just "for the moment". We shouldn't close this issue as not planned.

mtarnawa commented 2 weeks ago

@UlyssesZh - I can confirm the same. I was running latest-arm until subscriptions stopped updating. After switching to master build I get 500 on all api calls to videos.

EDIT: Reverting to latest-arm64 image works fine, but no subscription updates.

Drikanis commented 2 weeks ago

I ended up running two instances of the invidious container, one on master and one on latest, and configuring my ingress to use master for only /feed and /channel URLs prefixes. This is probably not a good idea for my database consistency but it seems to work so far, and gives me playback as well as subscriptions.

bbsixzz commented 2 weeks ago

This is so confusing, I guess I'm going to do the same in the meantime.

markozajc commented 2 weeks ago

The revert patch from last month still does the trick for me.

unixfox commented 2 weeks ago

Locking this issue as we know that things broke, new code changes will be up soon. Be patient guys, rome wasn't built in a day.

iv-org / invidious

[Bug] "Please sign in" from residential IP, youtube.com works #4975