search

iv-org / invidious

Invidious is an alternative front-end to YouTube
https://invidious.io
GNU Affero General Public License v3.0
16.27k stars 1.81k forks source link

[Question] Search fails with Could not extract JSON, Youtube API error 400, when not logged in #4206

Closed nphantasm closed 11 months ago

nphantasm commented 11 months ago

I have a private instance and every time I try to search, I get the Youtube API error 400, when I'm not logged in. Video playback works fine. This has been happening for at least a month.

If I log in, search, copy the request, manually remove the cookie from headers and resend the request I get the original error.

I'm pretty sure something is broken in my configuration (I haven't seen this issue on any other instance), but I wasn't able to find anything regarding this in the documentation, example config file or in the issues.

Commit: 8087e64d (latest at the time of writing) Install: Manual install (no Docker)

Screenshot: invidious error

Logs from logged out session without modifying the request:

2023-10-27 09:50:40 UTC [error] YoutubeAPI: Got error 400 when requesting /youtubei/v1/search
2023-10-27 09:50:40 UTC [error] YoutubeAPI: Request contains an invalid argument.
2023-10-27 09:50:40 UTC [info] YoutubeAPI: POST data was: {"query" => "test", "context" => {"client" => {"hl" => "en", "gl" => "us", "clientName" => "WEB", "clientVersion" => "2.20230602.01.00", "clientScreen" => "WATCH_FULL_SCREEN", "osName" => "Windows", "osVersion" => "10.0", "platform" => "DESKTOP"}}, "params" => ""}
2023-10-27 09:50:40 UTC [warn] i18n: Missing translation key "Could not extract JSON. Youtube API returned error 400 with message:<br>"Request contains an invalid argument.""
2023-10-27 09:50:40 UTC [info] 500 GET /search?q=test 91.22ms

Full configuration:

---
channel_threads: 1
channel_refresh_interval: 30
feed_threads: 1
output: invidious.log
log_level: Info
database_url: postgres://kemal:REDACTED@localhost:5432/invidious?auth_methods=scram-sha-256
decrypt_polling: false
full_refresh: false
https_only: true
hmac_key: REDACTED
domain: invidious.fluffytail.org
use_pubsub_feeds: true
popular_enabled: true
captcha_enabled: true
login_enabled: true
registration_enabled: false
statistics_enabled: false
admins:
- nphantasm
external_port: 443
default_user_preferences:
  annotations: false
  annotations_subscribed: false
  autoplay: false
  captions:
  - ""
  - ""
  - ""
  comments:
  - youtube
  - ""
  continue: false
  continue_autoplay: true
  dark_mode: "auto"
  latest_only: false
  listen: false
  local: false
  locale: en-US
  watch_history: true
  max_results: 40
  notifications_only: false
  player_style: invidious
  quality: hd720
  quality_dash: 1080p
  default_home: Trending
  feed_menu:
  - Subscriptions
  - Playlists
  automatic_instance_redirect: false
  region: US
  related_videos: true
  sort: published
  speed: 1.0
  thin_mode: false
  unseen_only: false
  video_loop: false
  extend_desc: false
  volume: 100
  vr_mode: true
  show_nick: true
  save_player_pos: false
dmca_content: []
check_tables: false
cache_annotations: false
hsts: false
disable_proxy: false
enable_user_notifications: true
modified_source_code_url: ""
force_resolve:
port: 3001
host_binding: 0.0.0.0
pool_size: 100
use_innertube_for_captions: false
cookies: ""
captcha_api_url: https://api.anti-captcha.com
playlist_length_limit: 500
jobs:
  bypass_captcha:
    enable: true
  clear_expired_items:
    enable: true
  notification:
    enable: true
  pull_popular_videos:
    enable: true
  refresh_channels:
    enable: true
  refresh_feeds:
    enable: true
  statistics_refresh:
    enable: true
  subscribe_to_feeds:
    enable: true
  update_decrypt_function:
    enable: true
unixfox commented 11 months ago

Just to be sure, what's after "Current version" at the bottom of the page.

nphantasm commented 11 months ago

obrazek

unixfox commented 11 months ago

So you don't have any errors when you are logged in?

nphantasm commented 11 months ago

So you don't have any errors when you are logged in?

There's basically nothing even on debug level. I might try to get Invidious to log the successful request to Youtube as well and compare them.

Side note: I also found out, that my crystal version was 1.10.1. I downgraded to 1.8.2 and recompiled. It didn't fix anything, but it might have done some damage to the DB, although unlikely. I'll install Invidious on another server with the correct crystal version from the start, clean DB and report back.

Logged out and broken search (full debug log):

2023-11-01 18:54:39 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/search"
2023-11-01 18:54:39 UTC [debug] extract_items: Successfully unpacked container with "Extractors::ContinuationContent"
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:39 UTC [debug] parse_item: Successfully parsed via Parsers::ContinuationItemRendererParser
2023-11-01 18:54:39 UTC [debug] RefreshChannelsJob: <CHANNEL> fiber : Done
2023-11-01 18:54:39 UTC [debug] RefreshChannelsJob: <CHANNEL> : Spawning fiber
2023-11-01 18:54:39 UTC [debug] fetch_channel: <CHANNEL>
2023-11-01 18:54:39 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/browse"
2023-11-01 18:54:39 UTC [error] YoutubeAPI: Got error 400 when requesting /youtubei/v1/search
2023-11-01 18:54:39 UTC [error] YoutubeAPI: Request contains an invalid argument.
2023-11-01 18:54:39 UTC [info] YoutubeAPI: POST data was: {"query" => "test", "context" => {"client" => {"hl" => "en", "gl" => "us", "clientName" => "WEB", "clientVersion" => "2.20230602.01.00", "clientScreen" => "WATCH_FULL_SCREEN", "osName" => "Windows", "osVersion" => "10.0", "platform" => "DESKTOP"}}, "params" => ""}
2023-11-01 18:54:39 UTC [warn] i18n: Missing translation key "Could not extract JSON. Youtube API returned error 400 with message:<br>"Request contains an invalid argument.""
2023-11-01 18:54:39 UTC [info] 500 GET /search?q=test 205.81ms
2023-11-01 18:54:39 UTC [debug] extract_items: Successfully unpacked container with "Extractors::ContinuationContent"

Logged in and working search(slightly truncated debug log at the end):

2023-11-01 18:54:53 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/search"
2023-11-01 18:54:53 UTC [debug] RefreshChannelsJob: <CHANNEL> fiber : Done
2023-11-01 18:54:53 UTC [debug] RefreshChannelsJob: <CHANNEL> : Spawning fiber
2023-11-01 18:54:53 UTC [debug] fetch_channel: <CHANNEL>
2023-11-01 18:54:53 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/browse"
2023-11-01 18:54:53 UTC [debug] extract_items: Successfully unpacked container with "Extractors::ContinuationContent"
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::ContinuationItemRendererParser
2023-11-01 18:54:53 UTC [debug] RefreshChannelsJob: <CHANNEL> fiber : Done
2023-11-01 18:54:53 UTC [debug] RefreshChannelsJob: <CHANNEL> : Spawning fiber
2023-11-01 18:54:53 UTC [debug] fetch_channel: <CHANNEL>
2023-11-01 18:54:53 UTC [debug] YoutubeAPI: Using endpoint: "/youtubei/v1/browse"
2023-11-01 18:54:53 UTC [debug] extract_items: Successfully unpacked container with "Extractors::ContinuationContent"
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
2023-11-01 18:54:53 UTC [debug] parse_item: Successfully parsed via Parsers::RichItemRendererParser
... (A bunch of debug parsers output)
2023-11-01 18:54:53 UTC [info] 200 GET /search?q=test 798.09ms
nphantasm commented 11 months ago

Alright, I figured it out. The region on my server was set to "us" (all lowercase). The reason why it wasn't broken while logged in is because I have a different region set for my account.

I probably didn't restart Invidious when I found that originally, but already copied the config (above) and reverted it on the server. And because Invidious takes most configuration options at face value it doesn't log any errors in the config.

Thank you for your time. Closing.