ERROR[#01]: Couldn't curl website. Please check your network and try again.

[aris-sk]

Greetings...!!! I love this script But unfortunately since yesterday 31/3/2021 i get this error:

ERROR[#01]: Couldn't curl website. Please check your network and try again.

My network connection is totally fine. I suspect it 's a change in youtube api or something cause i have problem sending youtube urls to vlc also. Mpv works like a charm though. Anyone else with same problem? These are the versions of depedencies i have: ytfzf: 1.1.1 youtube-dl: 2021.02.04.1 fzf: 0.25.1 (6654239) jq: jq-1.5-1-a5b5cbe

I would appreciate some help

[smtnskn]

Same here. I ran the curl command in the get_yt_html function with the verbose option and got the following response:

*   Trying 142.250.179.206:443...
* Connected to www.youtube.com (142.250.179.206) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.google.com
*  start date: Mar 11 14:54:07 2021 GMT
*  expire date: Jun  3 14:54:06 2021 GMT
*  subjectAltName: host "www.youtube.com" matched cert's "*.youtube.com"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f0c295e51b0)
> GET /results?search_query=jim%20keller&sp=CAISBAgAEAE HTTP/2
> Host: www.youtube.com
> accept: */*
> accept-encoding: deflate, gzip, br
> authority: www.youtube.com
> user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.152 Safari/537.36
> accept-language: en-US,en;q=0.9
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 302 
< content-type: application/binary
< x-content-type-options: nosniff
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: Mon, 01 Jan 1990 00:00:00 GMT
< date: Thu, 01 Apr 2021 11:04:28 GMT
< location: https://consent.youtube.com/m?continue=https%3A%2F%2Fwww.youtube.com%2Fresults%3Fsearch_query%3Djim%2Bkeller%26sp%3DCAISBAgAEAE&gl=GB&m=0&pc=yt&uxe=23983172&hl=en&src=1
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=31536000
< server: ESF
< content-length: 0
< x-xss-protection: 0
< set-cookie: CONSENT=PENDING+201; expires=Fri, 01-Jan-2038 00:00:00 GMT; path=/; domain=.youtube.com
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
< 
* Connection #0 to host www.youtube.com left intact

See the location string? Apparently we are blocked by the Google consent form.

[qoheniac]

That's funny, this consent form was so annoying that I started to look for alternative ways to access YouTube content and found this lovely project here. Unfortunately, only to find out that this consent form annoys you, too. I'll keep my fingers crossed for you to work around this!

[smtnskn]

A workaround is to do the query with wget like so:

get_yt_html () {                                                                                                     
    local link="$1"                                                                                                  
    local query="$2"                                                                                                 

    printf "%s" "$(                                                                                                  
    wget -O - --quiet --no-cookie \                                                                                              
      --header 'authority: www.youtube.com' \                                                                        
      --header "user-agent: $useragent" \                                                                            
      --header 'accept-language: en-US,en;q=0.9' \                                                                   
      "${link}?search_query=${query}&sp=${sp}" \                                                                        
    )"                                                                                                               
}   

...but that causes get_yt_json to fail in turn.

Edit: No, hold up. I forgot the --no-cookie option. It works now.

[qoheniac]

That works indeed. Really cool, thank you!

[aris-sk]

Changing to wget instead of curl gives me the following errors:

Scraping Youtube... /usr/bin/ytfzf: 525: /usr/bin/ytfzf: --header: not found /usr/bin/ytfzf: 526: /usr/bin/ytfzf: --header: not found /usr/bin/ytfzf: 527: /usr/bin/ytfzf: --header: not found

[Edit]: Damn.. adding the --no-cookie option worked here too...!!! brilliant !

[aris-sk]

A workaround is to do the query with wget like so:

get_yt_html () {                                                                                                     
    local link="$1"                                                                                                  
    local query="$2"                                                                                                 

    printf "%s" "$(                                                                                                  
    wget -O - --quiet --no-cookie \                                                                                              
      --header 'authority: www.youtube.com' \                                                                        
      --header "user-agent: $useragent" \                                                                            
      --header 'accept-language: en-US,en;q=0.9' \                                                                   
      "${link}?search_query=${query}&sp=${sp}" \                                                                        
    )"                                                                                                               
}   

...but that causes get_yt_json to fail in turn.

Edit: No, hold up. I forgot the --no-cookie option. It works now.

Thanks for keeping this amazing script alive ! :)

[AD17102]

Hi! I tried replacing the get_yt_html method with ours(wget) and it still doesn't work. Any help or suggestion as to why?

[aris-sk]

the replacement for the get_yt_html function with the --no-cookie option is this:

get_yt_html () {

local link="$1"
local query="$2"                                                                                                                    
printf "%s" "$(
wget -O - --quiet \
  -H 'authority: www.youtube.com' \
  -H "user-agent: $useragent" \
  -H 'accept-language: en-US,en;q=0.9' \
  --no-cookie \
  "${link}?search_query=${query}&${sp}" \
)"

}

But there is a problem also with the youtube-dl script. I managed to fix this using youtube-dl as argument with an extra --add-header parameter. So one workaround could be to create a new shell script and add it to your path with the following:

!/bin/bash

url="$1" cmd=$(ytfzf -L $url) mpv --ytdl-raw-options=add-header=\"Cookie:\",format=best $cmd

Notice the add-header="Cookie:" parameter Finally use your new script to make your search in youtube using quotes

This solved it for me

[aris-sk]

the replacement for the get_yt_html function with the --no-cookie option is this:

get_yt_html () {

local link="$1"
local query="$2"                                                                                                                    
printf "%s" "$(
wget -O - --quiet \
  -H 'authority: www.youtube.com' \
  -H "user-agent: $useragent" \
  -H 'accept-language: en-US,en;q=0.9' \
  --no-cookie \
  "${link}?search_query=${query}&${sp}" \
)"

}

But there is a problem also with the youtube-dl script. I managed to fix this using youtube-dl as argument with an extra --add-header parameter. So one workaround could be to create a new shell script and add it to your path with the following:

!/bin/bash

url="$1" cmd=$(ytfzf -L $url) mpv --ytdl-raw-options=add-header="Cookie:",format=best $cmd

Notice the add-header="Cookie:" parameter Finally use your new script to make your search in youtube using quotes

This solved it for me

of course instead of mpv someone could use vlc replacing the last line with the following:

vlc $(youtube-dl --get-url --add-header "Cookie:" -f best $cmd)

[pystardust]

Please verify if its working now. Flag -L of curls seems to have fixed it. I have merged it to master. Try updating ytfzf

[aris-sk]

Please verify if its working now. Flag -L of curls seems to have fixed it. I have merged it to master. Try updating ytfzf

I verify that the script succeeds in getting the search results with the -L parameter using curl. The script works as expected fetching the results.

But as I noted above there is also a problem with the youtube-dl and the google consent probably, which makes it impossible to play protected videos in mpv, vlc or any player capable of playing youtube videos.

This is the output i get:

Search Youtube: system of a down Scraping Youtube... Opening Player: mpv Playing: https://www.youtube.com/watch?v=CSvFpBOe8eY [ytdl_hook] ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. [ytdl_hook] youtube-dl failed, trying to play URL directly ... Failed to recognize file format.

Exiting... (Errors when loading file) ERROR[#03]: Couldn't play the video/audio using the current player: mpv Try updating youtube-dl Check for new versions and report at: https://github.com/pystardust/ytfzf

Of course now the problem is combining ytfzf and youtube-dl to play all videos. I have the latest youtube-dl version (2021.02.04.1)

I changed the YTFZF_PLAYER parameters in the conf.sh using youtube-dl but couldn't make it work. The way that I temporarily managed to fix it, is with a new script i made, in which i pass ytfzf with -L parameter, as argument for mpv or vlc:

!/bin/bash

url="$1" cmd=$(ytfzf -L $url) mpv --ytdl-raw-options=add-header="Cookie:",format=best $cmd

Maybe we can implement a fix to the parameters we give to youtube-dl. I fixed it using the --add-header parameter with the value "Cookie:". I live in Greece and i experience this problem here. Probably it is not the same in other countries

[Kr1ss-XD]

@aris-sk Using the youtube-dl --add-header option might also work. See the youtube-dl(1) man page, section Verbosity / Simulation Options.

If that works, you could add that to the youtube-dl config file, so that it's always included.

There's also a --cookies option btw, which you can point to a cookiejar file.

[aris-sk]

@aris-sk Using the youtube-dl --add-header option might also work. See the youtube-dl(1) man page, section Verbosity / Simulation Options.

If that works, you could add that to the youtube-dl config file, so that it's always included.

There's also a --cookies option btw, which you can point to a cookiejar file.

Of course the --add-header option works, this is exactly what i m using here

But...damn. I didnt think to include it in the config file, though i was checking those arguments.

Creating a config file for youtube-dl (if its not already there) at ~/.config/youtube-dl named "config", and adding:

--add-header "Cookie:"

totally fixes every problem...at least for now Nice work guys :)

[AD17102]

OK so I updated ytfzf and the problem is still there. I can't do ytfzf -am or whatever to search for videos. BUT there is something I found. Turns out I can go to my search history using -H and I can play videos just fine. The problem only occurs when I search directly to youtube/ youtube-dl, AKA: I can't do search queries . I hope this gives some help...

[aris-sk]

OK so I updated ytfzf and the problem is still there. I can't do ytfzf -am or whatever to search for videos. BUT there is something I found. Turns out I can go to my search history using -H and I can play videos just fine. The problem only occurs when I search directly to youtube/ youtube-dl, AKA: I can't do search queries . I hope this gives some help...

did you try to set a config file for youtube-dl with the --add-header option i mentioned above?

[AD17102]

OK so I updated ytfzf and the problem is still there. I can't do ytfzf -am or whatever to search for videos. BUT there is something I found. Turns out I can go to my search history using -H and I can play videos just fine. The problem only occurs when I search directly to youtube/ youtube-dl, AKA: I can't do search queries . I hope this gives some help...

did you try to set a config file for youtube-dl with the --add-header option i mentioned above?

I can't seem to find it. It's not in ~/config because the youtube-dl directory doesn't exist. I tries reinstalling youtube-dl and it didn't work. I know I can create the directory with the file but are there any other files that are supposed to be in it?

[aris-sk]

OK so I updated ytfzf and the problem is still there. I can't do ytfzf -am or whatever to search for videos. BUT there is something I found. Turns out I can go to my search history using -H and I can play videos just fine. The problem only occurs when I search directly to youtube/ youtube-dl, AKA: I can't do search queries . I hope this gives some help...

did you try to set a config file for youtube-dl with the --add-header option i mentioned above?

I can't seem to find it. It's not in ~/config because the youtube-dl directory doesn't exist. I tries reinstalling youtube-dl and it didn't work. I know I can create the directory with the file but are there any other files that are supposed to be in it?

If this file doesnt exist, you just create it.

You create ~/.config/youtube-dl/config , and you add this line:

--add-header "Cookie:"

Notice that the folder is ~/.config and not ~/config. No need for other files in it. If there is no other problem and you have updated ytfzf it will work. At least i have not any kind of problem since i did this. You said you can't make queries. This has nothing to do with youtube-dl. Check if u have all dependencies for ytfzf.

[AD17102]

I tried the config file (.config/youtube-dl/config ) and added the --add-header "Cookie:" and it didn't work. I tried to use the command line for youtube-dl with the argument and I got this:

Usage: youtube-dl [OPTIONS] URL [URL...]

youtube-dl: error: wrong header formatting, it should be key:value, not "Cookie" All dependencies are installed and ytfzf is up to date. Whether it's curl or wget it gives the same error

[Kr1ss-XD]

@AD17102 I get this error message when I omit the colon (":") after Cookie. The complete argument should be

--add-header "Cookie:"

(you might even omit the quotation marks, but the colon is crucial)

[aris-sk]

I tried the config file (.config/youtube-dl/config ) and added the --add-header "Cookie:" and it didn't work. I tried to use the command line for youtube-dl with the argument and I got this:

Usage: youtube-dl [OPTIONS] URL [URL...]

youtube-dl: error: wrong header formatting, it should be key:value, not "Cookie" All dependencies are installed and ytfzf is up to date. Whether it's curl or wget it gives the same error

off course ":" is crucial.

[AD17102]

I tried the config file (.config/youtube-dl/config ) and added the --add-header "Cookie:" and it didn't work. I tried to use the command line for youtube-dl with the argument and I got this: Usage: youtube-dl [OPTIONS] URL [URL...] youtube-dl: error: wrong header formatting, it should be key:value, not "Cookie" All dependencies are installed and ytfzf is up to date. Whether it's curl or wget it gives the same error

off course ":" is crucial.

I think I found my main problem. For whatever reason when I call the /usr/bin command (the one that Makefile creates ) the problem still occurs, however when I call it locally in the ytfzf directory (./ytfzf) it works just fine. I tried deleting the contents of /usr/bin/ytfzf and replacing it with the line : ./PATH_TO_FILE/ytfzf and it still doesn't work. I can still use the script but I have to go to the ytfzf directory to use it... Hope this helps

[aris-sk]

I tried the config file (.config/youtube-dl/config ) and added the --add-header "Cookie:" and it didn't work. I tried to use the command line for youtube-dl with the argument and I got this: Usage: youtube-dl [OPTIONS] URL [URL...] youtube-dl: error: wrong header formatting, it should be key:value, not "Cookie" All dependencies are installed and ytfzf is up to date. Whether it's curl or wget it gives the same error

off course ":" is crucial.

I think I found my main problem. For whatever reason when I call the /usr/bin command (the one that Makefile creates ) the problem still occurs, however when I call it locally in the ytfzf directory (./ytfzf) it works just fine. I tried deleting the contents of /usr/bin/ytfzf and replacing it with the line : ./PATH_TO_FILE/ytfzf and it still doesn't work. I can still use the script but I have to go to the ytfzf directory to use it... Hope this helps

How did u get ytfzf and what os do you use? You cloned the git repo? You used a distro repository? You dont provide much info. Maybe u have done something wrong.

This line as stated, is not suitable for every distro:

curl -sL "https://raw.githubusercontent.com/pystardust/ytfzf/master/ytfzf" | sudo tee /usr/bin/ytfzf >/dev/null && sudo chmod 755 /usr/bin/ytfzf

I believe almost all distros have the '/usr/bin' folder in path so you should not have problem. Alternatively in a ubuntu based system you can try:

export PATH=$PATH:/usr/bin

then try again ytfzf

if this works, add the above line to you ~/.bashrc file

Another way is to clone the repo:

git clone https://github.com/pystardust/ytfzf

then:

cd ytfzf sudo make install

and then try again

[AD17102]

I tried the config file (.config/youtube-dl/config ) and added the --add-header "Cookie:" and it didn't work. I tried to use the command line for youtube-dl with the argument and I got this: Usage: youtube-dl [OPTIONS] URL [URL...] youtube-dl: error: wrong header formatting, it should be key:value, not "Cookie" All dependencies are installed and ytfzf is up to date. Whether it's curl or wget it gives the same error

off course ":" is crucial.

I think I found my main problem. For whatever reason when I call the /usr/bin command (the one that Makefile creates ) the problem still occurs, however when I call it locally in the ytfzf directory (./ytfzf) it works just fine. I tried deleting the contents of /usr/bin/ytfzf and replacing it with the line : ./PATH_TO_FILE/ytfzf and it still doesn't work. I can still use the script but I have to go to the ytfzf directory to use it... Hope this helps

How did u get ytfzf and what os do you use? You cloned the git repo? You used a distro repository? You dont provide much info. Maybe u have done something wrong.

This line as stated, is not suitable for every distro:

curl -sL "https://raw.githubusercontent.com/pystardust/ytfzf/master/ytfzf" | sudo tee /usr/bin/ytfzf >/dev/null && sudo chmod 755 /usr/bin/ytfzf

I believe almost all distros have the '/usr/bin' folder in path so you should not have problem. Alternatively in a ubuntu based system you can try:

export PATH=$PATH:/usr/bin

then try again ytfzf

if this works, add the above line to you ~/.bashrc file

Another way is to clone the repo:

git clone https://github.com/pystardust/ytfzf

then:

cd ytfzf sudo make install

and then try again

I FOUND THE PROBLEM. Apparently there was an old yfzfz file in /usr/local/bin and every time I called the program it launched that specific file. I deleted it and reinstalled everything and it works perfectly. I don't know what was the file doing in there considering that Makefile creates files in /usr/bin, but now if anyone has a similar problem:

Just delete all ytfzf files in your $PATH and reinstall with "sudo make uninstall " followed by "sudo make install".

Thanks a lot for the help and I', sorry for causing suffering for the last 5+ days lol....

[pystardust]

The problem seems to be resolved. I am closing this issue