curl: err 60 - Nothing was scraped

[seffs]

I just updated to v2.0 on MacOs Mojave. No results get retrieved.

I made sure to erase my config.sh first

[Euro20179]

What is the output of ytfzf --version, the install instructions for 2.0 actually just install the latest commit, so you may have installed 2.1.rc-1 if that's the case try updating to exactly 2.0

git clone https://github.com/pystardust/ytfzf
cd ytfzf
git checkout v2.0
sudo make install man

try running bash ytfzf test make sure your have jq installed (although you should cause... you used ytfzf before lol)

Other than that, I'm not sure, there's just not a lot of information here, and i wanted to make sure the only way to have that error show is if truely nothing was scraped, and it is.

[seffs]

So I just downgraded from 2.1.rc-1 to 2.0 . Same result.

Running bash ytfzf music didn't make any difference.

Running brew upgrade jq gives me: Warning: jq 1.6 already installed

If you tell me how, I can provide more information

[Euro20179]

So I just downgraded from 2.1.rc-1 to 2.0 . Same result.

Fair enough, they use the same mechanism for checking if nothing is scraped, only with a very slight change.

If you tell me how, I can provide more information

Yeah I'm not really sure what information I need, which is the problem.

More ideas:

• make sure you can visit peertube, odysee, and vid.puffyan.us in your webbrowser.
• Try running ytfzf -c T (I have hope for this one)
• one time just having the config existing fixed someone's issue, so maybe touch ~/.config/ytfzf/conf.sh
• could also try running YTFZF_CONFIG_FILE=/dev/null ytfzf search to make sure it's not reading a config file for sure

[seffs]

This might be of interest for you, especially the return code 60. I ran the script in debug mode bash -x. This is the output:

+ YTFZF_VERSION=2.0
+ c_red='\033[1;31m'
+ c_green='\033[1;32m'
+ c_yellow='\033[1;33m'
+ c_blue='\033[1;34m'
+ c_magenta='\033[1;35m'
+ c_cyan='\033[1;36m'
+ c_reset='\033[0m'
+ : 2
+ : 0
+ : 0
+ : 0
+ '[' 0 -eq 0 ']'
+ trap clean_up EXIT
+ '[' 0 -eq 0 ']'
+ trap exit INT TERM HUP
+ dep_check jq
+ command -v jq
+ : /Users/seff/.config/ytfzf
+ : /Users/seff/.config/ytfzf/conf.sh
+ : /Users/seff/.config/ytfzf/subscriptions
+ : /Users/seff/.config/ytfzf/thumbnail-viewers
+ : /Users/seff/.config/ytfzf/interfaces
+ '[' -f /Users/seff/.config/ytfzf/conf.sh ']'
+ : /Users/seff/.config/ytfzf/scrapers
+ : 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.152 Safari/537.36'
+ function_exists external_menu
+ type external_menu
+ function_exists search_prompt_menu
+ type search_prompt_menu
+ : 1
+ :
+ :
+ : left
+ : ueberzug
+ : /usr/lib/w3m/w3mimgdisplay
+ : alt-d
+ : alt-v
+ : alt-m
+ : alt-e
+ : alt-l
+ : alt-f
+ : ''
+ : Enter,double-click,alt-d,alt-v,alt-m,alt-e,alt-l,alt-f,
+ : 210
+ : 0
+ function_exists video_player
+ type video_player
+ function_exists audio_player
+ type audio_player
+ function_exists multimedia_player
+ type multimedia_player
+ function_exists downloader
+ type downloader
+ '[' -z '' ']'
+ dep_check yt-dlp
+ command -v yt-dlp
+ ytdl_path=youtube-dl
+ : /Users/seff/.cache/ytfzf
+ : 0
+ : /Users/seff/.cache/ytfzf/watch_hist
+ : 1
+ : 0
+ : 0
+ : multimedia_player
+ :
+ : 1
+ : best
+ : 0
+ : 0
+ : 0
+ : 1
+ : 0
+ : 0
+ : -
+ : youtube
+ : default
+ : 2
+ : https://vid.puffyan.us
+ : https://vid.puffyan.us
+ : 1
+ : 30
+ : relevance
+ : ''
+ : ''
+ : video
+ : ''
+ : US
+ : false
+ : ' youtube-subscriptions S SI T youtube-trending H history '
+ case "$long_opt_char" in
+ new_line='
'
++ printf '\t'
+ tab_space='   '
+ : '                                                                                                                   '
+ function_exists get_sort_by
+ type get_sort_by
+ function_exists data_sort_fn
+ type data_sort_fn
+ function_exists video_info_text
+ type video_info_text
+ function_exists thumbnail_video_info_text
+ type thumbnail_video_info_text
+ function_exists get_ueberzug_positioning_left
+ type get_ueberzug_positioning_left
+ function_exists get_ueberzug_positioning_right
+ type get_ueberzug_positioning_right
+ function_exists get_ueberzug_positioning_up
+ type get_ueberzug_positioning_up
+ function_exists get_ueberzug_positioning_down
+ type get_ueberzug_positioning_down
+ getopts ac:dfhlmn:rtvxADHI:LTU-: OPT
+ shift 0
+ : music
+ '[' 0 -eq 0 ']'
+ YTFZF_PID=32525
++ printf %s music
++ tr / _
+ SEARCH_PREFIX=music
+ SEARCH_PREFIX=music
+ session_cache_dir=/Users/seff/.cache/ytfzf/music-32525
+ session_temp_dir=/Users/seff/.cache/ytfzf/music-32525/tmp
+ thumb_dir=/Users/seff/.cache/ytfzf/music-32525/thumbnails
+ mkdir -p /Users/seff/.cache/ytfzf/music-32525/tmp /Users/seff/.cache/ytfzf/music-32525/thumbnails
+ : /Users/seff/.cache/ytfzf/music-32525/ids
+ : /Users/seff/.cache/ytfzf/music-32525/videos_json
+ keypress_file=/Users/seff/.cache/ytfzf/music-32525/tmp/menu_keypress
+ :
+ :
+ total_search_sort_by=relevance
+ total_search_upload_date=
+ IFS=,
+ set -f
+ for curr_scrape in '$scrape'
+ printf %s ' youtube-subscriptions S SI T youtube-trending H history '
+ grep -Fqv ' youtube '
+ '[' music = - ']'
+ '[' -z music ']'
+ manage_multi_filters
+ '[' -n relevance ']'
+ search_sort_by=relevance
+ total_search_sort_by=relevance
+ '[' -n '' ']'
+ function_exists manage_multi_custom_filters
+ type manage_multi_custom_filters
+ function_exists on_search
+ type on_search
+ scrape_website youtube music
+ scrape_type=youtube
+ _search=music
+ case $scrape_type in
+ scrape_invidious_search music /Users/seff/.cache/ytfzf/music-32525/videos_json search 1
+ page_query=music
+ output_json_file=/Users/seff/.cache/ytfzf/music-32525/videos_json
+ pagetype=search
+ page_num=1
+ '[' 1 -le 1 ']'
+ _tmp_json=/Users/seff/.cache/ytfzf/music-32525/tmp/yt-search-1.json
+ print_info 'Scraping YouTube (with https://vid.puffyan.us) (music, pg: 1)\n'
+ '[' 2 -ge 0 ']'
+ printf 'Scraping YouTube (with https://vid.puffyan.us) (music, pg: 1)\n'
Scraping YouTube (with https://vid.puffyan.us) (music, pg: 1)
+ _get_request https://vid.puffyan.us/api/v1/search -G --data-urlencode q=music --data-urlencode type=video --data-urlencode sort_by=relevance --data-urlencode date= --data-urlencode duration= --data-urlencode features= --data-urlencode region=US --data-urlencode page=1
+ _base_url=https://vid.puffyan.us/api/v1/search
+ shift 1
+ curl -f https://vid.puffyan.us/api/v1/search -s -L -G --data-urlencode q=music --data-urlencode type=video --data-urlencode sort_by=relevance --data-urlencode date= --data-urlencode duration= --data-urlencode features= --data-urlencode region=US --data-urlencode page=1 -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.152 Safari/537.36' -H 'Accept-Language: en-US,en;q=0.9' --compressed
+ return 60
+ return 60
+ handle_scrape_error 60
+ case "$1" in
+ '[' '!' -s /Users/seff/.cache/ytfzf/music-32525/videos_json ']'
+ die 4 'Nothing was scraped\n'
+ _return_status=4
+ print_error 'Nothing was scraped\n'
+ '[' 2 -ge 2 ']'
+ printf '\033[1;31mNothing was scraped\n\033[0m'
Nothing was scraped
+ exit 4
+ clean_up
+ '[' 0 -eq 0 ']'
+ '[' -d /Users/seff/.cache/ytfzf/music-32525 ']'
+ '[' 0 -eq 0 ']'
+ rm -r /Users/seff/.cache/ytfzf/music-32525

[seffs]

make sure you can visit peertube, odysee, and vid.puffyan.us in your webbrowser.

Yes, I can open them normally.

Try running ytfzf -c T (I have hope for this one)

No luck.

one time just having the config existing fixed someone's issue, so maybe touch ~/.config/ytfzf/conf.sh

Nope.

could also try running YTFZF_CONFIG_FILE=/dev/null ytfzf search to make sure it's not reading a config file for sure

No difference

[Euro20179]

I forgot -x existed, i will now ask everyone to use it lol

It seems curl is exiting with code 60, from the curl man page this means: "Peer certificate cannot be authenticated with known CA certificates." I can't exactly say what certificates are, but I believe that they are files or signed files or something, that let https servers know that something is secure, otherwise you won't be able to connect to https. I'm not really a web developer, so I'm not sure how to fix it, and everything that came up relating to curl exit code 60 on duckduckgo has to do with a php server.

[seffs]

Aaaand found it. Adding -k parameter did the trick. I guess this is MacOS specific. Let me research a little bit on this one

[Euro20179]

Although I would like to add -k to just fix it, that is HORRIBLY insecure, because something something bypassing validity checks (i think lol)

I guess I could add something like --insecure (no -k, so that people understand it's insecure, plus it's kinda niche, no need to waste a single letter on it)

EDIT: after reading more of that stack overflow thread, if people have this same issue, I may just point them to the thread, as adding a --insecure option adds a surprising amount of complexity to the code. If you want in your config you can add this though, which will allow you to use --insecure to enable -k whenever the script uses curl.

on_opt_parse () {
    option="$1"
    value="$2"
    case "$option" in
        insecure) echo "--insecure" >> ~/.curlrc ;;
    esac
}

on_exit () {
    clean_up
    rm ~/.curlrc
}

[ $__is_fzf_preview -eq 0 ] && trap on_exit EXIT

[seffs]

Yeah, that was also a big no for me. I do remember having this problem some time ago, but already forgot the domain.

So this is definitely related to my (outdated) version of Mac OS. Before updating my certificates, I will try my luck by using brew's curl. Currently compiling a dependency on this potato machine. I'll let you know

[Euro20179]

Currently compiling a dependency on this potato machine.

That's always fun, hopefuly it all works.

[seffs]

The solution is simply specifying the path for the certificates file. There are different ways to achieve this, but this is out of the scope of ytfzf. Some thoughts on handling this (might be relevant for Mac OS only):

1. Get SSL_CERT_FILE env variable from within script.
2. User-defined path in ytfzf config file
3. Add new task to Makefile 'curl-60' asking for cert path

In any case, using /usr/local/etc/ca-certificates/cert.pem might be the safest default. Other locations to consider are

• /usr/local/etc/openssl/
• /usr/local/share/ca-certificates
• /usr/share/ca-certificates (linux)

I can open a PR for this. Let me know

[Euro20179]

I can open a PR for this. Let me know

It might be better to just add this to your ~/.curlrc, if it doesn't work then we can do a pr if it doesn't add a lot of complexity.

[seffs]

Didn't know about ~/.curlrc, thanks!

I created/added this to the file:

cacert = /path/to/cert.pem

Then I had problems with python unable to find the certificates. Running brew reinstall openssl@1.1 fixed it. For older machines with no brew support this can be a good starting point. Don't forget to restart Terminal/iTerm2.

I have no other Mac around to test this further. If you are planning to add a 'Troubleshooting' section, this information could be useful.

[Euro20179]

If you are planning to add a 'Troubleshooting' section, this information could be useful.

Yeah i was thinking about mentioning this in the bugs section, if I do ill probably just link to your comment.

[seffs]

I also noticed it takes a little longer to parse the format results compared to v1.2.0 . I was using a patched youtube-dl instead of yt-dlp. Is this related?

[Euro20179]

I also noticed it takes a little longer to parse the format results compared to v1.2.0 . I was using a patched youtube-dl instead of yt-dlp. Is this related?

probably, to get the format menu, it basically just runs $youtube_dl_path --list-formats $url or something like that

[seffs]

Noted. After defining the best quality for me in .conf.sh this is barely noticeable. You can close this issue. Have a nice day!

[Euro20179]

Have a nice day!

you too!