Feature request: Proxy for web pages, but not streams

GrayFace commented 1 year ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm requesting a feature unrelated to a specific site
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

Existing --geo-verification-proxy option doesn't always do the job. It helps unblock some URLs, but not the others, e.g. it fails when the site itself does geo-restrictions. It would be great to have a proxy setting that would be used for everything to the point of getting an m3u8 link and then for direct connection to be used when downloading it.

Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

No response

pukkandan commented 1 year ago

~~That is exactly what --geo-verification-proxy does!~~ (I was wrong)

Provide verbose logs of where it does not work.

GrayFace commented 1 year ago

The URL is NSFW, but here's -vU log from this command: yt-dlp.exe https://www.cam4.com/danielseemily --geo-verification-proxy http://127.0.0.1:4001 --fixup never --concurrent-fragments 4 -vU -o rec.ts

[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, m
utagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releas
es/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 5590c57bd0433ed239a2deaaf92e2ad6f37fe50f53664c821575cafe106a
9421
yt-dlp is up to date (stable@2023.03.04)
[CAM4] Extracting URL: https://www.cam4.com/danielseemily
[CAM4] danielseemily: Downloading JSON metadata
ERROR: [CAM4] danielseemily: Unable to download JSON metadata: HTTP Error 403: F
orbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on
https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue t
emplate. Confirm you are on the latest version using  yt-dlp -U
tmpdump 2.4
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, m
utagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releas
es/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 5590c57bd0433ed239a2deaaf92e2ad6f37fe50f53664c821575cafe106a
9421
yt-dlp is up to date (stable@2023.03.04)
[CAM4] Extracting URL: https://www.cam4.com/danielseemily
[CAM4] danielseemily: Downloading JSON metadata
ERROR: [CAM4] danielseemily: Unable to download JSON metadata: HTTP Error 403: F
orbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on
https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue t
emplate. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 694, in extract
  File "yt_dlp\extractor\cam4.py", line 20, in _real_extract
  File "yt_dlp\extractor\common.py", line 1048, in download_content
  File "yt_dlp\extractor\common.py", line 1012, in download_handle
  File "yt_dlp\extractor\common.py", line 882, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 839, in _request_webpage

  File "yt_dlp\extractor\common.py", line 821, in _request_webpage
  File "yt_dlp\YoutubeDL.py", line 3742, in urlopen
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 569, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

If I specify --proxy instead, all goes well.

BTW, this extractor also fails with parse exception when a cam is offline, causing yt-dlp to exit even when "-R infinite" is specified. Should I make a separate issue about this?

pukkandan commented 1 year ago

BTW, this extractor also fails with parse exception when a cam is offline, causing yt-dlp to exit even when "-R infinite" is specified.

Retries are not meant to handle when URL is offline. You can try --wait-for-video and open a site enhancement request if it is not supported.

pukkandan commented 1 year ago

I had a misunderstanding of how the option behaves. Your original request for a new option is valid. But cam4 could also be fixed to make it work with --geo-verification-proxy. So I have split this into 2 separate issues.

coletdjnz commented 9 months ago

related: #1191

coletdjnz commented 5 months ago

In the meantime you can use ALL_PROXY and NO_PROXY env vars to achieve this, if you know the hosts.

yt-dlp / yt-dlp