Downloading from Skeb returns SSL error

[MarqFJA87]

Ever since this issue, I've resorted to using the executable version for whenever I have to download from a site that I can only access via Psiphon. It's been working fine up until a couple of weeks ago, when I started having such downloads fail all the time, giving me SSL errors. Below is a verbose report from attempting to download from Skeb. And yes, this is using the most up-to-date version of the executable. No, changing "https" to "http" does nothing.

[gallery-dl][debug] Version 1.20.5 - Executable
[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.19041
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.8
[gallery-dl][debug] Starting DownloadJob for 'https://skeb.jp/@sabamori55/works/5'
[skeb][debug] Using SkebPostExtractor for 'https://skeb.jp/@sabamori55/works/5'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): skeb.jp:443
[skeb][debug] HTTPSConnectionPool(host='skeb.jp', port=443): Max retries exceeded with url: /api/users/sabamori55/works/5 (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)')))) (1/5)
[urllib3.connectionpool][debug] Starting new HTTPS connection (2): skeb.jp:443
[skeb][debug] HTTPSConnectionPool(host='skeb.jp', port=443): Max retries exceeded with url: /api/users/sabamori55/works/5 (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)')))) (2/5)
[urllib3.connectionpool][debug] Starting new HTTPS connection (3): skeb.jp:443
[skeb][debug] HTTPSConnectionPool(host='skeb.jp', port=443): Max retries exceeded with url: /api/users/sabamori55/works/5 (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)')))) (3/5)
[urllib3.connectionpool][debug] Starting new HTTPS connection (4): skeb.jp:443
[skeb][debug] HTTPSConnectionPool(host='skeb.jp', port=443): Max retries exceeded with url: /api/users/sabamori55/works/5 (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)')))) (4/5)
[urllib3.connectionpool][debug] Starting new HTTPS connection (5): skeb.jp:443
[skeb][debug] HTTPSConnectionPool(host='skeb.jp', port=443): Max retries exceeded with url: /api/users/sabamori55/works/5 (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)')))) (5/5)
[skeb][error] HttpError: ProxyError: HTTPSConnectionPool(host='skeb.jp', port=443): Max retries exceeded with url: /api/users/sabamori55/works/5 (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)'))))

[Quesito24]

This is weird. I downloaded some artists' posts from Skeb just a few days ago without any problem. I really don't think they changed its code in this period of time. (After writing this, I tested it again.)

I just did this and it worked. gallery-dl.exe https://skeb.jp/@psyg2 (NSFW content) I need to emphasize I didn't use a VPN for this.

By the way, Psiphon is a really slow VPN. Maybe if you try other it will work. You can use a japanese VPN. It's https://www.vpngate.net/en

[MarqFJA87]

I use Psiphon to access Skeb specifically because I can't access the site otherwise due to local censorship.

[mikf]

In the middle of the error message it says:

Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy

Maybe urllib3 had some changes regarding proxy server URLs? Are you using an https:// URL as your proxy?

No, changing "https" to "http" does nothing.

For the skeb URL? That's expected since gallery-dl always uses https:// for skeb, regardless of the input URL, and, even if it didn't, skeb.jp automatically redirects from http:// to https://.

[MarqFJA87]

Are you using an https:// URL as your proxy?

I don't know. How do I check?

[mikf]

It's whatever you specified as proxy in your gallery-dl config file.

If you haven't explicitly set a proxy there, it is using a proxy from the HTTP_PROXY, HTTPS_PROXY, or ALL_PROXY environment variables, which are probably set by Psiphon.

Run echo %HTTP_PROXY% etc to see what they are. If any of them start with https://, that's your problem.

[MarqFJA87]

All I'm getting from running those commands are repeats of the text right after the "echo" command.

[Hrxn]

What are you using? CMD?

[MarqFJA87]

Yes.

[mikf]

I experimented a bit with Psiphon and found out what is going wrong here:

• Psiphon writes its proxy settings to registry entries, not environment variables
• Psiphon uses https://127.0.0.1:<port number> as https proxy, causing the error here
• Overwriting this proxy and fixing it to use http:// in gallery-dl does currently not work, because proxy settings from registry or envvars take precedence over how proxies are set here

So this needs some changes to gallery-dl's proxy handling and you will need to set a static port in Psiphon and explicitly set the proxy address with --proxy 127.0.0.1:<port number>.

[MarqFJA87]

So I will have to wait until a later update to gallery-dl, then? I can live with that.

[mikf]

Thought of a workaround to overwrite the registry settings:

set HTTPS_PROXY=http://127.0.0.1:<port number>

[MarqFJA87]

So... Did the solution get implemented and somehow I didn't notice, or has it been put on the backburner in favor of higher priorities?

[mikf]

Well, you can now override any environment proxy settings with --proxy and fix the wrong https:// proxy scheme set by Psiphon as mentioned in https://github.com/mikf/gallery-dl/issues/2357#issuecomment-1063202633, i.e. the third point is fixed.

[MarqFJA87]

I've run a test, and found out that I don't need to set Psiphon to a fixed port, I just need to lock up the logs menu in the app, which records which HTTP and SOCKS port the app has chosen for the current session from whatever ports are unreserved at the time. I then copy-paste the HTTP port number into the --proxy command from earlier, and voila.

I guess this means that this thread should be closed?

[Hrxn]

As opener, you can close this yourself.

[MarqFJA87]

Yeah, I just didn't want to presume.