search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

proxy does not work properly #927

Closed drowsy-probius closed 4 years ago

drowsy-probius commented 4 years ago

Hi My ISP uses deep packet inspection and blocks nsfw sites. So I uses green-tunnel local proxy ( https://github.com/SadeghHayeri/GreenTunnel ) I running several services (selenium crawling, etc... ) through my proxy server and it works perfectly.

But in gallery-dl, it does not. I ran with '--no-check-certificate' option, but the result was same. With '-R -1' option, it rarely works

[log]

gallery-dl --ignore-config --list-keywords --proxy {my proxy server} -R -1 --verbose https://hitomi.la/galleries/1699235.html [gallery-dl][debug] Version 1.14.3 [gallery-dl][debug] Python 3.7.3 - Linux-5.4.0-42-generic-x86_64-with-debian-10.5 [gallery-dl][debug] requests 2.24.0 - urllib3 1.25.10 [gallery-dl][debug] Starting KeywordJob for 'https://hitomi.la/galleries/1699235.html' [hitomi][debug] Using HitomiGalleryExtractor for 'https://hitomi.la/galleries/1699235.html' [urllib3.connectionpool][debug] Starting new HTTPS connection (1): ltn.hitomi.la:443 [hitomi][debug] HTTPSConnectionPool(host='ltn.hitomi.la', port=443): Max retries exceeded with url: /galleries/1699235.js (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)'))) (1/inf) [urllib3.connectionpool][debug] Starting new HTTPS connection (2): ltn.hitomi.la:443 [hitomi][debug] HTTPSConnectionPool(host='ltn.hitomi.la', port=443): Max retries exceeded with url: /galleries/1699235.js (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)'))) (2/inf) [urllib3.connectionpool][debug] Starting new HTTPS connection (3): ltn.hitomi.la:443 [hitomi][debug] HTTPSConnectionPool(host='ltn.hitomi.la', port=443): Max retries exceeded with url: /galleries/1699235.js (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)'))) (3/inf) [urllib3.connectionpool][debug] Starting new HTTPS connection (4): ltn.hitomi.la:443 [hitomi][debug] HTTPSConnectionPool(host='ltn.hitomi.la', port=443): Max retries exceeded with url: /galleries/1699235.js (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)'))) (4/inf) [urllib3.connectionpool][debug] Starting new HTTPS connection (5): ltn.hitomi.la:443 [hitomi][debug] HTTPSConnectionPool(host='ltn.hitomi.la', port=443): Max retries exceeded with url: /galleries/1699235.js (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)'))) (5/inf) [urllib3.connectionpool][debug] Starting new HTTPS connection (6): ltn.hitomi.la:443 [hitomi][debug] HTTPSConnectionPool(host='ltn.hitomi.la', port=443): Max retries exceeded with url: /galleries/1699235.js (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)'))) (6/inf) [urllib3.connectionpool][debug] Starting new HTTPS connection (7): ltn.hitomi.la:443 [hitomi][debug] HTTPSConnectionPool(host='ltn.hitomi.la', port=443): Max retries exceeded with url: /galleries/1699235.js (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)'))) (7/inf) [urllib3.connectionpool][debug] Starting new HTTPS connection (8): ltn.hitomi.la:443 [hitomi][debug] HTTPSConnectionPool(host='ltn.hitomi.la', port=443): Max retries exceeded with url: /galleries/1699235.js (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1056)'))) (8/inf)

mikf commented 4 years ago

I installed the green-tunnel proxy for testing purposes on my own machine and it works over here:

$ gallery-dl -v --proxy 127.0.0.1:8000 https://hitomi.la/galleries/1699235.html
[gallery-dl][debug] Version 1.14.4-dev - Git HEAD: 2b88c90
[gallery-dl][debug] Python 3.8.5 - Linux-5.4.50-1-lts-x86_64-with-glibc2.2.5
[gallery-dl][debug] requests 2.24.0 - urllib3 1.25.10
[gallery-dl][debug] Starting DownloadJob for 'https://hitomi.la/galleries/1699235.html'
[hitomi][debug] Using HitomiGalleryExtractor for 'https://hitomi.la/galleries/1699235.html'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): ltn.hitomi.la:443
[urllib3.connectionpool][debug] https://ltn.hitomi.la:443 "GET /galleries/1699235.js HTTP/1.1" 200 2057
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): hitomi.la:443
[urllib3.connectionpool][debug] https://hitomi.la:443 "GET /galleries/1699235.html HTTP/1.1" 200 324
[urllib3.connectionpool][debug] https://hitomi.la:443 "GET /cg/elf-ninshin-taishi---serena-chan-%ED%95%9C%EA%B5%AD%EC%96%B4-1699235.html HTTP/1.1" 200 2246
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): ca.hitomi.la:443
[urllib3.connectionpool][debug] https://ca.hitomi.la:443 "GET /images/a/32/bae503b3690cebe9d8e3e51f92c0b1e36b329e2ea554ea9478f3a5fdce9af32a.png HTTP/1.1" 200 448885
/tmp/hitomi/1699235 Elf Ninshin Taishi - Serena-chan/hitomi_1699235_001.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): ba.hitomi.la:443
[urllib3.connectionpool][debug] https://ba.hitomi.la:443 "GET /images/2/cd/2390ed29b02f3da5755eb8a7039eb11fe06e6ad6bd85fad8779daa5f7742dcd2.png HTTP/1.1" 200 1949115
/tmp/hitomi/1699235 Elf Ninshin Taishi - Serena-chan/hitomi_1699235_002.png
[urllib3.connectionpool][debug] https://ca.hitomi.la:443 "GET /images/b/89/29c8943594eb240f24fc6ab29052520b0bf0a916c82fc345c3c6c30a4a81e89b.png HTTP/1.1" 200 2441726

It might help to upgrade your Python interpreter to 3.8 and your OpenSSL library to the latest version (1.1.1.g for me). Try the standalone executable, which comes with both of them prepackaged.

Pretty sure this is also not a gallery-dl specific issue, but a more general Python one. You might have more luck asking the folks at requests and urllib3 for help, since all network traffic from here is done via those libraries.

drowsy-probius commented 4 years ago

Does the standalone executable file run properly regardless of host environment?

$ ./gallery-dl.bin [16455] Error loading Python lib '/tmp/_MEIJtiQyf/libpython3.8.so.1.0': dlopen: /lib/x86_64-linux-gnu/libm.so.6: version 'GLIBC_2.29' not found (required by /tmp/_MEIJtiQyf/libpython3.8.so.1.0)

mikf commented 4 years ago

Seems like the included Python3.8 interpreter needs glibc version 2.29 or higher to run, and this might be impossible to achive without upgrading your entire distro.

I'm not really sure how to solve your problem, but maybe you can find something by searching for "SSL WRONG_VERSION_NUMBER". The first few results all made the mistake of trying to establish a SSL/TLS connection to a regular HTTP server, but that's definitely not what's happening here.

Could you also the run the following script and see if the same error pops up, to make sure this is a Python problem and not a gallery-dl one?

import requests

URL = "https://hitomi.la/"
PROXY = "http://127.0.0.1:8000"  # adjust proxy address as needed

requests.get(URL, proxies={"https": PROXY})
drowsy-probius commented 4 years ago 
import requests
requests.get('https://hitomi.la', proxies={"https":"{my_proxy_server}"})

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 667, in urlopen
    self._prepare_proxy(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 932, in _prepare_proxy
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 362, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='hitomi.la', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='hitomi.la', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)')))

request.get does not work. But I think my proxy server works well.

curl https://hitomi.la --proxy {my_proxy_server}
<!DOCTYPE html>
<html>
<head>
<title>Recently Added | Hitomi.la</title>
<meta name="description" content="Hitomi.la is the best source of free Recently Added hentai, doujinshi/doujins, and manga."/>
<meta name="viewport" content="width=device-width,initial-scale=1.0">
<link rel="stylesheet" href="//ltn.hitomi.la/djtemplate.css">
<link rel="stylesheet" href="//ltn.hitomi.la/page.css">
<link rel="stylesheet" href="//ltn.hitomi.la/navbar.css">
<link rel="stylesheet" href="//ltn.hitomi.la/search.css">
<link href="/index-all.atom" type="application/atom+xml" rel="alternate" title="ATOM Feed" />
<script src="//ltn.hitomi.la/jquery.min.js"></script>
<script src="//ltn.hitomi.la/common.js"></script>
<script src="//ltn.hitomi.la/js.cookie.js"></script>
<script src="//ltn.hitomi.la/moveimage.js"></script>
<script src="//ltn.hitomi.la/limitlists.js"></script>
<script src="//ltn.hitomi.la/moment-with-locales.js"></script>
<script src="//ltn.hitomi.la/date.js"></script>
<script src="//ltn.hitomi.la/i8Wspns7.js"></script>
<script src="//ltn.hitomi.la/paging.js"></script>
<script src="//ltn.hitomi.la/galleryblock.js"></script>
...
mikf commented 4 years ago

I case your proxy address starts with https://, try it with http:// (that's what I've read on some other issue with a similar problem, maybe it helps)

Otherwise ask the folks at Requests, urllib3, stackoverflow, etc but this isn't a bug that can be fixed on gallery-dl's end.