About IrfanView and Proxy(socks.wrapmodule(urllib))

[bluerthanever]

• [Y] Did you read FAQ section?
• [Y] Did you test with the latest releases or commit ?

First of all. I would like to give you a big hug because I can finally remove my python2.7 now. Actually I have 2 minor questions I would like to ask.

1. I installed IrfanView and I set up the config file, but it NEVER started. Which then turns out to be that I installed the 64bit version, and here it was set to use the 32bit version. Not a big thing though. https://github.com/Nandaka/PixivUtil2/blob/88a3db78e681ab028ad1048a07be17dd06e6ba2f/PixivHelper.py#L276

2. This question is about "socks.py". I just downloaded the new version "20191214" and I was trying to run it in Pycharm after I did some alteration, instead of using the made-ready exe. The thing is, if I don't put any "socks.py" file in the directory, there would be an error like:

   Traceback (most recent call last):
   File "F:/PixivUtil2-20191214/PixivUtil2.py", line 2485, in <module>
   main()
   File "F:/PixivUtil2-20191214/PixivUtil2.py", line 2321, in main
   __br__ = PixivBrowserFactory.getBrowser(config=__config__)
   File "F:\PixivUtil2-20191214\PixivBrowserFactory.py", line 716, in getBrowser
   _browser = PixivBrowser(defaultConfig, defaultCookieJar)
   File "F:\PixivUtil2-20191214\PixivBrowserFactory.py", line 64, in __init__
   self._configureBrowser(config)
   File "F:\PixivUtil2-20191214\PixivBrowserFactory.py", line 93, in _configureBrowser
   socks.wrapmodule(urllib)
   AttributeError: module 'socks' has no attribute 'wrapmodule'

   which I think means I have "socks.py" in my system but not the one you are using.

If I put the "socks.py" file which you had provided in one of someone else's previous issues, it goes like this:

Traceback (most recent call last):
  File "F:/PixivUtil2-20191214/PixivUtil2.py", line 2485, in <module>
    main()
  File "F:/PixivUtil2-20191214/PixivUtil2.py", line 2321, in main
    __br__ = PixivBrowserFactory.getBrowser(config=__config__)
  File "F:\PixivUtil2-20191214\PixivBrowserFactory.py", line 716, in getBrowser
    _browser = PixivBrowser(defaultConfig, defaultCookieJar)
  File "F:\PixivUtil2-20191214\PixivBrowserFactory.py", line 64, in __init__
    self._configureBrowser(config)
  File "F:\PixivUtil2-20191214\PixivBrowserFactory.py", line 93, in _configureBrowser
    socks.wrapmodule(urllib)
  File "F:\PixivUtil2-20191214\socks.py", line 107, in wrapmodule
    module.socket.socket = socksocket
AttributeError: module 'urllib' has no attribute 'socket'

which I think means I might be using the right "socks.py", but something is wrong with my "urllib"? I don't remember messing with "urllib" at all, so it should be what comes with python3 installer I guess.

So I am now a little confused... Could you provide the "socks.py" you are using, or tell me how you installed your socks, or should I be like using another "urllib"?

Looking forward to your reply. Rds

[bluerthanever]

BTW, I'm using PyCharm, and I had these installed in my computer, and my python version is 3.8 for this downloaded project. PySocks 1.7.1 SocksiPy-branch 1.1 socks => I don't know why but this one's version is 0

So I uninstalled them all and then installed PySocks only. And now I don't have to put "socks.py" under the directory, but the "module 'urllib' has no attribute 'socket'" is still there. And I noticed that in requirements, SocksiPy-branch is needed, but if I install this only, it goes back to the "module 'socks' has no attribute 'wrapmodule'" error.

And another thing, if I use the made-ready exe, it gives "module 'socks' has no attribute 'wrapmodule'" when I had only PySocks installed.

[bluerthanever]

Actually I just commented this line, and the tool seems to be working fine without this code so far, downloading fanbox posts now... https://github.com/Nandaka/PixivUtil2/blob/88a3db78e681ab028ad1048a07be17dd06e6ba2f/PixivBrowserFactory.py#L93 Now I am more confused....

[Nandaka]

looks like socks doesn't use wrapmodule anymore. I have updated the code based on https://stackoverflow.com/a/14512227, can you try it?

[bluerthanever]

That was quick. Hold on for a sec.

[bluerthanever]

Works fine for me so far. Thanks. =)

---

Hey, sorry to bother again. I just want you to know that I can't connect via socks5 proxy if I only have sockspy-branch installed. I can connect with pysocks installed though. Not sure if it's just me.

[bluerthanever]

Hey. Sorry to bother again. I am not sure if I did something with the modules or what, but today I try to run the scripts with a socks5 proxy as usual, and got these:

PixivDownloader2 version 20200524
https://github.com/Nandaka/PixivUtil2/releases
Donate at https://bit.ly/PixivUtilDonation
Reading \PIXIV\config.ini ...
done.
<urlopen error [Errno 11004] getaddrinfo failed> 1 2
<urlopen error [Errno 11004] getaddrinfo failed> 1 2
<urlopen error [Errno 11004] getaddrinfo failed> 1 2
Error at open_with_retry(): (<class 'urllib.error.URLError'>, URLError(gaierror(11004, 'getaddrinfo failed')), <traceback object at 0x000001997FFBD480>)
Traceback (most recent call last):
  File "D:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 1229, in do_open
    h.request(str(req.get_method()), str(req.get_selector()), req.data,
  File "D:\Program Files\Python\Python38\lib\http\client.py", line 1230, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "D:\Program Files\Python\Python38\lib\http\client.py", line 1276, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "D:\Program Files\Python\Python38\lib\http\client.py", line 1225, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "D:\Program Files\Python\Python38\lib\http\client.py", line 1004, in _send_output
    self.send(msg)
  File "D:\Program Files\Python\Python38\lib\http\client.py", line 944, in send
    self.connect()
  File "D:\Program Files\Python\Python38\lib\http\client.py", line 1392, in connect
    super().connect()
  File "D:\Program Files\Python\Python38\lib\http\client.py", line 915, in connect
    self.sock = self._create_connection(
  File "D:\Program Files\Python\Python38\lib\socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "D:\Program Files\Python\Python38\lib\socket.py", line 914, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11004] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\PixivUtil2-master\PixivBrowserFactory.py", line 172, in open_with_retry
    return self.open(url, data, timeout)
  File "D:\Program Files\Python\Python38\lib\site-packages\mechanize\_mechanize.py", line 257, in open
    return self._mech_open(url_or_request, data, timeout=timeout)
  File "D:\Program Files\Python\Python38\lib\site-packages\mechanize\_mechanize.py", line 287, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "D:\Program Files\Python\Python38\lib\site-packages\mechanize\_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "D:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 425, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "D:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 414, in _call_chain
    result = func(*args)
  File "D:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 1276, in https_open
    return self.do_open(conn_factory, req)
  File "D:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 1233, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11004] getaddrinfo failed>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "PixivUtil2-Patched.py", line 50, in <module>
    PixivUtil2.main()
  File "F:\PixivUtil2-master\PixivUtil2.py", line 1501, in main
    PixivHelper.check_version()
  File "F:\PixivUtil2-master\PixivHelper.py", line 965, in check_version
    result = br.open_with_retry("https://raw.githubusercontent.com/Nandaka/PixivUtil2/master/PixivConstant.py", retry=3)
  File "F:\PixivUtil2-master\PixivBrowserFactory.py", line 190, in open_with_retry
    raise PixivException("Failed to get page: {0}, please check your internet connection/firewall/antivirus."
PixivException.PixivException: 9005 Failed to get page: https://raw.githubusercontent.com/Nandaka/PixivUtil2/master/PixivConstant.py, please check your internet connection/firewall/antivirus., hasDumpPage=N, pageData=None

I have tried reinstall all modules in requirements, and also uninstall then install python 3.8 all over again, GAWD that took me long enough, and I still got the same error. I checked the proxy gateway and it seems to me that the proxy settings didn't work at all, no data was sent thought proxy.

Could you please help me out with this? Thanks in advance.

[bluerthanever]

Can I still reopen this? or do I need to create a new one?

[bluerthanever]

BTW, HTTP proxies work fine. It's just socks5 ones that doesn't work I guess.

[Nandaka]

no changes on that portion though? also error is different.

urllib.error.URLError: <urlopen error [Errno 11004] getaddrinfo failed>

this error because it cannot resolve the dns. Your isp block the access to pixiv?

EDIT:

PixivException.PixivException: 9005 Failed to get page: https://raw.githubusercontent.com/Nandaka/PixivUtil2/master/PixivConstant.py, please check your internet connection/firewall/antivirus., hasDumpPage=N, pageData=None

looks like access to github...

[bluerthanever]

yup... i can not access pixiv directly, but fanbox.cc, yes. weirdly. and as a matter of fact, i can access github directly, just that the connection is slow. but there's no partial proxy settings so everything should go through proxy if useproxy is set to true, right?

[Nandaka]

can you get the latest commit?

[bluerthanever]

Just did. and nothing changed.

[Nandaka]

hmm, try to access github with the socks proxy from browser? or try to set checknewversion to False in config.ini

[bluerthanever]

hmm... i can access github from browser with socks5 proxy. and i set checknewversion to false and then it went on fine......

why is that...? that the socks5 proxy set as follows doesn't parse the address with proxy? and that i can not get ip address of github directly but can get that of pixiv?

socksType = socks.PROXY_TYPE_SOCKS5 if parseResult.scheme == 'socks5' else socks.PROXY_TYPE_SOCKS4
                PixivHelper.get_logger().info(f"Using SOCKS5 Proxy= {parseResult.hostname}:{parseResult.port}")
                # https://stackoverflow.com/a/14512227
                socks.setdefaultproxy(socksType, parseResult.hostname, parseResult.port)
                socket.socket = socks.socksocket

[Nandaka]

not sure about that, but I have updated the logic to pass the config too.

Also you can try to use simple dns crypt https://simplednscrypt.org/, which I use to access reddit/etc when I was in Indonesia

[Nandaka]

reading the documentation in https://pypi.org/project/PySocks/ and source code, the set_default_proxy() shoud resolve the dns by default.

[bluerthanever]

haha. i have that installed actually... it's just that when i try to access domestic websites, it gets really slow... so i don't have it on all the time. and i tend to just use proxies...

i just tried to set rdns to false and still got the same error...

[bluerthanever]

i just turned simpledns on and tried again. and now it has no error. so... maybe socks doesn't actually resolve dns?

[bluerthanever]

I just found something somewhere and I flushed the dns and added the following code under socket.socket = socks.socksocket and i somehow can get through now.

                def getaddrinfo(*args):
                    return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]
                socket.getaddrinfo = getaddrinfo

[Nandaka]

gotcha, that one from stackoverflow, right? I will update and push the commit.

[bluerthanever]

okay. thank you very much. ;p

[bluerthanever]

Hey, Nandaka. It's me again. I ran into another problem. After the update on getaddrinfo, the program throws following errors. I tried to locate where it occurs and I think it happens when resolving ip address of oauth.secure.pixiv.net, and I guess it's because it has the same pattern as an ip address and the method automatically assumes that it is one..

Using OAuth to retrieve member info for: *****
Traceback (most recent call last):
  File "C:\Program Files\Python\Python38\lib\site-packages\socks.py", line 809, in connect
    negotiate(self, dest_addr, dest_port)
  File "C:\Program Files\Python\Python38\lib\site-packages\socks.py", line 443, in _negotiate_SOCKS5
    self.proxy_peername, self.proxy_sockname = self._SOCKS5_request(
  File "C:\Program Files\Python\Python38\lib\site-packages\socks.py", line 520, in _SOCKS5_request
    resolved = self._write_SOCKS5_address(dst, writer)
  File "C:\Program Files\Python\Python38\lib\site-packages\socks.py", line 583, in _write_SOCKS5_address
    addr_bytes = socket.inet_pton(family, host)
OSError: illegal IP address string passed to inet_pton

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python\Python38\lib\site-packages\urllib3\contrib\socks.py", line 90, in _new_conn
    conn = socks.create_connection(
  File "C:\Program Files\Python\Python38\lib\site-packages\socks.py", line 209, in create_connection
    raise err
  File "C:\Program Files\Python\Python38\lib\site-packages\socks.py", line 199, in create_connection
    sock.connect((remote_host, remote_port))
  File "C:\Program Files\Python\Python38\lib\site-packages\socks.py", line 47, in wrapper
    return function(*args, **kwargs)
  File "C:\Program Files\Python\Python38\lib\site-packages\socks.py", line 814, in connect
    raise GeneralProxyError("Socket error", error)
socks.GeneralProxyError: Socket error: illegal IP address string passed to inet_pton

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "C:\Program Files\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "C:\Program Files\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "C:\Program Files\Python\Python38\lib\site-packages\urllib3\connection.py", line 308, in connect
    conn = self._new_conn()
  File "C:\Program Files\Python\Python38\lib\site-packages\urllib3\contrib\socks.py", line 121, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x000002657E2F1DF0>: Failed to establish a new connection: illegal IP address string passed to inet_pton

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python\Python38\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Program Files\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 724, in urlopen
    retries = retries.increment(
  File "C:\Program Files\Python\Python38\lib\site-packages\urllib3\util\retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: SOCKSHTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x000002657E2F1DF0>: Failed to establish a new connection: illegal IP address string passed to inet_pton'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\PixivUtil2-master\PixivUtil2.py", line 1628, in main
    np_is_valid, op_is_valid, selection = main_loop(ewd, op_is_valid, selection, np_is_valid, args)
  File "PixivUtil2-Patched.py", line 23, in main_loop_patched
    PixivUtil2.main_loop(False, False, None, False, [])
  File "F:\PixivUtil2-master\PixivUtil2.py", line 1366, in main_loop
    menu_fanbox_download_by_id(op_is_valid, args)
  File "PixivUtil2-Patched.py", line 41, in menu_fanbox_download_by_ids
    PixivUtil2.processFanboxArtistById(artist_id, end_page, title_prefix)
  File "F:\PixivUtil2-master\PixivUtil2.py", line 1057, in processFanboxArtistById
    artist = __br__.fanboxGetArtistById(id)
  File "F:\PixivUtil2-master\PixivBrowserFactory.py", line 893, in fanboxGetArtistById
    self.getMemberInfoWhitecube(artist.artistId, pixivArtist)
  File "F:\PixivUtil2-master\PixivBrowserFactory.py", line 624, in getMemberInfoWhitecube
    login_response = self._oauth_manager.login()
  File "F:\PixivUtil2-master\PixivOAuth.py", line 98, in login
    oauth_response = requests.post(self._url,
  File "C:\Program Files\Python\Python38\lib\site-packages\requests\api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "C:\Program Files\Python\Python38\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Program Files\Python\Python38\lib\site-packages\requests\sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Program Files\Python\Python38\lib\site-packages\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "C:\Program Files\Python\Python38\lib\site-packages\requests\adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: SOCKSHTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x000002657E2F1DF0>: Failed to establish a new connection: illegal IP address string passed to inet_pton'))
press enter to exit.

I tried to print out the args and return value of getaddrinfo, and here's what I got:

args: ('oauth.secure.pixiv.net', 443, <AddressFamily.AF_UNSPEC: 0>, <SocketKind.SOCK_STREAM: 1>, 6, <AddressInfo.AI_ADDRCONFIG: 1024>)

current version:return value: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('oauth.secure.pixiv.net', 443))]

previous version (without the overridden getaddrinfo) with SimpleDNS on return value: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('210.140.131.223', 443)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('210.140.131.219', 443))]

I will try to work this arround....

[bluerthanever]

I tried to change getaddrinfo to this, and it seem to be working for now. but it just doesn't feel so right...

orig_getaddrinfo = socket.getaddrinfo
def getaddrinfo(*args):
    try:
        return orig_getaddrinfo(*args)
    except socket.gaierror:
        return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]

[Nandaka]

can you try on different socks5 servers? I don't use proxy 😛

[bluerthanever]

I just tried several servers, doesn't look like on the server side

[lybtongji]

I got the same error.

[lybtongji]

I tried to change getaddrinfo to this, and it seem to be working for now. but it just doesn't feel so right...

orig_getaddrinfo = socket.getaddrinfo
def getaddrinfo(*args):
    try:
        return orig_getaddrinfo(*args)
    except socket.gaierror:
        return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]

It's working for me too.

[bluerthanever]

Thanks a lot... Will probably look into getaddrinfo sometime

[bluerthanever]

Sorry to bother again but some other errors happened again... When I try to use option 4, to download from list from PIXIV main site, after processing some hundred of creators, it raises errors like this:

2020-07-21 17:08:08,542 - PixivUtil20200712 - ERROR - Error at open_with_retry(): (<class 'RecursionError'>, RecursionError('maximum recursion depth exceeded'), <traceback object at 0x00000125E410BC40>)
2020-07-21 17:08:08,546 - PixivUtil20200712 - ERROR - Traceback (most recent call last):
  File "F:\PixivUtil2-master\PixivBrowserFactory.py", line 192, in open_with_retry
    return self.open(url, data, timeout)
  File "C:\Program Files\Python\Python38\lib\site-packages\mechanize\_mechanize.py", line 257, in open
    return self._mech_open(url_or_request, data, timeout=timeout)
  File "C:\Program Files\Python\Python38\lib\site-packages\mechanize\_mechanize.py", line 287, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "C:\Program Files\Python\Python38\lib\site-packages\mechanize\_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "C:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 425, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 414, in _call_chain
    result = func(*args)
  File "C:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 1276, in https_open
    return self.do_open(conn_factory, req)
  File "C:\Program Files\Python\Python38\lib\site-packages\mechanize\_urllib2_fork.py", line 1229, in do_open
    h.request(str(req.get_method()), str(req.get_selector()), req.data,
  File "C:\Program Files\Python\Python38\lib\http\client.py", line 1230, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Program Files\Python\Python38\lib\http\client.py", line 1276, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Program Files\Python\Python38\lib\http\client.py", line 1225, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Program Files\Python\Python38\lib\http\client.py", line 1004, in _send_output
    self.send(msg)
  File "C:\Program Files\Python\Python38\lib\http\client.py", line 944, in send
    self.connect()
  File "C:\Program Files\Python\Python38\lib\http\client.py", line 1392, in connect
    super().connect()
  File "C:\Program Files\Python\Python38\lib\http\client.py", line 915, in connect
    self.sock = self._create_connection(
  File "C:\Program Files\Python\Python38\lib\socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "F:\PixivUtil2-master\PixivBrowserFactory.py", line 127, in getaddrinfo
    return orig_getaddrinfo(*args)
  File "F:\PixivUtil2-master\PixivBrowserFactory.py", line 127, in getaddrinfo
    return orig_getaddrinfo(*args)
  File "F:\PixivUtil2-master\PixivBrowserFactory.py", line 127, in getaddrinfo
    return orig_getaddrinfo(*args)
  [Previous line repeated 972 more times]
RecursionError: maximum recursion depth exceeded

I didn't use loop structure in the new getaddrinfo, why would it act like this? Weird.