search

AdamGagorik / ffxiahbot

A python module for interacting with a darkstar server.
MIT License
22 stars 57 forks source link

I dont understand #22

Closed AlexandreCA closed 3 years ago

AlexandreCA commented 4 years ago

Hi! It's me or we have a problem?

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "E:\pydarkstar\pydarkstar\scrubbing\scrubber.py", line 34, in soup handle = urlopen(url) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 222, in urlopen return opener.open(url, data, timeout) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 531, in open response = meth(req, response) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 563, in error result = self._call_chain(args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 503, in _call_chain result = func(args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 755, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 525, in open response = self._open(req, data) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 543, in _open '_open', req) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 503, in _call_chain result = func(*args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1360, in https_open context=self._context, check_hostname=self._check_hostname) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1319, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)> [2020-01-25 21:05:53,534][ 2628][ERROR]: urlopen failed (attempt 9) Traceback (most recent call last): File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1317, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1244, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1290, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1239, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1026, in _send_output self.send(msg) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 966, in send self.connect() File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1414, in connect server_hostname=server_hostname) File "C:\Users\Server\Anaconda3\lib\ssl.py", line 423, in wrap_socket session=session File "C:\Users\Server\Anaconda3\lib\ssl.py", line 870, in _create self.do_handshake() File "C:\Users\Server\Anaconda3\lib\ssl.py", line 1139, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "E:\pydarkstar\pydarkstar\scrubbing\scrubber.py", line 34, in soup handle = urlopen(url) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 222, in urlopen return opener.open(url, data, timeout) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 531, in open response = meth(req, response) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 563, in error result = self._call_chain(args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 503, in _call_chain result = func(args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 755, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 525, in open response = self._open(req, data) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 543, in _open '_open', req) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 503, in _call_chain result = func(*args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1360, in https_open context=self._context, check_hostname=self._check_hostname) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1319, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)> [2020-01-25 21:05:54,672][ 2628][ERROR]: urlopen failed (attempt 10) Traceback (most recent call last): File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1317, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1244, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1290, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1239, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1026, in _send_output self.send(msg) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 966, in send self.connect() File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1414, in connect server_hostname=server_hostname) File "C:\Users\Server\Anaconda3\lib\ssl.py", line 423, in wrap_socket session=session File "C:\Users\Server\Anaconda3\lib\ssl.py", line 870, in _create self.do_handshake() File "C:\Users\Server\Anaconda3\lib\ssl.py", line 1139, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "E:\pydarkstar\pydarkstar\scrubbing\scrubber.py", line 34, in soup handle = urlopen(url) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 222, in urlopen return opener.open(url, data, timeout) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 531, in open response = meth(req, response) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 563, in error result = self._call_chain(args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 503, in _call_chain result = func(args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 755, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 525, in open response = self._open(req, data) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 543, in _open '_open', req) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 503, in _call_chain result = func(*args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1360, in https_open context=self._context, check_hostname=self._check_hostname) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1319, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)> [2020-01-25 21:05:54,766][ 2628][ERROR]: the maximum urlopen attempts have been reached [2020-01-25 21:05:54,766][ 2628][ERROR]: caught unhandled excetion Traceback (most recent call last): File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1317, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1244, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1290, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1239, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1026, in _send_output self.send(msg) File "C:\Users\Server\Anaconda3\lib\http\client.py", line 966, in send self.connect() File "C:\Users\Server\Anaconda3\lib\http\client.py", line 1414, in connect server_hostname=server_hostname) File "C:\Users\Server\Anaconda3\lib\ssl.py", line 423, in wrap_socket session=session File "C:\Users\Server\Anaconda3\lib\ssl.py", line 870, in _create self.do_handshake() File "C:\Users\Server\Anaconda3\lib\ssl.py", line 1139, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "E:\pydarkstar\pydarkstar\logutils.py", line 45, in capture yield File ".\scrub.py", line 21, in pydarkstar.apps.scrub.run.main() File "E:\pydarkstar\pydarkstar\apps\scrub\run.py", line 35, in main data = scrubber.scrub(force=True, threads=-1, urls=opts.urls, ids=opts.itemids) File "E:\pydarkstar\pydarkstar\scrubbing\ffxiah.py", line 63, in scrub urls = self._get_category_urls() File "E:\pydarkstar\pydarkstar\scrubbing\ffxiah.py", line 194, in _get_category_urls soup = self.soup(path) File "E:\pydarkstar\pydarkstar\scrubbing\scrubber.py", line 34, in soup handle = urlopen(url) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 222, in urlopen return opener.open(url, data, timeout) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 531, in open response = meth(req, response) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 563, in error result = self._call_chain(args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 503, in _call_chain result = func(args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 755, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 525, in open response = self._open(req, data) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 543, in _open '_open', req) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 503, in _call_chain result = func(*args) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1360, in https_open context=self._context, check_hostname=self._check_hostname) File "C:\Users\Server\Anaconda3\lib\urllib\request.py", line 1319, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)>

AdamGagorik commented 3 years ago

I just tried the scrub command on my end (its been some time) and I am seeing it working.

Screen Shot 2021-06-26 at 11 45 42 AM 
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: config     = config.yaml
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: verbose    = True
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: silent     = False
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: stub       = items
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: overwrite  = False
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: backup     = True
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: stock01    = 5
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: stock12    = 5
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: itemids    = ????
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee0088ca0) Options: urls       = ????
[2021-06-26 11:43:20][84529][DEBUG]: (0x7fdee02c9ee0) FFXIAHScrubber: init
[2021-06-26 11:43:20][84529][DEBUG]: (0x7fdee02c9ee0) FFXIAHScrubber: forcing redownload of data
[2021-06-26 11:43:20][84529][DEBUG]: (0x7fdee02c9ee0) FFXIAHScrubber: using passed ids
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee02c9ee0) FFXIAHScrubber: getting data
[2021-06-26 11:43:20][84529][INFO ]: (0x7fdee02c9ee0) FFXIAHScrubber: threads = -1
[2021-06-26 11:43:20][84529][DEBUG]: (0x7fdee02c9ee0) FFXIAHScrubber: open (000000/000003,  0.00) http://www.ffxiah.com/item/0
[2021-06-26 11:43:21][84529][DEBUG]: (0x7fdee02c9ee0) FFXIAHScrubber: open (000001/000003, 33.33) http://www.ffxiah.com/item/1
[2021-06-26 11:43:22][84529][DEBUG]: (0x7fdee02c9ee0) FFXIAHScrubber: open (000002/000003, 66.67) http://www.ffxiah.com/item/2
[2021-06-26 11:43:23][84529][DEBUG]: (0x7fdee02c9ee0) FFXIAHScrubber: item count = 3
[2021-06-26 11:43:23][84529][DEBUG]: (0x7fdee02c9ee0) FFXIAHScrubber: data count = 3
[2021-06-26 11:43:23][84529][DEBUG]: (0x7fdea0bebe50) ItemList: init
[2021-06-26 11:43:23][84529][DEBUG]: backup (old): /Users/adam/workspace/pydarkstar/bin/items.csv
[2021-06-26 11:43:23][84529][DEBUG]: backup (new): /Users/adam/workspace/pydarkstar/bin/items.csv.1
[2021-06-26 11:43:23][84529][INFO ]: (0x7fdea0bebe50) ItemList: overwriting file...
[2021-06-26 11:43:23][84529][INFO ]: (0x7fdea0bebe50) ItemList: save /Users/adam/workspace/pydarkstar/bin/items.csv
[2021-06-26 11:43:23][84529][INFO ]: exit

I did a search on the urllib.error.URLError and this post seems to suggest something funny happened when you installed some python modules (bs4 probably).

You might try removing your Python environment and installing stuff from scratch to see if the error goes away.

You might also try some of the recommendations in that Stack Overflow post. They suggest a few different ways to execute the following script, which I think is the root cause of your issue.

Install Certificates.command
AdamGagorik commented 3 years ago

I think this is an environment setup issue and not an issue with the scrubbing app itself. Given that I have no real way to reproduce the error, I will close this and consider adding a note the documentation.