ConnectionError: HTTPSConnectionPool(host='bitcointalk.org', port=443): Max retries exceeded with url: /index.php?action=profile;u=689228;sa=showPosts;start=0 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5e1dbc2

[shawnlauzon]

View details in Rollbar: https://rollbar.com/vlabs/Venue/items/43/

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 375, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 632, in __protected_call__
    return self.run(*args, **kwargs)
  File "/code/venue/tasks.py", line 166, in scrape_forum_profile
    start=posts_scrape_start.replace(tzinfo=None)
  File "/code/venue/scrapers/bitcointalk.py", line 343, in scrape_posts
    posts = scraper.scrape_posts(forum_user_id, **kwargs)
  File "/code/venue/scrapers/bitcointalk.py", line 249, in scrape_posts
    resp = requests.get(url)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='bitcointalk.org', port=443): Max retries exceeded with url: /index.php?action=profile;u=689228;sa=showPosts;start=0 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5e1dbc2198>: Failed to establish a new connection: [Errno 101] Network unreachable',))Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
MaxRetryError: HTTPSConnectionPool(host='bitcointalk.org', port=443): Max retries exceeded with url: /index.php?action=profile;u=689228;sa=showPosts;start=0 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5e1dbc2198>: Failed to establish a new connection: [Errno 101] Network unreachable',))Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f5e1dbc2198>: Failed to establish a new connection: [Errno 101] Network unreachableTraceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network unreachable

[shawnlauzon]

I see no error in logdna, which is annoying and difficult to debug.

Also, it seems to be only this user. That seems odd.

As @realrhys points out, we're only seeing this in Dev, but we should still understand what is going on, if there's a code improvement possible.

[joemarct]

That's not difficult to debug: OSError: [Errno 101] Network unreachable is pretty clear to me this is about bitcointalk.org not being reachable at the time that particular request was sent.

I agree though that more concise log of which part of the scraping actually failed should have been logged in LogDNA. I'll work on that.