ConnectionError: HTTPSConnectionPool(host='bitcointalk.org', port=443): Max retries exceeded with url: /index.php?action=profile;u=1015588 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff4454ad780>: Failed to esta

[shawnlauzon]

View details in Rollbar: https://rollbar.com/vlabs/Venue/items/41/

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 375, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 632, in __protected_call__
    return self.run(*args, **kwargs)
  File "/code/venue/tasks.py", line 101, in scrape_forum_profile
    test_config=test_scrape_config)
  File "/code/venue/scrapers/bitcointalk.py", line 272, in verify_and_scrape
    test_config=test_config
  File "/code/venue/scrapers/bitcointalk.py", line 75, in get_profile
    self.make_request(url=profile_url, verify=False)
  File "/code/venue/scrapers/bitcointalk.py", line 55, in make_request
    verify=verify
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='bitcointalk.org', port=443): Max retries exceeded with url: /index.php?action=profile;u=1015588 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff4454ad780>: Failed to establish a new connection: [Errno -3] Try again',))Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
MaxRetryError: HTTPSConnectionPool(host='bitcointalk.org', port=443): Max retries exceeded with url: /index.php?action=profile;u=1015588 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff4454ad780>: Failed to establish a new connection: [Errno -3] Try again',))Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff4454ad780>: Failed to establish a new connection: [Errno -3] Try againTraceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
gaierror: [Errno -3] Try again

[shawnlauzon]

The same issue occurs with check/profile on https://venue-dev.volentix.io

[4ban]

@shawnlauzon Hi, I've started to look at the project and met some issues. I've never worked with docker before and I don't know how to debug the python code inside it.

I'm able to start backend via docker-compose up and play with API via /docs or via a local client. But I can't see any changes in my console that I'm doing in the python code.

I believe that I have to somehow connect to the python app (its terminal) inside a docker container via ssh, but I was unsuccessful without a manual.

I found the ports inside Dockerfile (5555, 8000, 9000) and probably I'm trying to use the wrong host (0.0.0.0), I'm not able to connect to it in any way vis command like ssh root@0.0.0.0 -p 8000 (I found this solution on StackOverflow).

Then I've tried some commands from official documentation:

• docker exec -it venue-server_web_1 /usr/bin/bash returns error: OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown
• docker attach venue-server_web_1 but it just stuck with no any output or response.

What I want: I want to add debug messages (print() ) into the code and see them in the terminal. I want to see logs (I guess the app has it) I want to create breakpoints (maybe there is a manual how to debug this app in vscode?)

P.S. LogDNA - I'm not sure how it works. I see the states for different sources, especially for, venue-deployment-68b97bf7c6-lngv5 (I think that this is for the current venue-dev state), but I don't know how to add my local backend as a source. Also, is this all information that is provided by these logs or I'm doing something wrong?

scrapinghub.com - I have been trying to verify some ID's from BTT from 4 pm, but I don't see my attempts in the logs. The attempts with IDs (12345678, 4kkkkk and etc) weren't my.

[shawnlauzon]

@joemarct Can you confirm if your fix resolves this problem?

And also could you provide a bit of guidance to @4ban on his questions?

[joemarct]

@shawnlauzon No, this is not resolved by my last PR, this is something else. This needs to be caught, logged, and re-raised as ScraperError so that it's handed over to the scraping retry mechanism we built.

[joemarct]

@4ban The command to get into the web container is docker-compose exec web sh. We are using Alpine Linux, it doesn't have bash installed by default. It does have sh so that's the one that is used.

Once you're inside, I guess you know your way around if you have good experience with Django/Python. We are using Supervisor to run multiple programs or processes. When you change anything in the code, you'll have to do supervisorctl restart all within the web container.

There is an option to re-create the containers altogether by doing docker-compose down and then just doing docker-compose up again. But this one also recreates the database, which is running on the postgres container.