Open ewebgh33 opened 10 months ago
OK I can see now that it's different version to that listed in pyproject.toml.
Then again, instructions on main page of this repo didn't say anything about needing a poetry install. So.
pip install poetry
poetry install
a couple of errors here
Cannot install pytest-sugar.
• Installing pytest-watch (4.2.0): Failed
FileNotFoundError
[Errno 2] No such file or directory: 'C:\\Users\\HESPEROS\\miniconda3\\envs\\nitterscraper\\lib\\site-packages\\virtualenv\\activation\\nushell\\activate.nu'
at <frozen importlib._bootstrap_external>:1048 in open_resource
Cannot install pytest-watch.
Still can't find docker. So maybe I need to start over and set up the env with 3.7, not 3.8? And then pip install docker, pip install poetry, poetry install.... ?
I don't know what am I missing here.
Progress
Deleted the env, and re-created it with python=3.7.
Had to manually run pip install markupsafe==2.0.1
because that was the next error I came up against.
Now a new error. It looked like it started to spin up docker
2024-01-19 19:27:11.878 | INFO | nitter_scraper.nitter:_get_client:31 - Docker connection successful.
2024-01-19 19:27:23.199 | INFO | nitter_scraper.nitter:start:155 - Running container inspiring_moore c5edb4e9e2.
2024-01-19 19:27:23.203 | INFO | nitter_scraper.nitter:stop:159 - Stopping container inspiring_moore c5edb4e9e2.
2024-01-19 19:27:28.597 | INFO | nitter_scraper.nitter:stop:162 - Container inspiring_moore c5edb4e9e2 Destroyed.
Now I get these errors
Traceback (most recent call last):
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\connection.py", line 175, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
raise err
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
sock.connect(sa)
OSError: [WinError 10049] The requested address is not valid in its context
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\connectionpool.py", line 722, in urlopen
chunked=chunked,
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\http\client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\http\client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\http\client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\http\client.py", line 1036, in _send_output
self.send(msg)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\http\client.py", line 976, in send
self.connect()
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\connection.py", line 205, in connect
conn = self._new_conn()
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\connection.py", line 187, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000201BC7A7788>: Failed to establish a new connection: [WinError 10049] The requested address is not valid in its context
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\requests\adapters.py", line 497, in send
chunked=chunked,
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\connectionpool.py", line 800, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='0.0.0.0', port=8008): Max retries exceeded with url: /dgnsrekt (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000201BC7A7788>: Failed to establish a new connection: [WinError 10049] The requested address is not valid in its context'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 6, in <module>
profile = nitter.get_profile("dgnsrekt")
File "C:\Scraping\nitter_scraper\nitter_scraper\nitter.py", line 114, in get_profile
return get_profile(username=username, not_found_ok=not_found_ok, address=self.address)
File "C:\Scraping\nitter_scraper\nitter_scraper\profile.py", line 202, in get_profile
response = session.get(url)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\requests\sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\requests\sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\requests\sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "C:\Users\COMPUTERFACE\miniconda3\envs\nitterscraper\lib\site-packages\requests\adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=8008): Max retries exceeded with url: /dgnsrekt (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000201BC7A7788>: Failed to establish a new connection: [WinError 10049] The requested address is not valid in its context'))
So my conclusion is this thing is just old and doesn't work any more. Is this true? Does anyone have this running, like today?
I got to the same place of you with a slightly different error:
2024-01-21 15:11:01.821 | INFO | nitter_scraper.nitter:_get_client:31 - Docker connection successful.
2024-01-21 15:11:03.146 | INFO | nitter_scraper.nitter:start:155 - Running container inspiring_shirley 4b5453b0c4.
2024-01-21 15:11:03.377 | INFO | nitter_scraper.nitter:stop:159 - Stopping container inspiring_shirley 4b5453b0c4.
2024-01-21 15:11:08.661 | INFO | nitter_scraper.nitter:stop:162 - Container inspiring_shirley 4b5453b0c4 Destroyed.
Traceback (most recent call last):
File "/home/esteban/dev/python/nitter/test.py", line 6, in <module>
profile = nitter.get_profile("dgnsrekt")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/esteban/dev/python/nitter/venv/lib/python3.11/site-packages/nitter_scraper/nitter.py", line 114, in get_profile
return get_profile(username=username, not_found_ok=not_found_ok, address=self.address)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/esteban/dev/python/nitter/venv/lib/python3.11/site-packages/nitter_scraper/profile.py", line 207, in get_profile
raise ValueError(f'Oops! Either "{username}" does not exist or is private.')
ValueError: Oops! Either "dgnsrekt" does not exist or is private.
do you make some progress ?
@estebanlm I gave up on Nitter as I found a few places around the web to get sets of free monthly API calls. Search google for API sites and see a) what gets you a free account (not all will offer that) and b) what services they connect to. Some will do twitter and give a few hundred API calls a month. This is more than enough for my purposes, and I don't need to muck around with a Nitter instance or docker any more - I just call with Python and Requests
:( sadly I can not do that, I will continue digging and see if I can make a PR (highly unlikely, I am very new to python ;)
Hi Tried to set this up today on Win11. So far everything seems to have installed. I already have Docker Desktop.
However I'm hitting a wall trying to run a simple scrape (the example from github on the main page). Running the .py, I get
Which is weird because I've literally just run pip install docker, as well as already having desktop.
Any tips here appreciated. Version mismatch? I ran into an earlier issue where I couldn't pip install nitter_scraper because it didn't like python=3.11. Am now running 3.8. But perhaps it also wants an older docker?
pip list shows: