API Couldn't be Updated : 'NoneType' object has no attribute 'group'. - Not able to scrape data from the package anymore API Update Error

bhavik-malhotra commented 9 months ago

Even If I skip the update API update option I still face the below error, any solutions for this @iSarabjitDhiman ?

bhavik-malhotra commented 9 months ago

config.UPDATE_API=False

bhavik-malhotra commented 9 months ago

Attaching the logfile for reference

2023-10-18 14:30:57,217 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 14:30:57,218 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x0000028486B65850>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:30:57,222 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 14:30:59,357 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:30:59,364 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 14:31:01,572 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:31:01,984 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x0000028486B65850>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:31:01,988 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 14:31:03,614 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 14:31:03,618 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 14:31:03,619 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 14:31:03,619 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 14:31:03,621 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x0000028486B65850>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:31:04,150 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:31:04,497 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:31:04,712 [ERROR] [Line No. 63] tweeterpy.api_util : _get_api_file_url :: Couldn't get the API Url.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 59, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
2023-10-18 14:31:04,717 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: 'NoneType' object has no attribute 'group' Couldn't get the latest API data.
2023-10-18 14:31:04,717 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 14:31:04,718 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 14:31:04,718 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
'NoneType' object has no attribute 'group'
2023-10-18 14:31:04,719 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 34, in __init__
    api_file_url = self._get_api_file_url(page_source)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 59, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
2023-10-18 14:33:40,535 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 14:33:40,538 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x0000019D96C846D0>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:33:40,551 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 14:33:42,897 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:33:42,902 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 14:33:45,005 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:33:45,453 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x0000019D96C846D0>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:33:45,459 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 14:33:47,400 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 14:33:47,405 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 14:33:47,407 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 14:33:47,407 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 14:33:47,407 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: Skipping API Updates. Couldn't get the latest API data.
2023-10-18 14:33:47,408 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 14:33:47,411 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 14:33:47,411 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
Skipping API Updates.
2023-10-18 14:33:47,413 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
Skipping API Updates.
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 32, in __init__
    raise Exception("Skipping API Updates.")
Exception: Skipping API Updates.
2023-10-18 14:37:26,398 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 14:37:26,398 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x000002C750C6A550>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:37:26,403 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 14:37:27,850 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:37:27,854 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 14:37:29,496 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:37:29,924 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x000002C750C6A550>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:37:29,928 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 14:37:31,662 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 14:37:31,665 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 14:37:31,666 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 14:37:31,666 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 14:37:31,666 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: Skipping API Updates. Couldn't get the latest API data.
2023-10-18 14:37:31,667 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 14:37:31,669 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 14:37:31,670 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
Skipping API Updates.
2023-10-18 14:37:31,671 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
Skipping API Updates.
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 32, in __init__
    raise Exception("Skipping API Updates.")
Exception: Skipping API Updates.
2023-10-18 14:49:03,928 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 14:49:03,929 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x0000023D93CBAF10>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:49:03,932 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 14:49:05,852 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:49:05,856 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 14:49:08,152 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:49:08,577 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x0000023D93CBAF10>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:49:08,581 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 14:49:10,315 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 14:49:10,318 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 14:49:10,319 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 14:49:10,319 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 14:49:10,319 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: Skipping API Updates. Couldn't get the latest API data.
2023-10-18 14:49:10,320 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 14:49:10,321 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 14:49:10,322 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
Skipping API Updates.
2023-10-18 14:49:10,322 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
Skipping API Updates.
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 32, in __init__
    raise Exception("Skipping API Updates.")
Exception: Skipping API Updates.

iSarabjitDhiman commented 9 months ago

Hi @bhavik-malhotra , If you take a closer look at the log, it says couldn't find the log file. Let me tell you what's happening and how the api updates work:

When config.UPDATE_API is set to True(which is by default) it updates the api(fetches latest api endpoints and stores/backup them in a temp directory). So as long as there is a backup file in the temp directory, you can skip Api updates by setting config.UPDATE_API to False and it will restore from the most recent saved/backup file available in the temp folder.

Fix: Set config.UPDATE_API to True for once, and it will create a backup file. Now you can skip the updates for the subsequent sessions. Edit: let me know if you still face any issues. Feel free to close the issue otherwise.

bhavik-malhotra commented 9 months ago

Hey @iSarabjitDhiman thanks for the quick response, the update API was set to true only by default, I ran into this issue today when i ran my script, it was working fine yesterday, this is the error when default UPDATE_API=True

bhavik-malhotra commented 9 months ago

This is the log file with default settings

2023-10-18 14:30:57,217 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 14:30:57,218 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x0000028486B65850>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:30:57,222 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 14:30:59,357 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:30:59,364 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 14:31:01,572 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:31:01,984 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x0000028486B65850>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:31:01,988 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 14:31:03,614 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 14:31:03,618 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 14:31:03,619 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 14:31:03,619 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 14:31:03,621 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x0000028486B65850>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:31:04,150 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:31:04,497 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:31:04,712 [ERROR] [Line No. 63] tweeterpy.api_util : _get_api_file_url :: Couldn't get the API Url.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 59, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
2023-10-18 14:31:04,717 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: 'NoneType' object has no attribute 'group' Couldn't get the latest API data.
2023-10-18 14:31:04,717 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 14:31:04,718 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 14:31:04,718 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
'NoneType' object has no attribute 'group'
2023-10-18 14:31:04,719 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 34, in __init__
    api_file_url = self._get_api_file_url(page_source)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 59, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
2023-10-18 14:33:40,535 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 14:33:40,538 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x0000019D96C846D0>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:33:40,551 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 14:33:42,897 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:33:42,902 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 14:33:45,005 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:33:45,453 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x0000019D96C846D0>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:33:45,459 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 14:33:47,400 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 14:33:47,405 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 14:33:47,407 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 14:33:47,407 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 14:33:47,407 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: Skipping API Updates. Couldn't get the latest API data.
2023-10-18 14:33:47,408 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 14:33:47,411 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 14:33:47,411 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
Skipping API Updates.
2023-10-18 14:33:47,413 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
Skipping API Updates.
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 32, in __init__
    raise Exception("Skipping API Updates.")
Exception: Skipping API Updates.
2023-10-18 14:37:26,398 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 14:37:26,398 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x000002C750C6A550>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:37:26,403 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 14:37:27,850 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:37:27,854 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 14:37:29,496 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:37:29,924 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x000002C750C6A550>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:37:29,928 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 14:37:31,662 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 14:37:31,665 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 14:37:31,666 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 14:37:31,666 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 14:37:31,666 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: Skipping API Updates. Couldn't get the latest API data.
2023-10-18 14:37:31,667 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 14:37:31,669 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 14:37:31,670 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
Skipping API Updates.
2023-10-18 14:37:31,671 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
Skipping API Updates.
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 32, in __init__
    raise Exception("Skipping API Updates.")
Exception: Skipping API Updates.
2023-10-18 14:49:03,928 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 14:49:03,929 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x0000023D93CBAF10>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:49:03,932 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 14:49:05,852 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 14:49:05,856 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 14:49:08,152 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 14:49:08,577 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x0000023D93CBAF10>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 14:49:08,581 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 14:49:10,315 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 14:49:10,318 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 14:49:10,319 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 14:49:10,319 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 14:49:10,319 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: Skipping API Updates. Couldn't get the latest API data.
2023-10-18 14:49:10,320 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 14:49:10,321 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 14:49:10,322 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
Skipping API Updates.
2023-10-18 14:49:10,322 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
Skipping API Updates.
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 32, in __init__
    raise Exception("Skipping API Updates.")
Exception: Skipping API Updates.
2023-10-18 15:11:53,360 [DEBUG] [Line No. 145] tweeterpy.tweeterpy : generate_session :: Trying to generate a new session.
2023-10-18 15:11:53,361 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x000001D8C5EF6DD0>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 15:11:53,364 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): www.twitter.com:443
2023-10-18 15:11:55,036 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 15:11:55,038 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): twitter.com:443
2023-10-18 15:11:56,942 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 15:11:57,378 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://api.twitter.com/1.1/guest/activate.json', 'session': <requests.sessions.Session object at 0x000001D8C5EF6DD0>, 'method': 'POST', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 15:11:57,381 [DEBUG] [Line No. 1048] urllib3.connectionpool : _new_conn :: Starting new HTTPS connection (1): api.twitter.com:443
2023-10-18 15:11:59,096 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2023-10-18 15:11:59,100 [DEBUG] [Line No. 437] charset_normalizer : from_bytes :: Encoding detection: ascii is most likely the one.
2023-10-18 15:11:59,102 [DEBUG] [Line No. 162] tweeterpy.tweeterpy : generate_session :: Session has been generated.
2023-10-18 15:11:59,102 [DEBUG] [Line No. 28] tweeterpy.api_util : __init__ :: Updating API...
2023-10-18 15:11:59,103 [DEBUG] [Line No. 20] tweeterpy.request_util : make_request :: {'url': 'https://www.twitter.com/', 'session': <requests.sessions.Session object at 0x000001D8C5EF6DD0>, 'method': 'GET', 'max_retries': 3, 'timeout': 5, 'skip_error_checking': False, 'kwargs': {}}
2023-10-18 15:11:59,682 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://www.twitter.com:443 "GET / HTTP/1.1" 301 0
2023-10-18 15:12:00,033 [DEBUG] [Line No. 546] urllib3.connectionpool : _make_request :: https://twitter.com:443 "GET / HTTP/1.1" 200 None
2023-10-18 15:12:00,268 [ERROR] [Line No. 63] tweeterpy.api_util : _get_api_file_url :: Couldn't get the API Url.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 59, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
2023-10-18 15:12:00,271 [WARNING] [Line No. 39] tweeterpy.api_util : __init__ :: 'NoneType' object has no attribute 'group' Couldn't get the latest API data.
2023-10-18 15:12:00,271 [DEBUG] [Line No. 40] tweeterpy.api_util : __init__ :: Trying to restore API data from the backup file.
2023-10-18 15:12:00,273 [WARNING] [Line No. 142] tweeterpy.api_util : _load_api_data :: Couldn't find the API backup file.
2023-10-18 15:12:00,273 [WARNING] [Line No. 151] tweeterpy.api_util : _load_api_data :: Couldn't restore API data from the backup file.
'NoneType' object has no attribute 'group'
2023-10-18 15:12:00,273 [ERROR] [Line No. 48] tweeterpy.api_util : __init__ :: API Couldn't be Updated.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 41, in __init__
    feature_switches, api_endpoints_data = self._load_api_data()
                                           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 34, in __init__
    api_file_url = self._get_api_file_url(page_source)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tweeterpy\api_util.py", line 59, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'

iSarabjitDhiman commented 9 months ago

Hey @iSarabjitDhiman thanks for the quick response, the update API was set to true only by default, I ran into this issue today when i ran my script, it was working fine yesterday, this is the error when default UPDATE_API=True

Oh I see.

Reason :

You have been scraping for a long time, they might have temporarily blocked your IP address, because it says "couldn't get the api file url" which means It couldn't load the twitter page.

Fix: Try after some time.

Try on a different machine with a different IP address.
Try using proxies or just connect your machine to a different network for testing.

Let me know how it goes.

Edit: I just checked it on my end, you're right. There's some issue with the script, twitter might have changed something. I will fix it soon. Thanks for the update. I will let u know once it's fixed.

bhavik-malhotra commented 9 months ago

Cool thanks:) let me know once you update the script I'll check and close the issue:)

iSarabjitDhiman commented 9 months ago

@bhavik-malhotra Just fixed it. Please test and let me know. Thanks.

bhavik-malhotra commented 9 months ago

Works great now thanks for the quick fix:) @iSarabjitDhiman

Glacier-Luo commented 9 months ago

@iSarabjitDhiman Hello bro, this problem seems still exists. i am using tweeterpy 1.0.10

2023-10-21 17:12:58,242 [ERROR] :: Couldn't get the API file Url.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "/Users/glacierluo/.local/share/virtualenvs/twitterAPI-0telnOEO/lib/python3.9/site-packages/tweeterpy/api_util.py", line 68, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
AttributeError: 'NoneType' object has no attribute 'group'
^C2023-10-21 17:13:00,067 [WARNING] :: Keyboard Interruption...
2023-10-21 17:13:00,068 [ERROR] :: feature_switches or endpoints_data is invalid.
NoneType: None
2023-10-21 17:13:00,068 [WARNING] :: Couldn't save API data.
Invalid API Data.
2023-10-21 17:13:00,068 [INFO] :: API Updated Successfully.

iSarabjitDhiman commented 9 months ago

@iSarabjitDhiman Hello bro, this problem seems still exists. i am using tweeterpy 1.0.10

2023-10-21 17:12:58,242 [ERROR] :: Couldn't get the API file Url.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "/Users/glacierluo/.local/share/virtualenvs/twitterAPI-0telnOEO/lib/python3.9/site-packages/tweeterpy/api_util.py", line 68, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
AttributeError: 'NoneType' object has no attribute 'group'
^C2023-10-21 17:13:00,067 [WARNING] :: Keyboard Interruption...
2023-10-21 17:13:00,068 [ERROR] :: feature_switches or endpoints_data is invalid.
NoneType: None
2023-10-21 17:13:00,068 [WARNING] :: Couldn't save API data.
Invalid API Data.
2023-10-21 17:13:00,068 [INFO] :: API Updated Successfully.

Hey @Glacier-Luo It's working I guess. I mean you will still see that error message but don't interrupt it by pressing ctrl+C. Let it finish. It will fail for the first time and will succeed the 2nd time. The first one is the original api update which is now failing, but the 2nd one is the one I added as a backup in case the first fails. I cannot remove the first one at the moment, because we are not sure if twitter removed it for good or its a temporary issue. The reason it failed to save the api data and showed that "feature_switches or endpoints_data is invalid" is because you interrupted it manually after the first attempt.

If you are not sure what to do, just let it finish the updates.

Let me know if this helps.

Glacier-Luo commented 9 months ago

@iSarabjitDhiman Hello bro, this problem seems still exists. i am using tweeterpy 1.0.10
2023-10-21 17:12:58,242 [ERROR] :: Couldn't get the API file Url.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "/Users/glacierluo/.local/share/virtualenvs/twitterAPI-0telnOEO/lib/python3.9/site-packages/tweeterpy/api_util.py", line 68, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
AttributeError: 'NoneType' object has no attribute 'group'
^C2023-10-21 17:13:00,067 [WARNING] :: Keyboard Interruption...
2023-10-21 17:13:00,068 [ERROR] :: feature_switches or endpoints_data is invalid.
NoneType: None
2023-10-21 17:13:00,068 [WARNING] :: Couldn't save API data.
Invalid API Data.
2023-10-21 17:13:00,068 [INFO] :: API Updated Successfully.
Hey @Glacier-Luo It's working I guess. I mean you will still see that error message but don't interrupt it by pressing ctrl+C. Let it finish. It will fail for the first time and will succeed the 2nd time. The first one is the original api update which is now failing, but the 2nd one is the one I added as a backup in case the first fails. I cannot remove the first one at the moment, because we are not sure if twitter removed it for good or its a temporary issue. The reason it failed to save the api data and showed that "feature_switches or endpoints_data is invalid" is because you interrupted it manually after the first attempt.

If you are not sure what to do, just let it finish the updates.

Let me know if this helps.

Oh! Thank you @iSarabjitDhiman . I understand now. I tested it again.

2023-10-21 18:04:52,280 [ERROR] :: Couldn't get the API file Url.
'NoneType' object has no attribute 'group'
Traceback (most recent call last):
  File "/Users/glacierluo/.local/share/virtualenvs/twitterAPI-0telnOEO/lib/python3.9/site-packages/tweeterpy/api_util.py", line 68, in _get_api_file_url
    api_file_name = re.search(api_file_regex, page_source).group(1)
AttributeError: 'NoneType' object has no attribute 'group'
2023-10-21 18:04:53,434 [INFO] :: API Updated Successfully.

That's ok.

iSarabjitDhiman / TweeterPy

API Couldn't be Updated : 'NoneType' object has no attribute 'group'. - Not able to scrape data from the package anymore API Update Error #37