dataabc / weiboSpider

新浪微博爬虫,用python爬取新浪微博数据
8.37k stars 1.98k forks source link

用user_id_list就报错,在config中设置user id就一切正常 #393

Closed Mordiggian closed 2 years ago

Mordiggian commented 2 years ago

为了更好的解决问题,请认真回答下面的问题。等到问题解决,请及时关闭本issue。

答:github

答:是

答:是

答:否

答:

答:

list index out of range Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/parser/info_parser.py", line 39, in extract_user_info if self.selector.xpath( IndexError: list index out of range 'NoneType' object has no attribute 'id' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/parser/index_parser.py", line 36, in get_user self.user.id = user_id AttributeError: 'NoneType' object has no attribute 'id' None


'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' expected str, bytes or os.PathLike object, not NoneType Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/writer/csv_writer.py", line 25, in init with open(self.file_path, 'a', encoding='utf-8-sig', TypeError: expected str, bytes or os.PathLike object, not NoneType 'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' 'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' 'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' 'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' 'NoneType' object has no attribute 'dict' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 313, in get_one_user self.write_user(self.user) File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 137, in write_user writer.write_user(user) File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/writer/txt_writer.py", line 29, in write_user [v + ':' + str(self.user.dict[k]) for k, v in self.user_desc]) File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/writer/txt_writer.py", line 29, in [v + ':' + str(self.user.dict[k]) for k, v in self.user_desc]) AttributeError: 'NoneType' object has no attribute 'dict' (my-env) Ray@Mac  ~/desktop/weibospider  python3 -m weibo_spider --config_path="config.json" list index out of range Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/parser/info_parser.py", line 39, in extract_user_info if self.selector.xpath( IndexError: list index out of range 'NoneType' object has no attribute 'id' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/parser/index_parser.py", line 36, in get_user self.user.id = user_id AttributeError: 'NoneType' object has no attribute 'id' None


'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' expected str, bytes or os.PathLike object, not NoneType Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/writer/csv_writer.py", line 25, in init with open(self.file_path, 'a', encoding='utf-8-sig', TypeError: expected str, bytes or os.PathLike object, not NoneType 'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' 'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' 'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' 'NoneType' object has no attribute 'nickname' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 226, in _get_filepath dir_name = self.user.nickname AttributeError: 'NoneType' object has no attribute 'nickname' 'NoneType' object has no attribute 'dict' Traceback (most recent call last): File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 313, in get_one_user self.write_user(self.user) File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/spider.py", line 137, in write_user writer.write_user(user) File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/writer/txt_writer.py", line 29, in write_user [v + ':' + str(self.user.dict[k]) for k, v in self.user_desc]) File "/Users/Moordiig/.pyenv/versions/my-env/lib/python3.9/site-packages/weibo_spider/writer/txt_writer.py", line 29, in [v + ':' + str(self.user.dict[k]) for k, v in self.user_desc]) AttributeError: 'NoneType' object has no attribute 'dict'

dataabc commented 2 years ago

感谢反馈。这个问题看起来像常见问题的问题2。但是如果这样不管如何输入user_id都会报错。你用这两种方式在短时间测试过吗?按理说,如果第一种方式报错,第二种也会报错。

Mordiggian commented 2 years ago

是的,我之前一直在自己的vps上使用user_id_list.txt配合crontab有快一年,然后从九月份一直报NoneType类的错,之前一直没空看,然后今天我试了在vps上重新下载新的再试依然不行。这个是我用自己电脑试的,自己电脑上在user_id_list上写id就正常爬取,用txt就不行

dataabc commented 2 years ago

我测试了下,没问题的。是不是修改过代码或者同时安装了pypi版?

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

Closing as stale, please reopen if you'd like to work on this further.