howie6879 / ruia

Async Python 3.6+ web scraping micro-framework based on asyncio
https://www.howie6879.com/ruia/
Apache License 2.0
1.75k stars 181 forks source link

示例代码运行报错 #149

Closed mogeko closed 2 years ago

mogeko commented 2 years ago

我在我的计算机中运行了一下快速开始.md 中的示例代码,但是它报错了

目测是依赖项 collections 的 bug?

我的环境:

报错信息:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspaces/myq-spider/myq_spider/__main__.py", line 30, in <module>
    HackerNewsSpider.start()
  File "/home/vscode/.local/lib/python3.10/site-packages/ruia/spider.py", line 364, in start
    spider_ins = cls(middleware=middleware, loop=loop, **spider_kwargs)
  File "/home/vscode/.local/lib/python3.10/site-packages/ruia/spider.py", line 156, in __init__
    if not self.start_urls or not isinstance(self.start_urls, collections.Iterable):
AttributeError: module 'collections' has no attribute 'Iterable'

代码:

import aiofiles

from ruia import AttrField, TextField, Item, Spider

class HackerNewsItem(Item):
    target_item = TextField(css_select='tr.athing')
    title = TextField(css_select='a.storylink')
    url = AttrField(css_select='a.storylink', attr='href')

    async def clean_title(self, value):
        return value

class HackerNewsSpider(Spider):
    start_urls = ['https://news.ycombinator.com/news?p=1',
                  'https://news.ycombinator.com/news?p=2']

    async def parse(self, response):
        async for item in HackerNewsItem.get_items(html=await response.text()):
            yield item

    async def process_item(self, item: HackerNewsItem):
        """Ruia build-in method"""
        async with aiofiles.open('./hacker_news.txt', 'a') as f:
            await f.write(str(item.title) + '\n')

if __name__ == '__main__':
    HackerNewsSpider.start()
howie6879 commented 2 years ago

@Mogeko 你好:

对于你的问题是因为Ruia暂时还没有对3.10做兼容,我稍后会修复,麻烦你这边先将Python环境更换为3.6~3.9。

mogeko commented 2 years ago

@Mogeko 你好:

对于你的问题是因为Ruia暂时还没有对3.10做兼容,我稍后会修复,麻烦你这边先将Python环境更换为3.6~3.9。

谢谢,已经解决了

howie6879 commented 2 years ago

谢谢,已经解决了

我会解决掉和3.10的兼容问题,到时候再关闭issue