search

aio-libs / aiohttp-demos

Demos for aiohttp project
https://demos.aiohttp.org
Other
716 stars 219 forks source link

Implement demo of idiomatic crawler using aiohttp client #45

Open jettify opened 5 years ago

jettify commented 5 years ago

Lots of people using aiohttp client to crawl internet [1], I think to encourage good practices and idiomatic approach it is good idea to have specific demo for this purposes. Good starting point is [2]

[1] https://medium.com/@cgarciae/making-an-infinite-number-of-requests-with-python-aiohttp-pypeln-3a552b97dc95 [2] http://www.aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html

K0Te commented 5 years ago

I'd like to work on this issue, it seems to be an interesting task.

webknjaz commented 5 years ago

Go ahead :)

gyermolenko commented 5 years ago

hey @K0Te No pressure, but is there any progress for this task?

K0Te commented 5 years ago

Hi @gyermolenko,

Sorry for delay - I'll try to submit PR soon.

gyermolenko commented 5 years ago

@K0Te :ok_hand: no hurry , just wanted to know issue status

Sab0tag3d commented 3 years ago

Did you see https://github.com/aio-libs/aiohttp/blob/master/examples/legacy/crawl.py ?