search

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://scrapy.org
BSD 3-Clause "New" or "Revised" License
51.16k stars 10.35k forks source link

Twisted and asyncio #6219

Closed abebus closed 3 weeks ago

abebus commented 3 months ago

Do you guys have/had thoughts for migrating to native asyncio? What are your thoughts about Twisted? Isn’t it feels deprecated?

Gallaecio commented 3 months ago

Twisted works well and is actively developed.

Given that, and the fact that you can use asyncio code with Scrapy, I am not sure there is a point to migrating away from Twisted.

abebus commented 3 months ago

Yes, but it feels somewhat like a crutch, because conversion of futures to defers sometimes can be unpredictable. I've had cases where scrapy engine closes all of its resources before some asyncio tasks were finished. Soon I will try reproduce this behaviour and provide it as a case in a new issue.

wRAR commented 3 months ago

I had a thought about replacing more of the native Deferred code and especially inlineCallbacks code with async def functions, as is recommended by Twisted itself, but rewriting the Scrapy core to remove Twisted completely sounds very hard.

dream2333 commented 2 months ago

Migrating Scrapy from Twisted to asyncio was not an easy task. It requires a lot of work, including rewriting most of the code, and may also introduce new bugs. in addition, there are some functional differences between Twisted and asyncio, which may result in some features not being implemented on asyncio

shraddhakannurkar commented 2 months ago

The perception of Twisted as being deprecated might come from the fact that asyncio is now part of the Python standard library and has gained a lot of traction, leading to a preference for asyncio in new projects. However, Twisted is still actively maintained and has a loyal user base. It's also worth noting that Twisted has been working on integrating with asyncio through the twisted.internet.asyncioreactor module, which allows Twisted to run on top of asyncio's event loop, giving developers the best of both worlds in some cases.

dream2333 commented 2 months ago

The perception of Twisted as being deprecated might come from the fact that asyncio is now part of the Python standard library and has gained a lot of traction, leading to a preference for asyncio in new projects. However, Twisted is still actively maintained and has a loyal user base. It's also worth noting that Twisted has been working on integrating with asyncio through the twisted.internet.asyncioreactor module, which allows Twisted to run on top of asyncio's event loop, giving developers the best of both worlds in some cases.

Yes, maybe in the near future we'll be able to migrate more parts to asyncio, as we did from python2 to python3

wRAR commented 2 months ago

Yes, maybe in the near future we'll be able to migrate more parts to asyncio, as we did from python2 to python3

I don't think it's possible to "migrate more parts to asyncio" except for what I've already described, but maybe you have some ideas?