Closed matthewcummings closed 8 months ago
I have updated the demo in https://github.com/scrapfly/python-scrapfly/tree/master/examples/scrapy/demo, is this still happening?
I'm curious how you get AttributeError: 'NoneType' object has no attribute 'set_value'
Am I missing a point where it's possible to disable this module https://docs.scrapy.org/en/latest/topics/stats.html ?
I did not explicitly disable the stats module, it appears to be missing for some reason. I will try your update and see if that makes a difference, thank you.
I just got into the same problem. This is a problem with a StatsCollector that was introduced in release 2.11.0
.
You can check in the diffs in scrapy/crawler.py
file.
They do not initialize the stats property anymore, they set to None by default.
Line 77 [removed] -> Line 85 [added]
Ok, I made changes on main branch, I tested against scrapy from 2.6 to 2.11 and should works as expected - will tag soon if no negative feedback
@jjsaunier I will try this out in the next day or two, thank you.
Same error for me with scrapfly sdk 0.8.10 and scrapy 2.11.0, "stats" attribute is missing in crawler object
@DJousto Can you try with main branch, it's not tagged yet
Sorry @jjsaunier can you detail what you mean by main branch ? main branch of "python-scrapfly" ? there is only one branch : "master"
yes I mean the "master" sorry, with github rename I was confused
pip install git+https://github.com/scrapfly/python-scrapfly.git@master
OK, following this I fall into another error
2023-11-20 16:04:54 [scrapfly] ERROR: [Failure instance: Traceback: <class 'TypeError'>: ExecutionEngine.crawl() takes 2 positional arguments but 3 were given
F:\python310\lib\site-packages\twisted\internet\defer.py:661:callback
F:\python310\lib\site-packages\twisted\internet\defer.py:763:_startRunCallbacks
F:\python310\lib\site-packages\twisted\internet\defer.py:857:_runCallbacks
F:\python310\lib\site-packages\twisted\internet\defer.py:1750:gotResult
---
When I run the Scrapy examples like the Bea one, I get the following error:
I'm using Python 3.11 with: scrapfly-sdk==0.8.9 Scrapy==2.11.0