search

khpeek / funda-scraper

Scraper of the Dutch real estate website www.funda.nl, implemented in Python with Scrapy
186 stars 85 forks source link

AttributeError: 'FundaSpider' object has no attribute '_rules' #5

Open jeroenvermunt opened 2 years ago

jeroenvermunt commented 2 years ago

When running the code after forking, I immediately got the following error:

AttributeError: 'FundaSpider' object has no attribute '_rules'

I solved it by adding the following line to init():

super(FundaSoldSpider, self).__init__(place=place)

The error stopped appearing, but now the following lines show up during scraping, giving an empty json file in the end.

[scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.funda.nl/koop/verkocht/eindhoven/p259/> (referer: None)
[scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.funda.nl/koop/verkocht/eindhoven/p268/> from <GET http://www.funda.nl/koop/verkocht/eindhoven/p268/>

there are the following possibilities:

  1. the fix solved the error but broke the scraper
  2. the fix solved the error, but the scraper does not work on a different aspect.

I'm am not sure which one it is, anyone have any idea on how to get the scraper to work?

haydnjm commented 2 years ago

This fixes those problems + some others that prevent it from working, just in case you didn't figure it out yet! (only changes for the currently for-sale houses, not the sold ones)