search

ademjemaa / fbcrawl

facebook crawler
2 stars 4 forks source link

Link spider traceback issue #2

Open ademjemaa opened 5 years ago

ademjemaa commented 5 years ago

I keep running into one single error, it does not affect the spider's output or input but its obnoxious to see and i cant pinpoint the exact source of the problem

Traceback (most recent call last): File "c:\programdata\anaconda3\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback yield next(it) File c:\programdata\anaconda3\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output for x in result: File "c:\programdata\anaconda3\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr> return (_set_referer(r) for r in result or ()) File "c:\programdata\anaconda3\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr> return (r for r in result or () if _filter(r)) File "c:\programdata\anaconda3\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr> return (r for r in result or () if _filter(r)) File "C:\Users\adam\OneDrive\Documents\fbcrawl-master\fbcrawl\fbcrawl\spiders\Link.py", line 39, in parse_page temp_post = response.urljoin(post[0]) IndexError: list index out of range

if anyone knows why it happens or how to fix it let me know below

surafelkifetew1 commented 3 years ago

if you solve the above problem, please what was the solution you used?