search

scrapinghub / splash

Lightweight, scriptable browser as a service with an HTTP API
BSD 3-Clause "New" or "Revised" License
4.08k stars 514 forks source link

SplashRequest not able to handle fragments URL even after applying dont_filter and DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter' #1111

Open bob251 opened 3 years ago

bob251 commented 3 years ago

Suppose I am using "https://www.geesey-ferguson.com/obituaries/obituary-listings#496" this URL and as I kept on increasing end digit it is always redirecting to "https://www.geesey-ferguson.com/obituaries/obituary-listings" . Please suggest if I am doing anything wrong. Because I saw one github comments and it was writen use: SplashRequest to overcome this problem.
This is my request: yield SplashRequest( url=next_page_url, callback=self.parse, args={"wait":5}, dont_filter=True, endpoint='render.html', splash_headers={ 'Authorization': basic_auth_header(self.settings[], ''), }, ) Thank you

vryazanov commented 3 years ago

That's a wrong repository to ask this question. The right is https://github.com/scrapy-plugins/scrapy-splash