Closed en-GB closed 2 years ago
no. scrapy allready filters duplicate requests automatically. but i do agree that this is ugly.
.getall()[-1]
would let me keep the if else structure. it would also cut the number of filtered requests down to just 1 on the final page. would that be acceptable?
To be honest, I'm not 100% sure how scrapy handles duplicated requests. We do request the same URLs multiple times (currently once a week), which obviously works and is important to us. I would prefer to avoid duplicate requests whenever possible and not rely on a feature I don't fully understand.
In this case it easy to omit the duplicated request at the end:
.getall()
returns a list of size one, we know it's either the first or the last pagep
querylen(... .getall()) == 1
and stop yielding requests when there is this p
query?thats fair. parsing the url isnt necessary tho. i think an explicit parameter is_first_page
is better.
alternatively i could parse the pagination bar properly.
Absolutely agree! That's much better than parsing the URL
as discussed in the sync. since then alex confirmed that this is a problem