fix zalando pagination

[en-GB]

as discussed in the sync. since then alex confirmed that this is a problem

[en-GB]

no. scrapy allready filters duplicate requests automatically. but i do agree that this is ugly. .getall()[-1] would let me keep the if else structure. it would also cut the number of filtered requests down to just 1 on the final page. would that be acceptable?

[se-jaeger]

To be honest, I'm not 100% sure how scrapy handles duplicated requests. We do request the same URLs multiple times (currently once a week), which obviously works and is important to us. I would prefer to avoid duplicate requests whenever possible and not rely on a feature I don't fully understand.

In this case it easy to omit the duplicated request at the end:

• If .getall() returns a list of size one, we know it's either the first or the last page
• The following URLs are correspond to an example:
  • https://en.zalando.de/mens-clothing-shirts/?cause=animal_welfare.fair_conditions.reducing_emissions.reusing_materials.sustainable_beauty.water_saving
  • https://en.zalando.de/mens-clothing-shirts/?cause=animal_welfare.fair_conditions.reducing_emissions.reusing_materials.sustainable_beauty.water_saving&p=16
• The difference is just the p query
• What do you think about parsing the URL if len(... .getall()) == 1 and stop yielding requests when there is this p query?

[en-GB]

thats fair. parsing the url isnt necessary tho. i think an explicit parameter is_first_page is better. alternatively i could parse the pagination bar properly.

[se-jaeger]

Absolutely agree! That's much better than parsing the URL