search

scrapy-plugins / scrapy-playwright

🎭 Playwright integration for Scrapy
BSD 3-Clause "New" or "Revised" License
992 stars 108 forks source link

how to use playwright with SitemapSpider #249

Closed zaheerkzz closed 10 months ago

zaheerkzz commented 10 months ago

Im using sitemap to get urls for scrapping and with playwright i have issue integrating:

for entry in entries: for substr in testing_urls: if substr in entry["loc"]: yield scrapy.Request(substr, callback=self.parse_item, meta={"playwright": True})

giving me following error: yield d["loc"] TypeError: 'Request' object is not subscriptable

elacuesta commented 10 months ago

I guess your d object is a Request but you're treating it as a dict; hard to know without more context. Closing, this looks like a general Python question, not specific to this package.