search

ispras / scrapy-puppeteer

Library that helps use puppeteer in scrapy.
BSD 3-Clause "New" or "Revised" License
49 stars 4 forks source link

Custom puppeteer requests #25

Open kuwaitman opened 5 months ago

kuwaitman commented 5 months ago

Hi, is there a way i can custom the puppeteer request, for example to add puppeteer extra functionality?

thank you.

mxsnq commented 4 months ago

Hi!

Some functionalities of Puppeteer that are not covered by our existing actions can be leveraged through CustomJsAction. This action allows you to execute custom JavaScript functions with access to Puppeteer's Page object. An example of such function can be found here.

Other scenarious require patching the service. In order to use puppeteer-extra plugins you'll have to enable them in the app.js module. Currently there are only puppeteer-extra-plugin-stealth and puppeteer-extra-plugin-recaptcha. Once enabled, you can access this functionality through CustomJsActions or by defining your own actions. For the latter, you'll need to define new PuppeteerServiceAction subclass on the client side and to add a corresponding action route in the service.