ispras / scrapy-puppeteer

Library that helps use puppeteer in scrapy.

BSD 3-Clause "New" or "Revised" License

49 stars 4 forks source link

Context restoring #27

Open MatthewZMSU opened 1 month ago

MatthewZMSU commented 1 month ago

Description

Implemented new downloader middleware for restoring contexts when they are destroyed. Added example spider. Added info about the middleware to README.

Restructured the project to have more than 1 middleware. Changed init.py in the corresponding way to provide backward compatibility.

Type of change

[x] New feature (non-breaking change which adds functionality)
[x] This change requires a documentation update

How Has This Been Tested?

I tested the middleware with several spuder on my PC.

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings

MatthewZMSU commented 1 month ago

Service and Recaptcha middlewares were mostly copied from previous middleware.py file. I only fixed typo and added proper logging.