Open Bchass opened 1 year ago
Someone is attempting to deploy a commit to a Personal Account owned by @neokd on Vercel.
@neokd first needs to authorize it.
Can the proxy be made like an parameter? Like if user set as True then proxy works or else it tries to scrape without proxy.?
Can the proxy be made like an parameter? Like if user set as True then proxy works or else it tries to scrape without proxy.?
Are you referring to a user-agent a browser would use?
No, for now we were thinking that proxy is getting complicated so if we allow user to pass an argument that they want to use scrapper or not.
No, for now we were thinking that proxy is getting complicated so if we allow user to pass an argument that they want to use scrapper or not.
I see. I'll look into adding that option for the scraper.
@Bchass looks good I'll review it fully and merge it. Thinking to publish the scrapper as package on Pypi?
Is this project being maintained anymore?
@Bchass yeah the project is maintained. There was merge conflict so i didn't merge this PR.
Description
Fixed a bug and switched over to a
deque
instead of a listRelated Issues
https://github.com/neokd/DataStorehouse/issues/112
Changes Made
load_proxy_list()
doesn't rely on an absolute path anymore, this wasn't working as intended.rotate_proxy()
Switched over to a deque for better efficiency when rotating proxiesScreenshots (if applicable)
N/A
Checklist
Please review and check the following before submitting your pull request:
Additional Notes
I would like to add test cases for this