Closed Jacobojijo closed 2 weeks ago
@mvolz, any PR review on this?
@mvolz, I realized the issue was that the package.json was not in sync with package-lock.json because there was an update of package.json that was not reflected on package-lock.json. I have fixed and you can now test the PR.
@mvolz, PR merge?
Thanks!
This is to fix issue #95
Solution
I modified the
scraping.js
file to use more browser-like User-Agent and Accept headers. Here are the key changes:Added constants for user agent and accept header:
Updated all
preq.get()
calls to usegetWithHeaders()
instead.Modified the
meta()
function calls to include the headers:This should improve the reliability of the test suite, especially when dealing with websites that have stricter requirements for incoming requests.