Closed Jacobojijo closed 1 month ago
Hello @mvolz, I have made the requested changes to the pull request. You can check and if there is still anything I should do, please inform me
Hello @mvolz would you please do the PR review for the changes made ?
@mvolz, can you review my PR? I also think a styling lint should be added to help in other future changes. I'm thinking of introducing one to this project. What do you think?
@mvolz, I guess this PR is done
This is to fix issue #95
Solution
I modified the
scraping.js
file to use more browser-like User-Agent and Accept headers. Here are the key changes:Added constants for user agent and accept header:
Updated all
preq.get()
calls to usegetWithHeaders()
instead.Modified the
meta()
function calls to include the headers:This should improve the reliability of the test suite, especially when dealing with websites that have stricter requirements for incoming requests.