Not sure if this has become an issue to anyone else, but certain websites are unable to be scraped without using a user agent. This is the simplest solution to this issue and doesn't require another dependency. A more robust solution would be using something like fake-useragent.
Not sure if this has become an issue to anyone else, but certain websites are unable to be scraped without using a user agent. This is the simplest solution to this issue and doesn't require another dependency. A more robust solution would be using something like fake-useragent.