Validate webpage scrape URLs using regex. The motivation is to avoid making web scrape requests unnecessarily when the URL is invalid. It is possible that there may be some URL's for which this regex match does not validate correctly. However, these cases should be very niche and this change will avoid several unnecessary web scrape requests for a number of cases where the URL is obviously invalid.
Type of change
New feature (change which adds functionality)
Tests
Tested locally with common valid and invalid URL's/patterns
Pull Review Summary
Description
Validate webpage scrape URLs using regex. The motivation is to avoid making web scrape requests unnecessarily when the URL is invalid. It is possible that there may be some URL's for which this regex match does not validate correctly. However, these cases should be very niche and this change will avoid several unnecessary web scrape requests for a number of cases where the URL is obviously invalid.
Type of change
Tests
Tested locally with common valid and invalid URL's/patterns