Removes the validate_url call when running a source through a Scraper.
Ensures no external network requests are made when running tests.
Adds a with_net_connection test method to allow real network calls in tests if needed (currently used to test private_address_check is working).
Motivation and context
A user created a source with a URL that returned a 301 redirect. This worked fine when testing, but was considered invalid when running through the scraper. The URL format is already validated when creating a source, and any issues when resolving the URL should be raised when running the actual Ingestor.
Checklist
[x] I have read and followed the CONTRIBUTING guide.
[x] I confirm that I have the authority necessary to make this contribution on behalf of its copyright owner and agree
to license it to the TeSS codebase under the
BSD license.
Summary of changes
validate_urlcall when running a source through aScraper.with_net_connectiontest method to allow real network calls in tests if needed (currently used to testprivate_address_checkis working).Motivation and context
A user created a source with a URL that returned a 301 redirect. This worked fine when testing, but was considered invalid when running through the scraper. The URL format is already validated when creating a source, and any issues when resolving the URL should be raised when running the actual
Ingestor.Checklist