Currently, the Scraper.get_html method will raise an exception for any HTTP error status code, causing the calling process to terminate. While this is the desired behaviour for exceptions and HTTP 5xx status codes, it would be more desirable to simply return None for HTTP 4xx status codes as retrying the request will not change anything.
Update the Scraper.get_html method to check the status code and act appropriately depending on the result: raise an exception for 5xx, return None for 4xx, otherwise return the root HTML element.
Currently, the Scraper.get_html method will raise an exception for any HTTP error status code, causing the calling process to terminate. While this is the desired behaviour for exceptions and HTTP 5xx status codes, it would be more desirable to simply return None for HTTP 4xx status codes as retrying the request will not change anything.
Update the Scraper.get_html method to check the status code and act appropriately depending on the result: raise an exception for 5xx, return None for 4xx, otherwise return the root HTML element.