Is your feature request related to a problem? Please describe.
Searching for links is an expensive operation. It requires performing an HTTP Get request to the uri to obtain the header information. If this command is ran 10 times successively in 20 minutes then requests will be ran each time it's executed even though the status of the site has probably not changed.
Describe the solution you'd like
In order to avoid looking up the same URLs consistently over small periods of time, we should add a cache that stores the values for a certain time period then updates if that given time has passed. This time can be stored in an environment variable and/or passed as a flag to the program itself. I would suggest using a key/value store such as Redis https://realpython.com/python-redis/
Is your feature request related to a problem? Please describe. Searching for links is an expensive operation. It requires performing an HTTP Get request to the uri to obtain the header information. If this command is ran 10 times successively in 20 minutes then requests will be ran each time it's executed even though the status of the site has probably not changed.
Describe the solution you'd like In order to avoid looking up the same URLs consistently over small periods of time, we should add a cache that stores the values for a certain time period then updates if that given time has passed. This time can be stored in an environment variable and/or passed as a flag to the program itself. I would suggest using a key/value store such as Redis https://realpython.com/python-redis/