Added a basic stepback delay function to the retry logic for downloader.py
It increases the total number of retry attempts from 3 -> 30 to allow for the scan to be more resilient to website instability before giving up and killing the scan.
To reduce the risk of increased numbers of requests worsening website performance and risking it being flagged as spamming, the stepback delay will ensure that requests will have some amount of delay between attempts (after the initial retry).
To increase the likelihood of a successful request, the delay will increase from 0s -> 3m in the hopes of giving the website a time window to recover before the scan gives up.
For simplicity reasons, it is batching the delay by groups of 5 attempts with the last 10 attempts having the largest delay, assuming if there is an outage that long it is likely a larger issue on the website side, so it is better not to spam them frequently.
I also modified the retry logging to display the current retry_delay seconds for easier monitoring/debugging and for potentially allowing future versions to fine-tune delays based on observed data (Ex. maybe having fewer steps with bigger delay jumps reduces total needed retries by ~30% during an outage).
This could eventually be improved to allow end users to configure these settings via the extension settings, but for now this seems like a sane default. Feel free to modify it further, but this gives a descent amount of resiliency.
Added a basic stepback delay function to the retry logic for downloader.py
It increases the total number of retry attempts from 3 -> 30 to allow for the scan to be more resilient to website instability before giving up and killing the scan.
To reduce the risk of increased numbers of requests worsening website performance and risking it being flagged as spamming, the stepback delay will ensure that requests will have some amount of delay between attempts (after the initial retry).
To increase the likelihood of a successful request, the delay will increase from 0s -> 3m in the hopes of giving the website a time window to recover before the scan gives up.
For simplicity reasons, it is batching the delay by groups of 5 attempts with the last 10 attempts having the largest delay, assuming if there is an outage that long it is likely a larger issue on the website side, so it is better not to spam them frequently.
I also modified the retry logging to display the current retry_delay seconds for easier monitoring/debugging and for potentially allowing future versions to fine-tune delays based on observed data (Ex. maybe having fewer steps with bigger delay jumps reduces total needed retries by ~30% during an outage).
This could eventually be improved to allow end users to configure these settings via the extension settings, but for now this seems like a sane default. Feel free to modify it further, but this gives a descent amount of resiliency.