andykais / scrape-pages

generalized scraper using a single instruction set for any site that can be statically scraped
https://scrape-pages.js.org
MIT License
6 stars 2 forks source link

add download.retry field #22

Open andykais opened 5 years ago

andykais commented 5 years ago

schema:

interface RetryInterface {
  limit?: number // defaults to zero
  retryOnStatusCodes: number[] // defaults to [202, 503, 429, 504]
  delay?: number // defaults to zero
}
type Retry = number | RetryInterface
interface DownloadConfigInit {
  ...
  retry: Retry | undefined
}

Investigate putting logic inside and outside observable.

investigate possibility of retrying page based solely on parsed content. E.g. if page has title "Too Many Retries" then retry same as regular retry (note would vastly change the api and should be thought about in a broader sense)