429 too many requests

[IoanaAlexandru]

I'm using textlint with Danger in a project via GitHub Actions.

I have a lot of GitHub links in my documentation, and when textlint runs, some of the checks fail with 429 too many requests, wrongfully marking links as dead.

Is there some way to delay the time between requests that the rule makes? Or maybe some other solution?

[IoanaAlexandru]

@azu for the record, silently adding a label without commenting means the person who posted the issue won't be notified. Luckily, I happened to look at this issue again today.

Not entirely sure what kind of info you need, but here's an example of where I'm running into this issue: https://github.com/acs-upb-mobile/acs-upb-mobile/pull/102#issuecomment-735269091

[azu]

It is difficult problem. It is reasonable that You did many request to GitHub actually and GitHub response "429 too many requests"

There is not perfect solusions.

We can do following as workaround.

• Make requests slow
  • https://github.com/textlint-rule/textlint-rule-no-dead-link/blob/260db6adc8849e134fdf66b357e9f50647ff0e73/src/no-dead-link.js#L19-L21
  • concurrency and interval experimental options may help you
• Add handling for 429 response
  • suppress 429 response as options?

I thought that we do not have enough information to get reasonable resolutions. actual request, actual response, actual limitation(not limited to GitHub.)

[IoanaAlexandru]

@azu I don't really understand the interval count, how does that work? I would like an option to ignore 429 responses just like ignoring redirects.

[chick-p]

@azu Hi. I'm in for similar trouble. My case is that a markdown file includes 20 links of GitHub.com, often responses some status 429 errors.

- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/1
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/2
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/3
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/4
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/5
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/6
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/7
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/8
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/9
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/10
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/11
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/12
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/13
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/14
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/15
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/16
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/17
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/18
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/19
- https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/20

textlint runs with --debug, the result is following as:

$ npx textlint --rule textlint-rule-no-dead-link test.md --debug
  textlint:cli textlint --version: 12.0.2 +0ms
  textlint:cli Running on files, stdin-filename: undefined +1ms
  textlint:module-loader config Config {
  textlint:module-loader   configFile: undefined,
  textlint:module-loader   rulesBaseDirectory: undefined,
  textlint:module-loader   disabledRules: [],
  textlint:module-loader   rules: [ 'no-dead-link' ],
  textlint:module-loader   disabledFilterRules: [],
  textlint:module-loader   filterRules: [],
  textlint:module-loader   presets: [],
  textlint:module-loader   plugins: [],
  textlint:module-loader   pluginsConfig: {},
  textlint:module-loader   rulesConfig: {},
  textlint:module-loader   filterRulesConfig: {},
  textlint:module-loader   rulePaths: [],
  textlint:module-loader   formatterName: undefined,
  textlint:module-loader   quiet: false,
  textlint:module-loader   color: true,
  textlint:module-loader   cache: false,
  textlint:module-loader   cacheLocation: '/Users/chick-p/practice-textlint-rule-no-dead-link/.textlintcache',
  textlint:module-loader   ignoreFile: '/Users/chick-p/practice-textlint-rule-no-dead-link/.textlintignore'
  textlint:module-loader } +0ms
  textlint:module-loader Loading rules from /Users/chick-p/practice-textlint-rule-no-dead-link/node_modules/textlint-rule-no-dead-link/lib/no-dead-link.js +2ms
  textlint:find-util findFiles ignore baseDir: /Users/chick-p/text-rule, normalizeIgnoreFilePath: /Users/chick-p/practice-textlint-rule-no-dead-link/.textlintignore +0ms
  textlint:find-util search patterns: [ 'test.md' ] +0ms
  textlint:find-util search ignore patterns: [ '**/.git/**', '**/node_modules/**' ] +0ms
  textlint:engine-core Process files [ '/Users/chick-p/practice-textlint-rule-no-dead-link/test.md' ] +0ms
  textlint:engine-core No Process files that are un-support extensions: [] +0ms
  textlint:kernel available extensions: [ '.md',     '.markdown', '.mdown',  '.mkdn', '.mkd',    '.mdwn', '.mkdown', '.ron', '.txt',    '.text' ] +0ms
  textlint:kernel use plugin: markdown +0ms
  textlint:core-task no-dead-link pushReport {"message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/19 is dead. (429 too many requests)","index":0} +0ms
  textlint:core-task no-dead-link pushReport {"message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/20 is dead. (429 too many requests)","index":0} +21ms
  textlint:core-task no-dead-link pushReport {"message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/17 is dead. (429 too many requests)","index":0} +1ms
  textlint:core-task no-dead-link pushReport {"message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/18 is dead. (429 too many requests)","index":0} +1ms
  textlint:cli lint results: [{"messages":[{"type":"lint","ruleId":"no-dead-link","message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/17 is dead. (429 too many requests)","index":1113,"line":17,"column":3,"severity":2},{"type":"lint","ruleId":"no-dead-link","message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/18 is dead. (429 too many requests)","index":1183,"line":18,"column":3,"severity":2},{"type":"lint","ruleId":"no-dead-link","message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/19 is dead. (429 too many requests)","index":1253,"line":19,"column":3,"severity":2},{"type":"lint","ruleId":"no-dead-link","message":"https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/20 is dead. (429 too many requests)","index":1323,"line":20,"column":3,"severity":2}],"filePath":"/Users/chick-p/practice-textlint-rule-no-dead-link/test.md"}] +6s
  textlint:@textlint/linter-formatter formatterName: stylish +0ms

/Users/chick-p/practice-textlint-rule-no-dead-link/test.md
  17:3  error  https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/17 is dead. (429 too many requests)  no-dead-link
  18:3  error  https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/18 is dead. (429 too many requests)  no-dead-link
  19:3  error  https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/19 is dead. (429 too many requests)  no-dead-link
  20:3  error  https://github.com/textlint-rule/textlint-rule-no-dead-link/pull/20 is dead. (429 too many requests)  no-dead-link

✖ 4 problems (4 errors, 0 warnings)

[massongit]

429 too many requests can include Retry-After header that indicates how long to wait before making a new request. How about waiting for the number of seconds written in this header if response includes this header? https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

[azu]

How about waiting for the number of seconds written in this header if response includes this header?

Yes. It is reasonable.

Probably, we need to add new option like maxWatingTime? maxRetryTime?. For example, if response includes Retry-After: 6000, we want to skip the checking.

[massongit]

I think that is a good idea.