Open devinshoemaker opened 4 years ago
@beeman has also experienced this and may be able to provide more information.
There's indeed a "Too Many Requests" error code exposed in the API: https://docs.dev.to/api/#operation/updateArticle
I couldn't find the delay though :thinking:
If you're willing to work on a fix I can assign this issue to you :smile_cat: Maybe we can set an arbitrary delay of 500ms in a constant for now (unless someone finds the actual exact time :pray:)
It's probably fastest to let the too many requests happen, then retry after the retry-after
header says so: https://github.com/thepracticaldev/dev.to/blob/5b62811c9874fa2770fe3f720abac1f80fe6279f/spec/requests/api/v0/articles_spec.rb#L564
Indeed, sounds like a good plan :+1:
@maxime1992 do you still want me to try and tackle this then? It should be easy to check for the retry-after
response header. I should have a little bit of time this evening.
@devinshoemaker if you're keen to, sure :)
I'll give it a go! It'd be nice to have this ready for the Nx plugin release.
I'm a bit stumped, to be honest. I'm not familiar with the got
library so I've been doing a little bit of research, and it seems the docs should be able to handle the retry-after
header gracefully.
Docs: https://github.com/sindresorhus/got#retry
I found the values for the rate limit on publishing posts. I'm not sure if the got
maxRetryAfter
property accepts seconds or milliseconds, so I tried both to be sure. However, with this change, I saw no improvements to the HTTP 429 error.
Here is my branch: https://github.com/devinshoemaker/dev-to-git/tree/fix-http-429
Here's the code that defines the rate limit values in dev.to
: https://github.com/thepracticaldev/dev.to/blob/5b62811c9874fa2770fe3f720abac1f80fe6279f/app/services/rate_limit_checker.rb#L11
I also noticed that the version of got
used in this repo is a couple of major versions behind. There are some fixes for the retry
option in more recent versions, but bumping up to even the next available version (10.0.0) causes a number of breaking changes that I'm not comfortable working through.
Hopefully, this give you all some more info to work off of. I might play around with this a bit more over the next couple days, but I'm not sure I'm familiar enough with the libraries and code base at this point.
When looking at some more DEV api usage, I came across this project, which seems to have a throttle for got
.
is there a way to exclude files that weren't changed from sending to the server?
When looking at some more DEV api usage, I came across this project, which seems to have a throttle for
got
.
@beeman Do you use that project at the end?
hanged from sending to the server?
It does that already
Thank you for quick response.
Why then Too many requests
if updated only 1 page?
I never investigated that so I cannot answer that one.
To support my claim, you can have a look here: https://github.com/maxime1992/dev-to-git/blob/9704e9150c4bb6d98790f66be9d6184eaca1a93d/src/article.ts#L116-L123
As for the actual issue there, I'm not actively maintaining that repo. I'm ok reviewing PR but I don't have any more time to put into it I'm afraid so the fix on this one will have to come from the community
I just made a small speculative fix. Unlikely to fix but a small attempt as a quick fix...
https://github.com/maxime1992/dev-to-git/commit/e34dfafeca7d0842a9163ac3510b7af1fdb9a17e
Before: https://github.com/maxime1992/my-dev.to/actions/runs/5165970105/jobs/9305789445 :red_circle:
After: https://github.com/maxime1992/my-dev.to/actions/runs/5166090709/jobs/9305977103 :heavy_check_mark:
Looks like it may be enough for now!
I believe that dev.to has a load balancer or something that prevents too many requests from a single host from being fired off too quickly. Both locally and in my CI/CD pipeline, I frequently get these errors. Due to these errors, it sometimes takes executing
dev-to-git
multiple times to get my posts updated. I think that a slight delay added between publishing posts could avoid this issue.