anuraghazra commented 2 years ago

Hi everyone, I see that all personal access tokens are currently failing, thus you will see the "Maximum retries exceeded" error.

I'm looking into the issue to figure out what caused it.

Ideally, it should be fixed in an hour when the PATs get rest.

Details

Public instance information

Explanation

Hey everyone, sorry for the recent downtimes.

For the sake of transparency, we get a huge amount of requests per month (it's kind of mind boggling for a side project which I built just for fun). Here's the vercel dashboard statistics:

As you can see even with 73% of the responses being cached by vercel the amount of live requests are huge, and the bottleneck here is Personal Access Tokens

Each PAT have 5000 points, and based on the complexity of the GraphQL query & resource limitations set by GitHub, it's actually very very low considering that amount of requests.

To circumvent the issue, we currently have 7 PATs (thats 35k points) which sometimes get exhausted. Thanks to @rickstaa we are going to add few more probably bumping it up to 10 or 12, hopefully that could help alleviate the downtimes.

Note: That these downtimes are temporary, and only lasts for 1hour, since PATs regenerate after 1 hour

So whats the idea way?

https://github.com/anuraghazra/github-readme-stats/issues/1471#issuecomment-979306704 Deploy on your own vercel instance. WHY? It's easy to do, free & reliable.

It reduces loads on our vercel instance.
Your own PAT will give you more concrete stats including private stats
It ensures that there won't be any downtime on your profile stats because you are the only one who is gonna hit your URL

Originally posted by @anuraghazra in https://github.com/anuraghazra/github-readme-stats/issues/2130#issuecomment-1270624232

rickstaa commented 1 year ago

@anuraghazra, I think we should also increase the retry number in the retry function.

https://github.com/anuraghazra/github-readme-stats/blob/94d0978d2bf7c5324513ac924cb42afe1e7774cc/src/common/retryer.js#L13

see https://github.com/anuraghazra/github-readme-stats/pull/2121.

JacobMGEvans commented 1 year ago

It's not really.

The BUG is back... I should be more clear lol

seilorjunior commented 1 year ago

the bug is back to me

krish4uu commented 1 year ago

I'm also getting maximum retries exceeded! Screen Shot 2022-10-06 at 10 11 59 AM

rickstaa commented 1 year ago

Posting this here as a reference:

Hey everyone, sorry for the recent downtimes.

For the sake of transparency, we get a huge amount of requests per month (it's kind of mind boggling for a side project which I built just for fun). Here's the vercel dashboard statistics:

As you can see even with 73% of the responses being cached by vercel the amount of live requests are huge, and the bottleneck here is Personal Access Tokens

Each PAT have 5000 points, and based on the complexity of the GraphQL query & resource limitations set by GitHub, it's actually very very low considering that amount of requests.

To circumvent the issue, we currently have 7 PATs (thats 35k points) which sometimes get exhausted. Thanks to @rickstaa we are going to add few more probably bumping it up to 10 or 12, hopefully that could help alleviate the downtimes.

Note: That these downtimes are temporary, and only lasts for 1hour, since PATs regenerate after 1 hour

So whats the idea way?

https://github.com/anuraghazra/github-readme-stats/issues/1471#issuecomment-979306704 Deploy on your own vercel instance. WHY? It's easy to do, free & reliable.

It reduces loads on our vercel instance.
Your own PAT will give you more concrete stats including private stats
It ensures that there won't be any downtime on your profile stats because you are the only one who is gonna hit your URL

Originally posted by @anuraghazra in https://github.com/anuraghazra/github-readme-stats/issues/2130#issuecomment-1270624232

rickstaa commented 1 year ago

This bug is improved when the following PRs are merged:

2121: Adds support for more PATs.
2124: Speeds up initial card loading.
2159: Removes one GraphQl function call, reducing the PAT deterioration rate.
2177: Makes sure that errors are not cached.
2178: Should be checked since it looks like some PATs are expired.

oleggtro commented 1 year ago

Hey there, I just encountered the same error as probably most here over the last month. Just curious: What would prevent us/you/someone from hosting this on a VPS or sth? The request load doesn't seem that bad (10 requests/s) after all. Or is there no issue with vercel at all and its just the GitHub API quota being used up?

rickstaa commented 1 year ago

Hey there, I just encountered the same error as most here over the last month. Just curious: What would prevent us/you/someone from hosting this on a VPS or sth? The request load doesn't seem that bad (10 requests/s). Or is there no issue with vertical at all, and it's just the GitHub API quota being used up?

@cloudybyte Thanks for letting us know and for your question. There exist Vercel Limits, but we are not close to hitting these because we using caching. The GraphQL limits currently cause the PAT error (see https://docs.github.com/en/graphql/overview/resource-limitations). Hosting on another provider will, therefore, not help.

The PRs explained in https://github.com/anuraghazra/github-readme-stats/issues/1471#issuecomment-1277550011 will improve the downtime. Apart from that, the best option is hosting your own Vercel instance since you can use your own GitHub quota (see https://github.com/anuraghazra/github-readme-stats#deploy-on-your-own-vercel-instance). I'm also planning on releasing a GitHub action version (see https://github.com/anuraghazra/github-readme-stats/issues/2179). This would allow for more accurate stats, no downtime and faster loading times. I am, however, waiting on some other PRs to be merged/completed before continuing this feature.

rickstaa commented 1 year ago

@cloudybyte When did you encounter the error? Our downtime monitor (queries every 15 min) does not show any downtime in the last seven days.

Maybe it was down for a minimal amount of time?

oleggtro commented 1 year ago

Its on my profile right now...

rickstaa commented 1 year ago

Very strange. I don't have this error when I request your card at https://github-readme-stats.vercel.app/api?username=cloudybyte&show_icons=true&locale=en&theme=cobalt.

[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?username=cloudybyte&show_icons=true&locale=en&theme=cobalt)](https://github.com/anuraghazra/github-readme-stats)

https://user-images.githubusercontent.com/17570430/196469473-607cb853-ee18-4631-aa74-41b18aef6adf.mp4

oleggtro commented 1 year ago

@cloudybyte Thanks for letting us know and for your question. There exist Vercel Limits, but we are not close to hitting these because we using caching. The GraphQL limits currently cause the PAT error (see https://docs.github.com/en/graphql/overview/resource-limitations). Hosting on another provider will, therefore, not help.

The PRs explained in #1471 (comment) will improve the downtime. Apart from that, the best option is hosting your own Vercel instance since you can use your own GitHub quota (see https://github.com/anuraghazra/github-readme-stats#deploy-on-your-own-vercel-instance). I'm also planning on releasing a GitHub action version (see #2179). This would allow for more accurate stats, no downtime and faster loading times. I am, however, waiting on some other PRs to be merged/completed before continuing this feature.

Its really not bothering me enough to set up my own vercel instance, but I wanted to improve the experience for everyone.

oleggtro commented 1 year ago

Very strange. I don't have this error when I request your card at https://github-readme-stats.vercel.app/api?username=cloudybyte&show_icons=true&locale=en&theme=cobalt.
[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?username=cloudybyte&show_icons=true&locale=en&theme=cobalt)](https://github.com/anuraghazra/github-readme-stats)

Thats really strange... Its working in your comment, but broken on my profile... Thats probably sth to look into

rickstaa commented 1 year ago

@cloudybyte Thanks for letting us know and for your question. There exist Vercel Limits, but we are not close to hitting these because we using caching. The GraphQL limits currently cause the PAT error (see https://docs.github.com/en/graphql/overview/resource-limitations). Hosting on another provider will, therefore, not help. The PRs explained in #1471 (comment) will improve the downtime. Apart from that, the best option is hosting your own Vercel instance since you can use your own GitHub quota (see https://github.com/anuraghazra/github-readme-stats#deploy-on-your-own-vercel-instance). I'm also planning on releasing a GitHub action version (see #2179). This would allow for more accurate stats, no downtime and faster loading times. I am, however, waiting on some other PRs to be merged/completed before continuing this feature.

Its really not bothering me enough to set up my own vercel instance, but I wanted to improve the experience for everyone.

That's very nice of you help is always welcome! 🚀

rickstaa commented 1 year ago

Very strange. I don't have this error when I request your card at https://github-readme-stats.vercel.app/api?username=cloudybyte&show_icons=true&locale=en&theme=cobalt.
[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?username=cloudybyte&show_icons=true&locale=en&theme=cobalt)](https://github.com/anuraghazra/github-readme-stats)
Thats really strange... Its working in your comment, but broken on my profile... Thats probably sth to look into

It could be a caching problem, but I thought I fixed it https://github.com/anuraghazra/github-readme-stats/commit/4df8094e5266f28f7367552980f4e00b0c9421a9. Maybe the header is still incorrect. Could you maybe do the following troubleshooting for me?

Steps

Open https://github-readme-stats.vercel.app/api?username=cloudybyte&show_icons=true&locale=en&theme=cobalt in a separate browser tab to see if it works.
Open https://github-readme-stats.vercel.app/api?username=cloudybyte&show_icons=true&locale=en&theme=cobalt in another browser to see if it works.
Can you check if your browser stored cache (see https://developer.chrome.com/docs/devtools/storage/cache/).

oleggtro commented 1 year ago

https://user-images.githubusercontent.com/49766357/196471465-dc215627-e6dc-4df0-ac6e-33cdf0e1ba08.mp4

oleggtro commented 1 year ago

1) works 2) works (tested in chromium, ff is my default) 3) there is apparently no local cache for *.vercel.app saved

That sounds a lot like a FF bug to me, if you prevented caching of errors by setting the correct response header...

rickstaa commented 1 year ago

works

works (tested in chromium, ff is my default)

there is apparently no local cache for *.vercel.app saved

That sounds a lot like a FF bug to me, if you prevented caching of errors by setting the correct response header...

Thanks for the information! 💪🏻 I will check the caching headers later this week. What is a FF bug 🤓?

rickstaa commented 1 year ago

O I think you mean Firefox! Good to know! The caching behaviour differs per browser. I will double-check the headers.

rickstaa commented 1 year ago

@cloudybyte I can reproduce your error in firefox. Can you maybe check the caching headers in FF for me (see video)?

https://user-images.githubusercontent.com/17570430/196475753-feb94563-c547-4a72-9850-dd935bb32082.mp4

rickstaa commented 1 year ago

@cloudybyte One last thing that would help after you gave me the cache header information is to check if the problem goes away when you click the Disable Cache button. 🙏🏻 For me, that solves the problem, so the no-storage header is not working on Firefox.

oleggtro commented 1 year ago

Ooh, I think I know where the issue is and why it only works when requesting directly: GitHub seems to cache the image themselves when building the profile page. The error is probably cached on their side. Thats why requesting it directly from the vercel instance works.

oleggtro commented 1 year ago

Mhm, maybe not. I tried requesting my profile with Chromium and FF in private mode and both are working. The only issue is with my non-private-mode FF (firefox-developer-edition 106.0b9 (64-bit) to be precise)

rickstaa commented 1 year ago

Ooh, I think I know where the issue is and why it only works when requesting directly: GitHub seems to cache the image itself when building the profile page. The error is probably cached on their side. That's why requesting it directly from the Vercel instance works.

Great detective work! Could be the issue. I will check if the GitHub CDN respects the no-store header.

oleggtro commented 1 year ago

Based on my other findings, it doesn't look like GitHub is being the issue.

rickstaa commented 1 year ago

Based on my other findings, it doesn't look like GitHub is the issue.

So it just works if you use the GitHub CDN URL in any other browser?

oleggtro commented 1 year ago

Yea

rickstaa commented 1 year ago

Yea

It looks like a particular Vercel + Firefox error. The card shown in https://github.com/anuraghazra/github-readme-stats/issues/1471#issuecomment-1282585613 is coming from the Vercel cache (time 15:30). I will check at the Vercel GitHub.

rickstaa commented 1 year ago

Yea

I think firefox is not respecting the no-cache header then. The private mode does not store cache

Yea

It looks like a particular Vercel + Firefox error. The card shown in #1471 (comment) is coming from the Vercel cache (time 15:30). I will check at the Vercel GitHub.

Changing the error cache header to Cache-Control: no-cache, no-store, must-revalidate might solve the issue (see https://stackoverflow.com/a/2068407/8135687).

Kentakoong commented 1 year ago

mine's not showing the GH Stats and the Top Langs

ps: I'm using the img tag so I guess it's a caching issue like said in the upper comments!

oleggtro commented 1 year ago

Can you try fetching the stats directly using this url: https://github-readme-stats.vercel.app/api?hide_rank=true&show_icons=true&include_all_commits=false&count_private=true&disable_animations=false&theme=dark&locale=en&hide_border=true&custom_title=Github Stats&username=Kentakoong and tell us which browser u're using? (Your profile is broken on my side, but fetching directly works fine...)

rickstaa commented 1 year ago

mine's not showing the GH Stats and the Top Langs

ps: I'm using the img tag so I guess it's a caching issue like said in the upper comments!

@Kentakoong, now it doesn't seem to be a caching issue anymore. The PATs quota for this hour has simply been used up. To decrease this occurrence, #2121 has to be merged, and some PATs have to be replaced (see #2121). I will check back in an hour.

rickstaa commented 1 year ago

Can you try fetching the stats directly using this url: https://github-readme-stats.vercel.app/api?hide_rank=true&show_icons=true&include_all_commits=false&count_private=true&disable_animations=false&theme=dark&locale=en&hide_border=true&custom_title=Github Stats&username=Kentakoong and tell us which browser u're using? (Your profile is broken on my side, but fetching directly works fine...)

You are right! Fetching directly still works, so maybe the PATs still have quota left, and this is a cached response on the server side that is still being shown from the GitHub CDN. Very strange problem.

Your syntax markdown

[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?hide_rank=true&show_icons=true&include_all_commits=false&count_private=true&disable_animations=false&theme=dark&locale=en&hide_border=true&custom_title=Github&nbsp;Stats&username=Kentakoong)](https://github.com/anuraghazra/github-readme-stats)

Your syntax with img tag

<div align="center">
<img src="https://github-readme-stats.vercel.app/api?hide_rank=true&show_icons=true&include_all_commits=false&count_private=true&disable_animations=false&theme=dark&locale=en&hide_border=true&custom_title=Github&nbsp;Stats&username=Kentakoong" height="150" alt="ghstats"  />
</div>

Your syntax in markdown while breaking the cache with &safdsaf=adsfsaf

[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?hide_rank=true&show_icons=true&include_all_commits=false&count_private=true&disable_animations=false&theme=dark&locale=en&hide_border=true&custom_title=Github&nbsp;Stats&username=Kentakoong&safdsaf=adsfsaf)](https://github.com/anuraghazra/github-readme-stats)

Your syntax with img tag while breaking the cache with &safdsaf=adsfsaf

<div align="center">
<img src="https://github-readme-stats.vercel.app/api?hide_rank=true&show_icons=true&include_all_commits=false&count_private=true&disable_animations=false&theme=dark&locale=en&hide_border=true&custom_title=Github&nbsp;Stats&username=Kentakoong&safdsaf=adsfsaf" height="150" alt="ghstats"  />
</div>

Static image of the results above in case they update

rickstaa commented 1 year ago

@cloudybyte and @ Kentakoong It Looks like the problem at the current time only is present when the <IMG> tag is used (see https://github.com/anuraghazra/github-readme-stats/issues/1471#issuecomment-1282690404). This, however, is, I think, a coincidence and could as well have been the other way around.

I think the GitHub CDN could still cause this caching problem. This behaviour can be seen in action if I used the cached image URL directly:

<div align="center">
<img src="https://camo.githubusercontent.com/bdc4b7001ae4db64d42b0567e58c40584162aff3cd12f250869298dafec66d78/68747470733a2f2f6769746875622d726561646d652d73746174732e76657263656c2e6170702f6170693f686964655f72616e6b3d747275652673686f775f69636f6e733d7472756526696e636c7564655f616c6c5f636f6d6d6974733d66616c736526636f756e745f707269766174653d747275652664697361626c655f616e696d6174696f6e733d66616c7365267468656d653d6461726b266c6f63616c653d656e26686964655f626f726465723d7472756526637573746f6d5f7469746c653d476974687562c2a0537461747326757365726e616d653d4b656e74616b6f6f6e67" height="150" alt="ghstats"  />
</div>

To be sure, I asked a question in the Vercel repository.

Static image in case GitHubs CDN gets updated

rickstaa commented 1 year ago

@cloudybyte I think I found the problem. The GitHub CDN needs the no-cache (see https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-anonymized-urls#an-image-that-changed-recently-is-not-updating). I updated the cache using the curl -X PURGE https://camo.githubusercontent.com/bdc4b7001ae4db64d42b0567e58c40584162aff3cd12f250869298dafec66d78/68747470733a2f2f6769746875622d726561646d652d73746174732e76657263656c2e6170702f6170693f686964655f72616e6b3d747275652673686f775f69636f6e733d7472756526696e636c7564655f616c6c5f636f6d6d6974733d66616c736526636f756e745f707269766174653d747275652664697361626c655f616e696d6174696f6e733d66616c7365267468656d653d6461726b266c6f63616c653d656e26686964655f626f726465723d7472756526637573746f6d5f7469746c653d476974687562c2a0537461747326757365726e616d653d4b656e74616b6f6f6e67 command and now your profile is working again.

oleggtro commented 1 year ago

Mhm, that sounds weird. Thanks for fixing it tho. hmu if u need help or anything

rickstaa commented 1 year ago

Yea strange behaviour indeed the GitHub CDN should also respect no-store header. I think 98f9045f987112bb3d4f1a0160f3e0d93ef5bd52 will fix the issues. Please feel free to comment below if it occurs again.

HADMARINE commented 1 year ago

Just informing that the server seems to be down for a while

rickstaa commented 1 year ago

This bug is improved when the following PRs are merged:

fix: fix retry max-out bug #2121: Adds support for more PATs.

feat: improve card loading speed #2124: Speeds up initial card loading.

feat: enable multi page star fetching for private vercel instances #2159: Removes one GraphQl function call, reducing the PAT deterioration rate.

fix: prevent totalCommitsFetch error result from being cached #2177: Makes sure that errors are not cached.

feat: add PAT monitoring functions #2178: Should be checked since it looks like some PATs are expired.

@HADMARINE, thanks for letting us know the instance was indeed down for about 10 minutes. @anuraghazra, can you look at the issues above if you find the time? Especially #2178 and #2159. It seems that only one of our PATs is still working. 🤔

jesterb0206 commented 1 year ago

@anuraghazra @rickstaa I'm receiving it this morning

showierdata9978 commented 1 year ago

self doing mine worked, I had to purge it though.

rickstaa commented 1 year ago

We are aware of the issue and actively monitoring the PATs thanks for letting us know. 👍🏻

dhiegodias commented 1 year ago

Down here as well :(

Thanks guys!

superbunny38 commented 1 year ago

How do you solve this..? I tried forking vercel like the readme says, but it won't solve.. I also changed the json file :'(

showierdata9978 commented 1 year ago

You need to create a GitHub personal access token, and put it in a env var called PAT_1

On Wed, Nov 30, 2022, 10:58 AM Chaeeun Ryu @.***> wrote:

[image: image] https://user-images.githubusercontent.com/48243487/204860253-14165d76-7320-40d4-a82d-a584b41d67b6.png

How do you solve this..? I tried forking vercel like the readme says, but it won't solve.. I also changed the json file :'(

— Reply to this email directly, view it on GitHub https://github.com/anuraghazra/github-readme-stats/issues/1471#issuecomment-1332466630, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQHW4PYQXZ6KQZLBC2Y6UOLWK6BS3ANCNFSM5IWV7LGQ . You are receiving this because you commented.Message ID: @.***>

superbunny38 commented 1 year ago

Thank you so much for your reply. I followed the link in readme (https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fanuraghazra%2Fgithub-readme-stats) and it didn't solve.. Do you know how I could get personal access token and put it in a env var called PAT_1? I have no idea how I should do it..

showierdata9978 commented 1 year ago

if you have ever used git on the cmdline its basically the same thing

here`s the url to make a token: https://github.com/settings/tokens how to add an env var to Vercel: https://vercel.com/docs/concepts/projects/environment-variables

dginovker commented 1 year ago

@rickstaa Is there a place we can provide you PAT tokens?

rickstaa commented 1 year ago

@dginovker Thanks for your offer. I, however, have more than enough we just need to add them to Vercel. 👍🏻 But most importantly, it is best if more people deploy their own Vercel instance (see https://github.com/anuraghazra/github-readme-stats#deploy-on-your-own-vercel-instance).

mrtnvgr commented 1 year ago

@rickstaa @anuraghazra this appeared again

anuraghazra / github-readme-stats

[Down Time] Card throws 'maximum retries exceeded' error #1471

Details

Public instance information

Explanation

2121: Adds support for more PATs.

2124: Speeds up initial card loading.

2159: Removes one GraphQl function call, reducing the PAT deterioration rate.

2177: Makes sure that errors are not cached.

2178: Should be checked since it looks like some PATs are expired.

Steps

Your syntax markdown

Your syntax with img tag

Your syntax in markdown while breaking the cache with &safdsaf=adsfsaf

Your syntax with img tag while breaking the cache with &safdsaf=adsfsaf

Static image of the results above in case they update

Static image in case GitHubs CDN gets updated