search

jupyter / nbviewer

nbconvert as a web service: Render Jupyter Notebooks as static web pages
https://nbviewer.jupyter.org
Other
2.21k stars 547 forks source link

GitHub Rate Limit #856

Open jakirkham opened 5 years ago

jakirkham commented 5 years ago

Describe the bug

Appear to be running into a GitHub Rate Limit when using nbviewer. Not sure if it is specific to me or whether this also affects others.

To Reproduce

Steps to reproduce the behavior:

  1. Go to https://nbviewer.jupyter.org/github/beckernick/cudf/blob/2eb214f2cbbc15dd18974eae63ba66fe607b571d/docs/cudf/source/10min-cudf-cupy.ipynb
  2. See error

Expected behavior

Ideally this would load render the notebook.

Screenshots

Screen Shot 2019-10-07 at 11 50 10 AM

Desktop (please complete the following information):

krinsman commented 5 years ago

Hi, this is expected behavior for NBViewer, not a bug. The GitHub API is such that one can only make so many requests every hour. My understanding is that Project Jupyter has a special setup with GitHub to have a higher rate limit than the default.

See also: https://github.com/jupyter/nbviewer/issues/673 https://github.com/jupyter/nbviewer/issues/855

https://developer.github.com/v3/#rate-limiting

jakirkham commented 5 years ago

I wonder if it would be possible for other GitHub users to "share a token" with nbviewer to avoid this issue. This would help bypass this kind of issue. Normally I think this is done by creating an app that requires some permissions to install itself on the GitHub user's account.

krinsman commented 5 years ago

That may be possible. I personally don't know that much about the options/possibilities with the GitHub API.

The code implementing the backend for this is here: https://github.com/jupyter/nbviewer/blob/master/nbviewer/providers/github/client.py

NBViewer already looks for a GITHUB_API_TOKEN via an environment variable, so perhaps it could be useful to somehow add a GUI for doing this.

Based on other issues getting NBViewer to work as expected with user-provided tokens seems like it might not be straightforward: https://github.com/jupyter/nbviewer/issues/402 https://github.com/jupyter/nbviewer/issues/478

Many users however might not have tokens which would make this useful -- I think the default rate limit for an authenticated API user is 5000 per hour, but that Project Jupyter has a special arrangement with GitHub to have a much larger rate limit than that for the NBViewer service.

Another thing to note is that right now NBViewer handles all requests for "raw" notebook files from GitHub via the GitHub provider, which uses the GitHub API and thus is affected by the rate limit: https://github.com/jupyter/nbviewer/blob/master/nbviewer/providers/github/handlers.py#L373 https://github.com/jupyter/nbviewer/blob/master/nbviewer/providers/github/handlers.py#L69

But, conceivably, if such requests were handled by the URL provider instead, it would be possible to load and render the notebook without using the GitHub API and thus without having to worry about getting throttled by the rate limit. I think the downside of this though is that it then makes it impossible to display helpful contextual information -- e.g. what repo this notebook is in, posted by which GitHub user, and then the NBViewer user can't browse around the repo or the user's profiles for similar useful notebooks. In any case that comes down more to a policy solution/decision rather than a technical one, since it is clearly technically feasible, but it nevertheless may not be desirable.

jakirkham commented 5 years ago

FWIW GitHub has an API around generating a token for an App and using it. Would recommend using that as opposed to dropping the token in some form somewhere.

krinsman commented 5 years ago

You don't mean this then, right? https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line

Since that's a GUI, not an API? (Admittedly that was all I had been familiar with previously.)

jakirkham commented 5 years ago

Right. Not that.

Probably you would want this : https://developer.github.com/v3/oauth_authorizations/#get-or-create-an-authorization-for-a-specific-app

nathanshammah commented 4 years ago

I am not sure this is related to the current issue, but links to notebooks at http://qutip.org/tutorials do not work right now.
Also https://nbviewer.jupyter.org itself gives a 503 error.

krinsman commented 4 years ago

@nathanshammah I'll be honest, I've never seen that error before, and don't know what it's due to. (At least at the moment.) Definitely not good though. We should probably make it a separate issue though, just in case.

Edit: See https://github.com/jupyter/nbviewer/issues/857

sidgupta234 commented 4 years ago

This makes a beautiful product unreliable, faced the same today. Maybe we can ask user for an Outh login and use the token for the session?

bollwyvl commented 4 years ago

Githack is back to doing what raw git used to:

https://raw.githack.com/

Slightly more inconvenient, but pretty much works every time.

On Thu, Nov 14, 2019, 13:02 Siddharth Gupta notifications@github.com wrote:

This makes a beautiful product unreliable, faced the same today. Maybe we can ask user for an Outh login and use the token for the session?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbviewer/issues/856?email_source=notifications&email_token=AAALCRAPWK6DZN3EOHJYVPLQTWHDZA5CNFSM4I6GPJNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEECXSOQ#issuecomment-554006842, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAALCRBGYE7SX4AJK4BNUKDQTWHDZANCNFSM4I6GPJNA .