gist-based hook not working - github "API rate limit exceeded for Marak."

[pyhedgehog]

https://hook.io/pyhedgehog/testjs/source:

Not valid JSON: {"message":"API rate limit exceeded for Marak.","documentation_url":"https://developer.github.com/v3/#rate-limiting"}

Maybe Conditional requests can help.

[Marak]

@pyhedgehog -

Could you try logging out and logging in again using Github login on hook.io?

I think your account might be cached and using my token ( we fixed this a few months ago, but only for new accounts / new logins ).

If not, I'll take another look at see why this is happening. It suppose to be keyed per user.

[pyhedgehog]

I was running hooks without login at all. Login solves problem. Thank you.

[Marak]

Something still seems wrong, I'm getting login errors now with Github accounts....

[Marak]

I've reached out to Github support to get my personal account limits updated.

hook.io should use each individual users OAuth token for Github API access ( if they've logged in since Jan 1st of this year ). Previously accounts and logins were using an OAuth token from my personal account. Logging out / logging in should fix the issue.

All the examples and tests still use my tokens, so they be using a lot of traffic. Getting the limits updated with help and I'll see if I can find what exactly is making so many requests.

[pyhedgehog]

I can once again note that according to docs conditional requests doesn't counts to rate limits unless object changed. This will be important for non-github users.

[pyhedgehog]

Strange: I'm logged in (with GitHub account - pyhedgehog), but right now https://hook.io/pyhedgehog/testjs tells "API rate limit exceeded for Marak.". I've tried to logout/login with no effect.

[Marak]

I'll look into your user document and see whats going on. The intermittent behavior is probably due to rate limit maxing out, then going down over time, then maxing out again.

There could still be a bug in the code for mapping OAuth tokens to individual users ( since we previously migrated this code from using Marak ). It could also have to do with req.checkAccess function, but I doubt it.

This shouldn't be happening since we updated to using tokens per user. Will investigate further.

[Marak]

To be clear, the API requests are suppose to be counted against the account which is hosting the service. For non-github users, they shouldn't ever be touching these code paths. Github accounts are required for using Gist or Github based sources on hook.io

[pyhedgehog]

Not checkAccess. Looking /source was buggy too. After switching sourceType to code and back gist it's fine. Seems caching is per hook.

[pyhedgehog]

My previous comment was wrong. Seems that was artefact of revived limits. Still every request via Marak user.

[Marak]

@pyhedgehog - I have found the issue at: https://github.com/bigcompany/hook.io/blob/master/lib/resources/hook/fetchHookSourceCode.js#L80

It seems I never switched over the legacy hook fetch source code to use new microcule API or to use new individual token per use API.

Should be an easy fix, but I'm thinking there must be somewhere else still referring to the old code since this should only affect /source and not the main service.

[Marak]

@pyhedgehog I figure it out!

You had old user document in cache ( previous to when we had github tokens per user ).

Once I deleted your user from cache, it repopulated immediate with correct token.

You should not longer be getting marak errors for your main service, and I'll have a patch out for /source code path shortly.

Thank you for bringing this to my attention. I need to invalidate the cache for all users.

[Marak]

Just invalidated all users at once. Need to remember to not do that again ( invalidate small sections at a time ), it's causing the couch to melt in order to repopulate. Should be up to speed again soon.

[Marak]

I got in touch with Github support and the issue is odd.

It seems that beta.gitlance.net application was causing a ton of requests to my personal OAuth token which caused the buffer I created for legacy services and APIs to exhaust quickly based on Github rate limits.

To give perspective, ~88% or so of the total API requests to Marak in the past week or so came from Gitlance.

For now I disabled access to Gitlance on my personal account and reached out to them to see what's going on. Moving forward I should be able to remove all references to my personal Github account in the code. All previous accounts have been migrated and I will update the /source path for Github and Gist based services.

[Marak]

Haven't seen this issue come up again.

Going to leave this open until I can add a few commits based on above comments.

[pyhedgehog]

It's still here.

[Marak]

Which service?

[pyhedgehog]

There are no service/running metric. I think it's /pyhedgehog/hooklog, but I see no branch not closed by hook.res.end().

[Marak]

@pyhedgehog Are you seeing this issue, or the other rate limit issue not related to github?

I've been testing a new update which should help.

[pyhedgehog]

You're right — I've mixed up this issue with #282.