Better Github token management + minor bug fixes

hisxo / gitGraber

gitGraber: monitor GitHub to search and find sensitive data in real time for different online services such as: Google, Amazon, Paypal, Github, Mailgun, Facebook, Twitter, Heroku, Stripe...

GNU General Public License v3.0

2k stars 328 forks source link

Better Github token management + minor bug fixes #5

Closed darkpills closed 4 years ago

darkpills commented 5 years ago

Adding advanced Github token management by using X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After from response
Adding default value to -k option to make it easier to launch a first command
Fix: the script would start even if with empty query parameter
Fix: some keywords would not be queried if all github token would have been used

hisxo commented 5 years ago

Hello @darkpills,

First, thanks for your help and for and these improvements! After a few tests, it seems that some of your modifications have introduced a bug that does not parse raw (JSON) and/or does not display the results in CLI and don't send Slack notification if a pattern is found.

For a same request, we compared the actual script and your script, and the new version don't find any tokens. Could you check on your side too?

We check this problem internally and we will be back shortly.

Thank you :)

Adrien

rmarot commented 4 years ago

Hello @darkpills

Thank you for the update and the enhancement regarding the token management !

It seems that some part of the code also needs to be updated because we have multiple occurences of HTTP requests to retrieve the different information and that each time we doing a request we have to set the headers with a working Github token.

For example, the parseResults function has a (dirty) hardcoded value for a specific token in the header :

headers = {'Accept': 'application/vnd.github.v3.text-match+json', 'Authorization': 'token ' + config.GITHUB_TOKENS[3] } However, the modifications suggested transform the config.GITHUB_TOKENS list to a dict which causes some issues.

I think that it would be good to ensure that each HTTP request uses the getGithubToken function to build the headers, and that the token status update is properly done after each request.

What is your feeling about that ?

Thanks !

darkpills commented 4 years ago

Hello,

Sorry, there were several occurence of the git call function. I did a fix in this last commit and improved the token storage to be PATH (URI) based instead of having counters for any PATH.

Hope this new commit fix the issue.

rmarot commented 4 years ago

Hey @darkpills

Thanks for the update, this looks nice. Let give us a review / try and we will do a feedback shortly.

darkpills commented 4 years ago

Thank you :) !

hisxo commented 4 years ago

Thanks @darkpills!

darkpills commented 4 years ago

You're welcome, really really nice tool! You did a great job!