I have changed the GitDorker for my own purpose in order to work as I prefer.
Those are the changes:
The threads depend on the number of tokens in the -tf parameter file, each thread uses only 1 token.
The sleep 60 seconds to don't reach the API rate limit has been changed in order to take the number of requests that are left from the response header "X-RateLimit-Remaining" (Each token has its own remaining number)
The sleep 60 seconds has been changed to sleep until the "X-RateLimit-Reset" header says, so we sleep only the necessary time.
There seems to be a "secondary-rate-limit" on GitHub API. As I haven't seen how to bypass it, I have added two things, random sleep between requests on the same thread (From 0.5 to 1.5) and a while to only stop the GitDorker when all requests have been correctly processed. (At least the requests where the problem is an API error)
In addition to the while to process all the Dorks, I have added a try-except in order to get the Control+C on the script and end the API requests but continue with the output process. (If the GitDorker process is too slow, you can stop the API requests and make a manually process with the URLs that are left)
Between loops in order to process all the dorks, I have added some output to see the time spent and the unique errors to see some debug information.
With all those changes, I can process all the Dorks without a problem and if I use more tokens, the process is faster.
Make any changes to the code as you want if you like this pull request and if you have any questions don't hesitate to contact me.
I have changed the GitDorker for my own purpose in order to work as I prefer. Those are the changes:
With all those changes, I can process all the Dorks without a problem and if I use more tokens, the process is faster. Make any changes to the code as you want if you like this pull request and if you have any questions don't hesitate to contact me.