Nhoya / gOSINT

OSINT Swiss Army Knife
GNU General Public License v3.0
613 stars 80 forks source link

Github module is only checking last 100 commits #1

Closed Te-k closed 6 years ago

Te-k commented 6 years ago

To get the list of commitsgitAPI.go calls the API manually with an url like https://api.github.com/repos/PyGithub/PyGithub/commits?per_page=5000. The last parameter does not seem to have any effect, only the last 100 commits are actually returned by this page.

It would be better to get all of them, only solution I know is to use the API (which is really slow for large repos)

Nhoya commented 6 years ago

Yea, seems that if per_page is > 100 it sets it back to 100, the solution is to implement pagination

Nhoya commented 6 years ago

The pagination for GitHub should work now, feel free to test it and leave a feedback :)

Please remember that according to GitHub doc, the rate limit for unauthenticated requests is 60 per hour.

P.S. BitBucket pagination is not yet implemented

Nhoya commented 6 years ago

Update: The pagination should work for BitBucket too

Note: Have in mind that the scraper on BitBucket will check for all the commits of all the branches so may take a while on big repos

Te-k commented 6 years ago

Works for me