This repo contains a GitHub scraper I wrote for my research project on civic hacking. At the moment it is very much focused on organizations and their members, i.e. it only takes a list or organizations as its input, not normal user accounts. It generates CSV spreadsheets or GEXF files to be used in network analysis software like Gephi. I tested it with Python 2.7.10 and 3.5.0, but I recommend using Python 3.
The scraper can do the following:
We recommend using pipenv
to keep your work isolated. (These steps are
noted and optional.)
cd github-scraper
pipenv install #pipenv
# Run this each time you start a new terminal session
pipenv shell
config.json
and add your GitHub user name and your personal access token to access the GitHub API.organizations.csv
and add the user account names of the organizations you want to scrape -- one organization per line! For example, if you want to scrape mySociety, Open Knowledge, and Ushahidi, your list will look like this:github_org,tags
mysociety
okfn
ushahidi
python github_scaper.py
and choose an option. You can perform several scrapes in one run by entering several numbers separated by commas (, ). The results will be stored in the data
subfolder.If you would like to skip archived repos, set envvar to
GHSCRAPER_SKIP_ARCHIVED=1
.