Scrap github org - Githubissues

Solution:

To scrap Github org, we can use the Github API to retrieve the necessary information. We can start by creating a personal access token on Github and using it to authenticate our requests to the API. Then, we can use the API endpoints to retrieve information about the organization, repositories, commits, authors, languages, and package dependencies. Once we have all the necessary information, we can write it to a CSV file using Python's CSV module or any other preferred method.

We can create a new file called dataset.csv with the necessary columns to store the information retrieved from the Github API. The columns should include commit dates, sizes, authors, languages, README/LICENSE, and package dependencies. Once the necessary columns are created, the data can be written to the file using Python's CSV module or any other preferred method.

To implement this solution, we can use the scrap_github_org.py file. We can start by importing the necessary libraries and creating a personal access token on Github. Then, we can use the Github API endpoints to retrieve information about the organization, repositories, commits, authors, languages, and package dependencies. We can handle errors and exceptions appropriately and respect Github's API rate limits.

Here's an example code snippet to get started:

import requests
import csv

# create personal access token on Github
token = 'your_personal_access_token'

# set headers for authentication
headers = {'Authorization': f'token {token}'}

# retrieve information about the organization
org_name = 'your_organization_name'
org_url = f'https://api.github.com/orgs/{org_name}'
org_response = requests.get(org_url, headers=headers)

# retrieve information about the repositories
repos_url = f'https://api.github.com/orgs/{org_name}/repos'
repos_response = requests.get(repos_url, headers=headers)

# retrieve information about the commits, authors, languages, and package dependencies
# for each repository
for repo in repos_response.json():
    commits_url = f'https://api.github.com/repos/{org_name}/{repo["name"]}/commits'
    commits_response = requests.get(commits_url, headers=headers)
    # retrieve other necessary information and write to dataset.csv
    # using Python's CSV module or any other preferred method

We can modify the code snippet to retrieve the necessary information and write it to dataset.csv. We can also add error handling and respect Github's API rate limits.

Changes on dataset.csv

As this is a new file, the plan is to create a CSV file with the necessary columns to store the information retrieved from the Github API. The columns should include commit dates, sizes, authors, languages, README/LICENSE, and package dependencies. Once the necessary columns are created, the data can be written to the file using Python's CSV module or any other preferred method.

Changes on scrap_github_org.py

To solve the task, you can use the Github API to retrieve the necessary information. You can start by creating a personal access token on Github and using it to authenticate your requests to the API. Then, you can use the API endpoints to retrieve information about the organization, repositories, commits, authors, languages, and package dependencies. Once you have all the necessary information, you can write it to a CSV file using a library like pandas. Remember to handle errors and exceptions appropriately and to respect Github's API rate limits.

ClickerAI / GithibRepo2ProjectDataset

Scrap github org #1