This is inspired by first-timers-only
issues:
How can we make it possible for new-comers to contribute to a project.
This web service tries to solve this by looking at the data:
To find first timer pull-requests, we look at organizations and their repositories.
What are first timer repositories?
What is a first-timer pull-request?
What is a first-timer issue? There is no automatic linking between the pull-request and the issue. Thus, we must assume that one of the following takes place:
For each organization submitted:
For each submitted repository
ENDING
is either .html
or .json
.
GET /
GET /source
GET /auth
Show a form to register a new authentication.POST /auth
Add username
and password
to those usable to scrape GitHub.
They will be tried and removed if invalid.GET /organizations<ENDING>
offset=0&limit=10
GET /organization/<organization><ENDING>
/organizations.json
Ending:json
html
svg
POST /organization<ENDING>
organization
for scraping.
This shows an html page with a link to the status of the organization.GET /repositories<ENDING>
GET /repository/<organization>/<repository><ENDING>
json
html
svg
POST /repository
repository
for scrapingGET /users<ENDING>
GET /organizations<ENDING>
.GET /user/<user><ENDING>
GET /organization/<user><ENDING>
.POST /user/<user>
When objects are defined, they contain minimal definitions.
They can be used to infer the most important data and find the full data.
E.g. repository["urls"]["json"]
always points to the repository endpoint.
repository
, user
, organization
and issue
have this in common:
{
"name": "<name>",
"urls": {
"html": "<html_url>",
"json": "<json_url>",
"github_html": "<github_html_url>",
"github_api": "<github_json_url>",
},
"last_update": "<start_time>",
}
}
last_update_time
times are given like this:
2011-01-26T19:01:12Z
in UTC.
You can parse it with
time.strptime("2011-01-26T19:01:12Z", "%Y-%m-%dT%H:%M:%SZ")
Additionally repository
, issue
always define
{
"full_name": "<full_name>"
}
python3 -m first_timer_scraper <CACHE_FOLDER> <SECRETS_FOLDER> <MODEL_FOLDER>
CACHE_FOLDER
is the folder where the scraped data is stored.
This is the cache. It is totally ok to remove all this data.SECRETS_FOLDER
is the folder where the secrets are stored.
These are the secrets to access the GitHub API.MODEL_FOLDER
is the storage place of the model/the data base this is built.
If you delete this, you can start scraping anew.You need to install Python 3 and pip. Under Ubuntu, you can do this:
sudo apt-get -y install python3 python3-pip
To install all required packages, execute
pip3 install --user -r requirements.txt
py -3 -m pip install --user -r requirements.txt
py -3 -m first_timer_scraper.app data secret model
This runs the docker container:
docker run --rm \
-p 8080:8080 \
-v "secret:/app/secret" \
-v "model:/app/model" \
niccokunzmann/first_timer_scraper
The parameters have the following meaning:
./secret
../model
--rm
is for development purposes.
It removes the container and all the cache when the container stops.When you ran the command, you can visit http://localhost:8080, submit credentials and scrape repositories.
You can build the Docker image like this:
docker build . -t niccokunzmann/first_timer_scraper
Deployment takes up to 24 hours from a merge into master:
The data model describes what is saved when scraped.
{
"loklak": {
"repos": {
"loklak_server": {
"first_timer_prs": {
"112": "contributor1"
},
"last_update_requested": "2011-01-26T19:01:12Z"
}
},
"last_update_requested": "2011-01-26T19:01:12Z",
"first_timer_prs":{}
},
"contributor1": {
"first_timer_prs": {
"loklak/loklak_server": {
"created_at" : "2011-01-26T19:01:12Z"
"number": 112 // lowest number wins
"last_update_requested": "2011-01-26T19:01:12Z"
}
},
"last_update_requested": "2011-01-26T19:01:12Z",
"repos": {}
}
}