search

rust-lang / rust-repos

Dataset of Rust source code repositories
MIT License
96 stars 31 forks source link

Add GitLab scraping #20

Open pietroalbini opened 6 years ago

pietroalbini commented 6 years ago

Scraping GitLab repositories is required if we want to test them on Crater (and we do). This issue tracks the implementation of the scraper.

The API calls we would need to make to scrape are:

This is currently blocked on:

ecstatic-morse commented 5 years ago

Gitlab CE now supports filtering /projects by programming language. I'm not sure when this will be deployed to gitlab.com.

However, Gitlab uses OFFSET and LIMIT to paginate the results of API calls, which means fetching the last page requires scanning the entire database. On my machine, fetching a page near the end of the database with https://gitlab.com/api/v4/projects?per_page=100&page=9000 takes about 14.5 seconds. If we assume that 10 seconds of that was spent in the database, fetching every project would take ~12.5 hours of database time.

Crater's use-case won't be quite that bad. If 5% of projects on Gitlab contain Rust, we'll need only 37.5 minutes of database time to scrape. If the result is cached, perhaps it's feasible to start running crater on Gitlab projects, but it's probably better to wait until keyset-based pagination is re-enabled.

pietroalbini commented 5 years ago

Gitlab CE now supports filtering /projects by programming language. I'm not sure when this will be deployed to gitlab.com.

That's great! Thanks for your work on this :heart:

Crater's use-case won't be quite that bad. If 5% of projects on Gitlab contain Rust, we'll need only 37.5 minutes of database time to scrape. If the result is cached, perhaps it's feasible to start running crater on Gitlab projects, but it's probably better to wait until keyset-based pagination is re-enabled.

Yeah, our results are cached, so it wouldn't that bad. If you ping me after the API is deployed on GitLab.com I can look into it, and if the performance is still bad I'll contact GitLab support to make sure we're not hurting them with the scrape.

ecstatic-morse commented 5 years ago

Will do!

ecstatic-morse commented 5 years ago

Looks like this will be deployed as part of the 11.8 release on February 22.

mathstuf commented 5 years ago

If needed, we can add API calls to the gitlab crate to support this. We already support depagination too.

mathstuf commented 5 years ago

There's also the question of which instances to scrape. gitlab.com is obvious, but the GNOME and freedesktop.org instances should probably also be on the list (at least).