Use the Github search API

[volkanunsal]

We need to use the Github API, rather than managing the data in our own database. The app needs to be totally rewritten to run without a backend.

[ameensol]

This is a bit tricky. If we're doing that, how would we, for example, keep track of which projects are currently active?

[volkanunsal]

"Last updated" date on the repo. And also we're going to be looking at the contents of the civic.json, and I believe there is a field for "project status."

[ameensol]

Right. I mean, how would we keep track of which projects total we are tracking? I think we'd need some sort of file or db that we maintain ourselves.

[volkanunsal]

For sure. I'd be interested in blacklisting, rather than whitelisting. We track everything by default, and filter out repos if they are spammers or trolls –– using a file, not a database.

[ameensol]

I agree that using a file is simpler than using a database. Right now, besides locally storing data for each project so it's a bit faster to render, we are only storing the metadata—which projects to track—and soon the betaNYC designation for each project (Community, Official, Featured). This could just be a JSON file.

I think it might be easier to whitelist though, using a file. If we blacklist we'd have to scrape all of github to find the projects in the first place. If we whitelist, we can go to the project repo's directly.

[volkanunsal]

Yeah, it has to be a JSON file. The whole idea is to make this a portal for discovering projects that have a civic.json file.

We are not going to scrape anything. We're going to use the Github Search API.

[ameensol]

Ah, that makes sense.