search

civictechindex / brigade-project-index

Brigade Project Index
https://brigade.cloud/
12 stars 19 forks source link

Read Democracy Lab #56

Open giosce opened 3 years ago

giosce commented 3 years ago

DemocracyLab has provided an API to fetch projects that they host. Mark Frischmuth told us is fine to call https://www.democracylab.org/api/projects every hour.

themightychris commented 3 years ago

Steps to implement this:

  1. Create a new "project repository" class here: https://github.com/codeforamerica/brigade-project-index/tree/master/crawler/lib/repositories/projects
    • It should extend the Projects abstract class like its peers File, GitHubTopic, and GitHubOrganization do
    • Each of these classes is ultimately a superclass of Map, and when loaded should be a map of ProjectId -> ProjectData where ProjectData is just an unstructured JSON data object
  2. Register that new repository here: https://github.com/codeforamerica/brigade-project-index/blob/master/crawler/lib/repositories/projects/index.js
    • This registration list is in priority order, the first class that returns true when its canLoadFromOrganization method is passed an entry from organizations.json is then in charge of loading a list of projects for that entry
  3. We'll want to PR a new entry for DemocracyLab into organizations.json and add the DemocracyLab projects repository class to the top position in the registry and hardcode it to just detect the democracylab entry and handle loading it. We can check that organization.projects_list_url == 'https://www.democracylab.org/api/projects'
  4. There are tests for each of these project repositories in here that (kind of unstably) check against some known data in the respective live APIs, add tests for this new class: https://github.com/codeforamerica/brigade-project-index/tree/master/crawler/lib/repositories/projects/__tests__

These docs cover how to test/debug changes like this locally: https://brigade.cloud/projects/crawler/contributing/crawler/