search

zmon / Code-for-America-Projects-Hub

Help Brigade members find projects to work on. Promote multi-Brigade collaboration. Onboard new members. Encourage public contribution of ideas. Karma. Visualize scope for Code for America. Provide filtered and sorted card list of projects and ideas for development. The list should be available nation-wide and by city.
http://codeforkc.org/Code-for-America-Projects-Hub/
15 stars 13 forks source link

Integrate Github API for Projects and map the API data to a unified Project schema. #61

Open kreeves opened 9 years ago

kreeves commented 9 years ago

Pull from Github the following information and make it simple to conform with final front-end JSON model.

kreeves commented 9 years ago

Perhaps we could open up the table for a bit of discussion about the API usage. I do have one concern related to it. We may run into an issue with the number of API calls that are required per page load.

For each project, we'll need to grab it's initial project information as well as all of it's issues. That's two calls to the API for each project. From what I can gather, there's a limit of 5000 calls to the API. At some time interval, that number gets refreshed.

There is a search API available. https://developer.github.com/v3/search/ I'm trying to see if we can somehow leverage that to limit the number of API calls.

I imagine it's also possible this could be an issue with the CFA API.

Thoughts? Am I over-thinking this concern? :)

rlh-aagis commented 9 years ago

I agree with the need to discuss as I too have concerns - they arose from pulling the initial brigade location data for the map using the CfA API, although I admit I am still researching this from a Leaflet Map API perspective...

Paul, can we discuss this next meeting? On Apr 7, 2015 9:02 AM, "kreeves" notifications@github.com wrote:

Perhaps we could open up the table for a bit of discussion about the API usage. I do have one concern related to it. We may run into an issue with the number of API calls that are required per page load.

For each project, we'll need to grab it's initial project information as well as all of it's issues. That's two calls to the API for each project. From what I can gather, there's a limit of 5000 calls to the API. At some time interval, that number gets refreshed.

There is a search API available. https://developer.github.com/v3/search/ I'm trying to see if we can somehow leverage that to limit the number of API calls.

I imagine it's also possible this could be an issue with the CFA API.

Thoughts? Am I over-thinking this concern? :)

— Reply to this email directly or view it on GitHub https://github.com/codeforkansascity/Code-for-America-Projects-Hub/issues/61#issuecomment-90568351 .

jstirnaman commented 9 years ago

Angular's $http service has built-in caching. We haven't configured it yet in development. Helpful overview: https://www.ng-book.com/p/Caching/ I think this will partially address the concerns.

jstirnaman commented 9 years ago

ProjectsService now pulls data from CFA and Github. Merging is simplistic, but it's a start. Still need to come up with a final attribute mapping/schema and update the views to match. Commit 911fc2071bf28c103cf4e6c672bf66f7505f5401

jstirnaman commented 9 years ago

The Github API returns alot of data for repos. Can we be more selective with the API?

jstirnaman commented 9 years ago

Closed by mistake.

jstirnaman commented 9 years ago

I don't see any way in the documentation to select only a subset of attributes from the API response. What is returned for /repos is already a subset of repo details.

jstirnaman commented 9 years ago

We can get contributors and issues as separate API calls if those are all we need from Github. The downside is that makes our app a little more complex.

jstirnaman commented 9 years ago

@aertoria @zmon I think we'll need to fetch issues for each Github repo (Project) through a separate API call. Since the Google Spreadsheet is the authority, it may be best to only fetch issues and contributors through separate API calls for each project in the sheet. However, that's a fairly loose linkage between the URLs in the spreadsheet and CFA API and the Github API URLs.

jstirnaman commented 9 years ago

@zmon @tangospring Is it enough to fetch all the labels for each project repo, i.e. using https://developer.github.com/v3/issues/labels/#list-all-labels-for-this-repository? Do we need any more info than that?

jstirnaman commented 9 years ago

Nevermind. I see from #17 that we'll need to return labels for only open issues. I think that means we'll have to use the /issues/ endpoint anyway.