search

karakanb / devo

A Chrome and Firefox extension that displays GitHub Trending, Hacker News, Product Hunt and Designer News on every new tab.
https://burakkarakan.com/devo/
MIT License
407 stars 47 forks source link

Add support for filtering based on languages and time range. #65

Closed aryan9600 closed 3 years ago

aryan9600 commented 3 years ago

First off, thanks for your work, this is an awesome extension! :D

It'd be nice if we could filter the GitHub trending repos based on the programming language and the time range, since GitHub does support it.

karakanb commented 3 years ago

Hey, thanks for the kind words.

It is something I've had in mind before as well, but the problem is that the trending information is not provided via an API, and I have a private ETL running somewhere that periodically fetches this page, parses it, builds the JSON, and stores it on DigitalOcean, which is what powers the extension for everyone. Providing this functionality would mean that I'd need to do the same operation for all the languages I'd like to provide, and for now, I think this would be too much of an effort to build and maintain. If there was an API for this then I'd update the extension to use the API directly in those cases, but there is none as far as I am aware.

I'll keep turning this idea in my mind to see if I can find an easy, quick, and cheap way of doing this, but for now, it seems unlikely. If you have ideas about how to get this from an API I am all ears.

pansila commented 3 years ago

There is a similar app I'm using, it has done the same ETL processes to extract the trending list per language, maybe we can use them.

gitter the api to get the trending list

export const getReposTrendingList = createApiAction(REPOSTRENDING, params => api.get('https://gitter-weapp.herokuapp.com/repositories', params))
export const getDevelopersTrendingList = createApiAction(DEVELOPERSTRENDING, params => api.get('https://gitter-weapp.herokuapp.com/developers', params))
export const getLanguageList = createApiAction(LANGUAGE, params => api.get('https://gitter-weapp.herokuapp.com/languages', params))
karakanb commented 3 years ago

In the case of Devo, I don't plan to depend on someone else's API to provide the basic functionality of the extension, and I don't see a strong enough value in the ability to filter languages and authors to even take the risk here. I should have said "from an official API" in my previous message, apologies.

I will close this issue until there is an easy API provided by GitHub that does this thing, and then I might consider maintaining the ETLs for this, but I am skeptical about it even if that existed, given that the same functionality exists properly on the GitHub Trending page and Devo includes a direct link to there on the page.

Feel free to re-open the issue it if something changes in GitHub API, and then I'll reconsider.

pansila commented 3 years ago

I'm wondering why can't we do ETLs in the browser? Just curious.

pansila commented 3 years ago

There is an unofficial but popular api. github-trending-api

You can deploy it on your API server if don't want to use their service.

Lots of projects building on it.

karakanb commented 3 years ago

Do you mean running ETLs as background scripts in the browsers as part of the extension? If that's what you mean, I'd rather not do that because the UIs change often, and when they change it is usually a breaking change, causing no one to be able to view Devo on their computers, and any such change would require me to build and push a new version, and everybody to update their extensions. What I do instead is to run ETLs on my own side and store the final file in the structure the extension expects; if anything breaks, I can simply fix the ETL on my infra and no one needs to do any other update. In addition to this, I also wouldn't want to run scripts like this on the users' devices, no. This also has no benefits in terms of my main concern, which is maintaining those ETLs.

An additional point in favor of using ETLs is that the cost and maintenance structure is way simpler with ETLs than real-life APIs. The first version of Devo was using Lambda, and my costs were increasing with the number of users, and I was constantly hitting rate limits of APIs and services due to that. By relying on ETLs, I have a fully predictable cost structure, my compute and data transfer rates are in free tiers of the cloud providers, and I don't need to care about hitting rate limits of any API. Therefore, even if GitHub provides an official API, I'd still need to build ETLs to manage listings for every language, and I simply don't see it worth the effort, given that everything is already available on GitHub Trending UI itself. Devo doesn't need it.

As per the other API suggestion, I am still of the same opinion: I will only rely on either the official APIs, or the things I build myself. I want Devo to exist for a long time, and this means relying on stuff that is reliable, or stuff that I can maintain by myself.