search

Punderthings / fossfoundation

Directory of non-profit FOSS Foundations, with detailed metadata.
https://fossfoundation.info/
Apache License 2.0
19 stars 13 forks source link

Add scraper for numfocus sponsored projects #3

Closed andrew closed 1 year ago

andrew commented 1 year ago

I've added a scraper for NumFOCUS's sponsored projects.

I left the scraper code commented out as it looks like you had issues with nokogiri on GitHub pages, but I've added in a generated csv from the output of the code, feel free to uncomment and run locally to verify.

For projects with no website I've used the numfocus website page (https://numfocus.org/project/openfhe for example)

I wasn't sure on the wording on the numfocus subproject markdown page, feel free to change it.

Also didn't add https://numfocus.org/sponsored-projects/affiliated-projects, if you'd like that added as well let me know.

I also didn't commit the Gemfile.lock, although it'd be a good idea to add one (unless it's causing you issues with github pages)

ShaneCurcuru commented 1 year ago

Thanks, this is awesome! Yes, I haven't had time to figure out the cleanest way to get a zero maintenance environment - which I think will mean just sticking with whatever GH Pages currently supports. And ensuring that we can add enough Actions or the like to do linting on commits, have a periodic re-scraping that auto-generates a PR once a month, etc. Any tips or assistance in better setting up the environment appreciated (especially for someone as expert with this stuff as you!)

andrew commented 1 year ago

A GitHub action set up to run on a schedule (like cron) would be good for updating the csv files.

You could also use github actions to build the html site using this action: https://github.com/actions/jekyll-build-pages which would give a little more control over the environment that the site gets built in than regular github pages.

ShaneCurcuru commented 1 year ago

LOL, I keep thinking I can just make Actions work without actually understanding them, but no, you can't. Off to investigate, or to just hack a bad solution for the time being. https://github.com/Punderthings/fossfoundation/actions/runs/5058854157/jobs/9079550741

andrew commented 1 year ago

@ShaneCurcuru I took a brief look at the error in that action and I think I can fix/rework it, will take a look later.

That jekyll action I suggested seems to use quite a lot of fixed versions of dependencies, I can see why it got you into dependency hell! An easier approach might just be to have an action that generates and commits the html onto a branch, which then github pages can deploy without needing to install any extra dependencies.