search

flattenthecurve / guide

https://www.flattenthecurve.com
Creative Commons Attribution 4.0 International
38 stars 33 forks source link

Prototype of script for pulling resources from spreadsheet to md files #241

Closed rousik closed 4 years ago

rousik commented 4 years ago

This script processes responses in the Resource responses and turns them into md files under _resources/. Few things to note:

  1. OAuth authentication upon first run is somewhat tedious. The access has not been "approved by google" so you need to click through the security warnings.
  2. Resources are generated into files with naming scheme YYYY-MM-DD-lowercased_title_of_the_resource.md. Spaces replaced with underscores and only first 25 characters of the resource name is retained.
  3. Attributes like country, category etc is taken verbatim from the spreadsheet, with the exception of United States being translated into USA
  4. There's functionality to use approved_by column (expected to be in the J column), but this is not enabled yet. We expect human reviewers to sanitize the data (esp. category) and then drop their name into that column. This functionality can be enabled by toggling ENABLE_VERIFICATION constant to True.
  5. Script still needs to be run manually.

Next step is to add Slack notification when new responses are added to the sheet.

rousik commented 4 years ago

Note that the output of this script is part of this pull request. We can retain the old resources files and remove it from this pull request if that's preferred.

mverzilli commented 4 years ago

Super cool! I'll need some time to review this. @matiasgarciaisaia if you want to lend a hand here, can you explore ways of triggering Jan's import script? There are two possible triggers: we can either trigger it each time a resource is posted to the spreadsheet, or simply have a "cron" somewhere take care of this daily.

rousik commented 4 years ago

OAuth tokens to access google spreadsheets that is used by this script may not be suitable to fully automated cron-like access, yet. I will look into whether there are alternatives to this. There may be some but google public api auth is a mess.

rousik commented 4 years ago

So for now I would suggest running this script by hand when new resources land. This might be an acceptable strategy while the incoming rate of resources remains low & we can invest into further automation once this no longer scales well.

rousik commented 4 years ago

I'm willing to be oncall for running this integration process in the interim.

rousik commented 4 years ago

Can we abandon this pull request and move the disussion over to https://github.com/flattenthecurve/guide/pull/248 which is copy of this? I have realized I shouldn't be working in the master branch of my fork and I'm trying to salvage my master branch now.