search

flattenthecurve / guide

https://www.flattenthecurve.com
Creative Commons Attribution 4.0 International
38 stars 33 forks source link

Resource automation #248

Closed rousik closed 4 years ago

rousik commented 4 years ago

This script processes responses in the Resource responses and turns them into md files under _resources/. Few things to note:

  1. OAuth authentication upon first run is somewhat tedious. The access has not been "approved by google" so you need to click through the security warnings. Resources are generated into files with naming scheme YYYY-MM-DD-lowercased_title_of_the_resource.md. Spaces replaced with underscores and only first 25 characters of the resource name is retained.

  2. Attributes like country, category etc is taken verbatim from the spreadsheet, with the exception of United States being translated into USA

  3. There's functionality to use approved_by column (expected to be in the J column), but this is not enabled yet. We expect human reviewers to sanitize the data (esp. category) and then drop their name into that column. This functionality can be enabled by toggling ENABLE_VERIFICATION constant to True.

  4. Script still needs to be run manually. I don't think OAuth tokens are suitable for cron-like automation. As discussed on the other thread the volume of Resources is still low so relying on human ops may be appropriate here.

rousik commented 4 years ago

@mverzilli and @matiasgarciaisaia, would you mind taking a look and checking if this looks good?

mverzilli commented 4 years ago

Looks great! Just wondering about the credentials file, can you explain why it needs to be in the repo?

rousik commented 4 years ago

PTAL. I have added "Approved Resources" tab to the original spreadsheet and published that tab to web. This means that we can fetch that as CSV w/o needing any authentication. This makes the import script much nicer and simpler. This also gives us ability to sanitize the resource texts as we wish, which means that less sanitization is needed in the code.

I have also reverted changes to _resources to make it simple.

It should be possible to run this script in a cron-like fashion.

rousik commented 4 years ago

The procedure for absorbing and approving new Resources should be documented somewhere, probably at https://github.com/flattenthecurve/guide/wiki/Processes. We can link this with the slack message to #resources channel to make it smooth.

rousik commented 4 years ago

@mverzilli and @matiasgarciaisaia , should be good to go now.