visualizingthefuture / examples-repository

Repository for https://visualizingthefuture.github.io/examples-repository
Other
5 stars 2 forks source link

change processing of metadata .csv files to create lists out of some fields #63

Closed amzoss closed 3 years ago

cassws commented 3 years ago

What we learned

  1. wax will accept JSON for colleciton metadata 1b. HOWEVER, lists must use double quotes (which is a general JSON practice, might have been my fiddling that created unacceptable list)
  2. our process of Survey -> Google Sheet -> manual prep now should
    1. end in JSON
    2. safely parse list JSON-appropriate (using brackets and double quotes)
    3. we may think further about whether to accomplish this in sheet scripting, local scripting (eg helper python scripts), manual, etc.
  3. May wish to contribute back to wax addendum that we can use JSON to build out pages (wax pages will accept JSON, not specified in documentation)

Proposed alternative workflow

  1. Get CSV from Google Sheets to use a better delimiter (eg pipe delimited lists)
  2. Wax task that converts pipe delimited lists to JSON arrays a. As an issue, this might be framed as "csv currently not allowing expected metadata behavior of ingesting list of options. proposing "list-safe" options eg that look for delimiter (this task could even allow custom delimter). list-safe-ingest b. bc there's still a ruby tech mountain, start by writing a python script that we can run locally

Angela will work on pipe separation from google output

https://stackoverflow.com/questions/49248498/how-can-i-export-to-csv-with-pipe-delimiter

AND think about long-term parsing processing wiht multiple collections

Cass will work on Python script to generate JSON

For mid-long term, A+C will reconvene and consider standardizing this as 1 Google Sheet and 1 Wax Task (or something along these lines) and potential contribution as "sheets-parsing-tools"

Immediate outcomes

Generate two issues: google sheet script and python script

cassws commented 3 years ago

Note: may still wish to consider snytax such as LIST in header names to indicate where scripts should look for delimiters (as opposed to globally looking for pipes, eg)_

cassws commented 3 years ago

Pursuing as two separate tasks below so closing here! We can still use this for discussion if needed.