Document the data used to generate the visualization

[patcon]

The pca key of the participationInit endpoint is a giant escaped JSON string of all the data that renders to client-participation visualization. (This data is sometimes referred to in codebase as "preload data" for a conversation.)

It comes from the math_main database table, where there's a data column with this JSON string. The math worker populates this field with all its processed data for a conversation, and then the server later sends it to client when initializing or updating a convo.

We should document the structure of this data, since it's important to understand if someone wishes to work on the visualization.

To Do Candidates

• [ ] make a gist of sample data from https://pol.is/3ntrtcehas
• [ ] maybe: convert PCA JSON string to object before sending from server
• [ ] low priority: convert away from dataframe paradigm
  • premise: use the more readable "array of objects", rather than "object of arrays" of dataframes
• [ ] improve documentation of data
  • [ ] pick @metasoarous's brain sometime and write it down (will get stale)
  • [ ] investigate generating docs from inline (using closure.spec? something else?)

[patcon]

Here are some rough notes from a conversation with @metasoarous about the pca data and it's origins in math subcomponent:

• pca key in conversation response obj, preload data
  • pca key is escaped string of json
• looked to R data frame paradigm: pandas.
  • only works if whole library works that way. so bad design pattern outside pandas.
• organized by column
• everything resigned around arrays in R
• one object is vector of one frame
• structures can have row names
• original idea of team polis: "let's copy that!" (assuming we're going to have big data scale for pol.is).
• chose to use zippable arrays instead of list of objects. but data compression means this doesn't really matter
• "dict of arrays" instead of "array of dicts"