What are the needs/wants for data management and curation?

[MattF-NSIDC]

The demo currently reads data from Google Sheets, which is an approach we had considered for our MVP. Does this meet the needs of US AON, and if not, what can be improved?

E.g.:

• Do we want the ability to export to and import from a serialized (less human-manageable) format?
• Do we want the ability to export to and import from user-owned Google Sheets?
• Is a full in-app editing experience required?

[MattF-NSIDC]

With Google Sheets, I think we have these options for access from JS:

• Use Sheets API (requires auth by API key)
• Use "published" sheets data (File > Share > Publish to web). The demo does this but due to Google's CORS settings, requires a proxy. We can create our own proxy or use the same one as the demo (allorigins.win), though the latter may be objectionable from a security perspective.

[MattF-NSIDC]

I think the demo code warrants closer examination to better understand how it's handling CORS restrictions. I'm seeing cases where Google is sending the CORS header allowing origin http://localhost when running the demo through a webserver. When opening the demo as a raw HTML file, there is no "origin", so this doesn't work? 😕

[MattF-NSIDC]

Forgot to update. :point_up: that seems to be the case. CORS has not been an issue w/ Google Sheets' published data in my development environment.

This data is modeled like an RDB in the existing Google Sheets. Nodes have many links. Nodes have one type (column). Looks like there may be plans for subtypes of nodes. Each node type has a color. Links each have a weight and performance. There are also chart-level config values (e.g. filters).

This can be dumped to JSON and managed by hand, but it would be very easy to make errors in managing relationships by hand. This is also true of Google sheets, maybe to a lesser degree because of the visual "tables" layout. Maybe we can do things like make conditional formatting rules that highlight rows that violate referential integrity rules, but that sounds a little awful to maintain.

I see a relational DB as the end goal, but building that out means thinking about authentication and I don't think we have the time to add that to the pile. Have we considered building this visualization as an Airtable extension? Airtables is like "Google Sheets but relational", and it has a free tier. I don't know if that would meet our needs.

[MattF-NSIDC]

From AON:

• Too much room for error in Google sheets. It's difficult to calculate the weight from knowing the "performance" and "criticality". Currently performance determines line color.

[MattF-NSIDC]

AON wants to improve:

• The data-collection process is currently error-prone and time-consuming
• The process of transforming collected data into a visualizable dataset is error-prone and time-consuming