mozilla-rally / rally-sdk

This is the Rally partner support library, `rally.js`, used to manage the lifecycle of Rally studies.
0 stars 4 forks source link

Discussion: Data Exploration Tools #10

Open knowtheory opened 3 years ago

knowtheory commented 3 years ago

@hamilton & I talked last week about the tools needed to prototype extensions.

Every extension project goes through an exploration phase where study authors evaluate their intended data collection. The exploration phase almost always consists of creating some individual measurement probes, figuring out a structure to store the measurements into a local database, and then some way to read or dump the stored data back out to assess whether the data collection functions as intended for the analysis needs of the authors.

Some of these are going to be common capabilities shared across different extensions, and it will be worth our while to document the patterns, as well as deciding whether we'd like to turn those patterns into libraries to aid project development.

Specifically, we'd like to raise:

Data storage patterns

Probes produce data that need to go into an event log. The details of those events and how they're keyed is going to be unique to each project, however the ergonomics of how to get data into a database, and how to get the data back out (for example an options page where the playtester can export/download their collected data).

Could we build some svelte components to make exporting/dumping data trivial?

Mini analysis env

Since the data is stored locally, one question we have is whether it might just be good to plug some analysis tools directly into this SDK which can be used at development/playtesting time, to tighten the development/analysis loop. Without additional tools users will need to periodically dump data (as JSON or the like) import it into another data store and analyze them.

Could we hook up jupyter lite or the equivalent directly into an extension?