Workflow for table steps

[larsyencken]

Context

Shelf is meant to be a productive small-scale ETL framework. To be productive, it needs a nice workflow for:

• Taking snapshots of upstream data
• Massaging those snapshots into standard formats
• Cleaning and harmonising data into interoperable form

Currently shelf add --edit is a productive way of adding a snapshot to the content store.

Problem

• There is no nice way of adding a new table to the table store
• There is no guided iteration story on interactively creating data steps
• There are no guidelines on how to name derivative steps (e.g. owid/etl uses meadow and garden for formatting and harmonisation steps, respectively)

Technical notes

• Jupyter notebooks have a great interactive story, perhaps we can generate an interactive notebook and load it in edit mode (or else, a Marimo notebook)
• For Nushell, perhaps a streamlit app that lets you write Nushell at the top and preview the output below would be helpful for writing Nushell steps

[larsyencken]

I'm wondering if shelf add should become shelf snapshot, and perhaps shelf derive <dep> [<dep> [...]] would be a good interactive helper for creating a new table step.

[larsyencken]

Currently we have shelf new-table <qualified_name> <deps>. But now that we have both Python and SQL steps, we don't create a step template automatically.

It is working to provide a version-independent step definition though.