Add dagster to documentation

[bendnorman]

Dagster introduces a lot of new concepts and code to PUDL. Make sure we continue the tradition of super rad documentation.

• [x] Release notes
• [x] Overview of dagster concepts: Resources, ops, jobs, config. I think readers should have a basic understand of the concepts and how they relate to PUDL, we don't need to rewrite the dagster docs.
• [x] Update Creating a Workspace to include the env vars. Should we describe the env vars as the main way to manage the workspace with a note that you can also use pudl_setup but it will be deprecated soon? What warning will be produced if there is no pudl_settings or env var available? Include DAGSTER_HOME var.
• [x] Running the ETL pipeline page will need to be completely rewritten.
  • [ ] New yaml file structure (ConfigSchemas should be explained)
  • [ ] CLI vs Dagit
  • [ ] What happens when you run pudl_etl with just eia860? I'd expect a dagster dependency error?
• [x] Update the development set up page.
  • [ ] Set DAGSTER_HOME variable
• [x] Update Working with the Datastore page if the pudl_datastore command is deprecated
• [x] Update Cloning the Ferc Form 1 DB page to reflect any CLI changes.
• [ ] How to reload assets in dagit.
• [ ] how to configure dagster assets and resources
• [ ] Delete database when you create a schema change
• [ ] How to add new assets
• [ ] warning about assets persisting between branches
• [x] Explain how to setup PUDL_OUTPUTS and PUDL_CACHE if you have an existing pudl workspace setup. Does pudl know to create an output directory and use the existing pudl-work/data/ for PUDL_CACHE? Add a note about the env vars needing to be set each time you rebuild the environment.
• [x] Add links to the IO Managers we use.
• [x] At note about how PUDL_CACHE can get big.
• [ ] Change PUDL_CACHE -> PUDL_ZENODO_CACHE and PUDL_OUTPUT -> PUDL_OUTPUTs
• [x] Include link to datastore resource in Resources section.
• [x] Add example of a pudl job.
• [x] Update ferc_to_sqlite language to include non form 1 forms.
• [x] Move dagster deprecations into deprecations section.
• [x] Add links to the dagster definitions.
• [ ] How to deploy settings files?
• [ ] Update the ETL Design page with information on how to add new data sources using assets
• [ ] Create a Jupyter Notebook example of how to access assets. #2356

[jdangerx]

We can split this into multiple PRs - just update the checklist with which categories are spoken for.

[jdangerx]

@bendnorman will flesh out some of the bullet points above as well.

[bendnorman]

Here are the documentation changes I'm planning on making:

Running the ETL

• Start with an introduction to dagster. What it is and the problems it solves. Not planning on going into detail on the concepts.
• describe the two main jobs
  • mostly use the existing language
• How to run the ETL
  • set DAGSTER_HOME env var
  • use dagit
    • how to configure the datasets
    • how to configure the resources
    • How to run portions of the ETL using dagit
  • can also use the cli commands
    • borrow existing language

Cloning FERC Form1 DB

• how to clone using dagit
• how to clone using the cli

Release Notes

• Everything that has changed!

We could also add dagster concept explanations and uses to the Data and ETL Design Guidelines section though we could punt this part of the docs once dagster-asset-etl is merged in. @catalyst-cooperative/inframundo what do you think?

[zschira]

I think what you've laid out here seems like a good minimal set of docs updates before merging to dev. I think detailed explanations of dagster concepts can wait, as long as there's sufficient info to get people up and running with the new architecture. I also think doing a live run through with our team once we merge to dev will be helpful while we continue to flesh out the docs.