We want to integrate column and table metadata (e.g. text descriptions) into the source definition in pudl_catalog.yaml so that users can understand what data is available when browsing the catalog. This information is currently being written into the column and table metadata within the Parquet files during ETL, so it could be read from there. It could be exported from our Pydantic metadata models when we generate pudl_catalog.yaml.
[ ] Identify or create an appropriate structure / format for table & column level metadata in the pudl_catalog.yaml. This should include at least:
[ ] Text description of the table (Resource.description)
[ ] Primary key of the table (Resource.schema.primary_key)
[ ] Text descriptions for each column (Field.description)
[ ] Licensing terms for the data (Resource.license)
[ ] The original source of the data (Resource.sources)
[ ] Creator(s) / Maintainer(s) of the dataset (Resource.contributors)
[ ] Add a Resource.to_intake_data_source() method that can generate the Intake data source level metadata entry.
We want to integrate column and table metadata (e.g. text descriptions) into the source definition in
pudl_catalog.yamlso that users can understand what data is available when browsing the catalog. This information is currently being written into the column and table metadata within the Parquet files during ETL, so it could be read from there. It could be exported from our Pydantic metadata models when we generatepudl_catalog.yaml.pudl_catalog.yaml. This should include at least:Resource.to_intake_data_source()method that can generate the Intake data source levelmetadataentry.