Audit AssetMaterializations in integrations

[schrockn]

We should make all the AssetMaterializations in our integrations end up producing sensible results in the asset catalog. E.g.:

@dagster_type_materializer(
    Selector(
        {
            "csv": {
                "path": StringSource,
                "sep": Field(StringSource, is_required=False, default_value=","),
            },
            "parquet": {"path": StringSource},
            "table": {"path": StringSource},
            "pickle": {"path": StringSource},
        },
    )
)
def dataframe_materializer(_context, config, pandas_df):
    check.inst_param(pandas_df, "pandas_df", pd.DataFrame)
    file_type, file_options = list(config.items())[0]

    if file_type == "csv":
        path = file_options["path"]
        pandas_df.to_csv(path, index=False, **dict_without_keys(file_options, "path"))
    elif file_type == "parquet":
        pandas_df.to_parquet(file_options["path"])
    elif file_type == "table":
        pandas_df.to_csv(file_options["path"], sep="\t", index=False)
    elif file_type == "pickle":
        pandas_df.to_pickle(file_options["path"])
    else:
        check.failed("Unsupported file_type {file_type}".format(file_type=file_type))

    return AssetMaterialization.file(file_options["path"])

in dagster_pandas

I'm actually not sure how, at this point, this asset materialization appears in the asset catalog but it probably doesn't make sense.

[sryza]

AssetMaterializations in integrations:

• dagstermill yields an AssetMaterialization for the notebook it produces
• dagster-pandas dagster_type_materializer yields an AssetMaterialization with the file path
• dagster-pyspark dagster_type_materializer yields an AssetMaterialization with the file path
• dagster-dask dagster_type_materializer yields an AssetMaterialization with the file path
• dagster-aws file_handle_to_s3 yields an AssetMaterialization with the file path
• lakehouse yields an AssetMaterialization with lakehouse asset name
• dagster-dbt solids yield AssetMaterializations with keys that are basically the name of the solid

Starter thoughts on what to do with these:

• Deprecate the materializers for pandas, pyspark, and dask, as well as file_handle_to_s3 in dagster-aws. We can't offer materializers that capture the full space of how users might want to materialize these objects. Make sure it's easy for users to substitute IO managers.
• For dagstermill, let users provide an asset key function.
• Leave lakehouse how it is - the keys are already fully user-pluggable.
• For dagster-dbt, expose some way of logging structured metadata without requiring an entry in the asset catalog, as that's what these materializations are used for. Also, for dagster-dbt solids that run models and actually mutate external assets, consider adding AssetMaterializations to represent this with user-provided asset key function.

[sryza]

Going to close this because our asset-related integrations now focus on software-defined assets instead of dynamic AssetMaterializations. Since this issue was filed, we added load_assets_from_dbt_project, define_dagstermill_asset, SDA-compatible IO managers for Pandas/PySpark/BigQuery/Snowflake/DuckDB, and made our S3/GCP/Azure IO managers SDA-compatible.