In order to support the Metrics Calculation Service's workflow of writing direct to gcs it's calculated metrics, we need to have a different sqlmesh materialization than currently available for trino.
During the factories model generation, if the current engine is “trino” then we will use this new custom materialization. This custom materialization has this general process:
Receives a generated sql query from the model
Generates a temporary destination path in gcs to receive parquet files
This should include a date and be something like {bucket}/{some_static_temp_path}/{date}/{random_id}. This will ensure that we can clean any files that may be left over from runs that don’t complete successfully.
Submits the query to metrics calculation server
Polls for the calculation job to be completed every 5 seconds
Triggers a trino query to import the parquet files into iceberg storage
Deletes the temporary destination path in gcs
NOTE: If using duckdb we should just use the current workflow as that will work without issue and doesn't require the metrics calculation service
What is it?
In order to support the Metrics Calculation Service's workflow of writing direct to gcs it's calculated metrics, we need to have a different sqlmesh materialization than currently available for trino.
During the factories model generation, if the current engine is “trino” then we will use this new custom materialization. This custom materialization has this general process:
{bucket}/{some_static_temp_path}/{date}/{random_id}. This will ensure that we can clean any files that may be left over from runs that don’t complete successfully.NOTE: If using duckdb we should just use the current workflow as that will work without issue and doesn't require the metrics calculation service