At the moment we are manually tweaking the dbt manifest.json files produced by CaDeT to reduce the scale of the ingestion job for the dev environment (and potentially test) to allow more rapid iteration for testing.
It would be preferable if this step was automated and reproducible e.g. via GitHub Actions.
Requirements:
[ ] Create reproducible script (e.g. using jq) to prune the manifest.json file
[ ] Create GitHub Action workflow to run the script, outputting the reduced manifest file
[ ] GitHub Action will need to assume a role with suitable permissions to allow fetching / depositing files within CaDeT S3 bucket path
[ ] Coordinate with CaDeT the location / name of the reduced file if this will be output into their CaDeT bucket
At the moment we are manually tweaking the dbt
manifest.jsonfiles produced by CaDeT to reduce the scale of the ingestion job for thedevenvironment (and potentiallytest) to allow more rapid iteration for testing.It would be preferable if this step was automated and reproducible e.g. via GitHub Actions.
Requirements:
jq) to prune themanifest.jsonfile