It should be possible to declare external tables that your project depends on, but are not generated by the dataform project. For example, if I automatically push my web logs to BigQuery to the table weblogs.logs then in my dataform project I can create a file:
definitions/external_tables.js:
declare("weblogs.logs");
And reference them in other queries, e.g. definitions/example.sql:
select * from ${ref("weblogs.logs")}
This would allow us to show external tables as part of the DAG, and would be useful for debugging, as well as simplifying queries (in BigQuery).
[ ] Add a new proto type Declaration
[ ] Add declarations to CompiledGraph proto
[ ] Make it possible to ref() a declaration
[ ] Implement a Declaration class similar to Materialization that just stores a Target which is derived from the provided name
[ ] Make sure declarations complete immediately as a no-op during execution of the graph
BenBirt
commented
5 years ago
It should be possible to declare external tables that your project depends on, but are not generated by the dataform project. For example, if I automatically push my web logs to BigQuery to the table
weblogs.logsthen in my dataform project I can create a file:definitions/external_tables.js:And reference them in other queries, e.g.
definitions/example.sql:This would allow us to show external tables as part of the DAG, and would be useful for debugging, as well as simplifying queries (in BigQuery).
DeclarationdeclarationstoCompiledGraphprotoref()a declarationDeclarationclass similar toMaterializationthat just stores aTargetwhich is derived from the provided name