DataJunction / dj

A metrics platform.
http://datajunction.io
MIT License
34 stars 15 forks source link

Generate notebook for exporting cubes / namespaces #1070

Closed shangyian closed 3 months ago

shangyian commented 3 months ago

Summary

This PR adds a feature to generate a notebook for exporting cubes and namespaces. For cubes, the generated notebook will include all nodes referenced by the cube, i.e., all metrics and their upstream transforms. By default this export excludes sources and dimensions, but these can be included with a flag. For namespaces, the generated notebook includes all nodes in the namespace.

In the exported notebook, users can modify NAMESPACE_MAPPING to map current namespaces to new ones and rerun the notebook to recreate the nodes in those new namespaces. For each exported node with a query, we will modify the query to replace the node names with the namespace mapping as an f-string. This does mean that queries with comments will have the comments stripped in the exported versions.

These changes also switch to using jinja templates for the generated client code snippets. This makes it easier to read, as generating them through string manipulation was difficult to maintain.

Test Plan

Ran locally

Deployment Plan

N/A

netlify[bot] commented 3 months ago

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
Latest commit c88f1af2eba1278b63c5b19e1436c9caf8f360f9
Latest deploy log https://app.netlify.com/sites/thriving-cassata-78ae72/deploys/667a2703af6785000822e5a7
shangyian commented 3 months ago

@agorajek and @samredai going to merge this now to get it in, but feel free to leave comments and I'll address them in a follow-up PR