latitude-dev / latitude

Developer-first embedded analytics
https://latitude.so
GNU Lesser General Public License v3.0
719 stars 30 forks source link

Feature: compiler can read metadata without resolving values #434

Closed csansoon closed 3 weeks ago

csansoon commented 1 month ago

Describe your changes

Some procedures require extracting metadata from the query, like the defined configuration or knowing which functions are being used, without actually having to resolve the query.

The current way to do this is just compiling the query, which will return some additional metadata along with other compiled output. The problem with this approach is that compiling a query is not always free, as some queries may use runQuery. This function is executed in compile time, since we need the result to be able to resolve other parts of the query. Since we need to get some of the query metadata instantly in some cases, we cannot depend on running queries just to read its configs.

To fix this issue, I have divided the current compilation process in two independent functions: compile and readMetadata:

In order to read something as "metadata", it must be static and not depend on other values, parameters or functions resolved during run-time. Since I want the config to be part of the query metadata, I had to add these restrictions to the {@config} tag:

Checklist before requesting a review


πŸš‚ TRENECITO. πŸš‰ Previous station: #439 πŸš‰ Next station: #444

changeset-bot[bot] commented 1 month ago

πŸ¦‹ Changeset detected

Latest commit: 40017c39b6d6aeb8c3dc94c3406512daa86c4104

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages | Name | Type | | ------------------------------------- | ----- | | @latitude-data/source-manager | Minor | | @latitude-data/sql-compiler | Major | | @latitude-data/server | Patch | | @latitude-data/cli | Patch | | @latitude-data/athena-connector | Patch | | @latitude-data/bigquery-connector | Patch | | @latitude-data/clickhouse-connector | Patch | | @latitude-data/databricks-connector | Patch | | @latitude-data/duckdb-connector | Patch | | @latitude-data/materialized-connector | Patch | | @latitude-data/mssql-connector | Patch | | @latitude-data/mysql-connector | Patch | | @latitude-data/postgresql-connector | Patch | | @latitude-data/snowflake-connector | Patch | | @latitude-data/sqlite-connector | Patch | | @latitude-data/test-connector | Patch | | @latitude-data/trino-connector | Patch |

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR