TobikoData / sqlmesh

Efficient data transformation and modeling framework that is backwards compatible with dbt.
https://sqlmesh.com
Apache License 2.0
1.81k stars 160 forks source link

global variables don't pass to python models when python model defined with kw-only syntax #3345

Closed Ben-Epstein closed 6 days ago

Ben-Epstein commented 6 days ago

With a model defined as


@model(
    "example.foo",
    columns={
        "ts": "datetime",
        "id": "int",
        "foo": "str"
    },
    kind=ModelKindName.FULL,
)
def execute(
    context: ExecutionContext,
    start: datetime,
    end: datetime,
    execution_time: datetime,
    *,
    custom_mult: int = 1,
    **kwargs: Any,
) -> pd.DataFrame:

if I set a variable in my config

variables:
  custom_mult: 5

it won't get passed through, even though sqlmesh specifies that these are always passed as keyword-only arguments.

Please note that arguments must be specified explicitly; in other words, variables can be accessed using kwargs.

If i remove the *, then it passes through as expected. I think this would be nice functionality to add because it allows python models to be represented semantically correctly, mapping to how they are used

treysp commented 6 days ago

@izeigerman Should "variables can be accessed using kwargs" be "variables CANNOT be accessed using kwargs" here?

izeigerman commented 6 days ago

If i remove the *, then it passes through as expected.

I guess that's the core issue here.

@treysp not sure how kwargs are relevant.