databrickslabs / lsql

Lightweight SQL execution wrapper only on top of Databricks SDK
https://pypi.org/project/databricks-labs-lsql/
Other
9 stars 3 forks source link

SqlBackend cannot create schema for dataclasses inheriting from dataclasses #103

Open ericvergnaud opened 4 months ago

ericvergnaud commented 4 months ago

Say you have the following:

@dataclass
class MyBaseDataClass:
    path: str

@dataclass
class MyDerivedDataClass(MyBaseDataClass):
    name: str

Then SqlBackend._schema_for fails because field_type for path is 'str'(type name) rather than <type str> (type instance)

ericvergnaud commented 4 months ago

It's unclear how to reproduce this issue in lsql. It happens consistently in ucx when (for example) JobProblem derives from Problem. It doesn't happen in ucx if Derived and Base are in the same file. This smells like a Python bug (there seems to be many dataclass related Python bugs), we don't have any code that writes reflection data. There is an easy fix, but waiting for comments.