kaiko-ai / typedspark

Column-wise type annotations for pyspark DataFrames
Apache License 2.0
65 stars 4 forks source link

Provide auto-complete in notebooks for table names starting with an underscore #315

Closed nanne-aben closed 7 months ago

nanne-aben commented 7 months ago

Suppose we'd make this table.

df.write.saveAsTable("default._person")

And then we try to load it using Databases() (or any of the related classes).

db = Databases()

Then we don't get auto-complete, because our notebook assumes that we don't want to access private variables. To circumvent this, we rename the attribute as such:

person, _person = db.default.u_person()

The underlying table is not renamed, solely the class attribute used for autocomplete.

When renaming the attribute leads to a naming conflict (e.g. because u_person already exists), we resolve the conflict by adding more underscores (e.g. _person would then become u__person).