search

kaiko-ai / typedspark

Column-wise type annotations for pyspark DataFrames
Apache License 2.0
65 stars 4 forks source link

Autocomplete table names #120

Closed nanne-aben closed 1 year ago

nanne-aben commented 1 year ago

This PR allows for autocomplete when loading tables

from typedspark import Databases

DBs = Databases(spark)
persons, Person = DBs.default.person_table.load()
hahamark1 commented 1 year ago

General question, is this something which runs quickly in databricks: DBs = Databases(spark) ?

nanne-aben commented 1 year ago

On our own Databricks environment it runs in about 4 seconds. I imagine there are companies with many more databases, hence I've included the Timeout class, which gives a warning and a suggestion to run the code for an individual database.