Describe the data you'd like, be specific, link the docs
Instead of yielding database rows as lists of dictionaries (which go through standard relational normalizer in dlt) we can yield arrow tables on panda frames.
[ ] add support of yielding panda frames via pandas.io.sql module. mind that this method does not preserve certain types ie. decimals
[ ] add support for pyarrow backend where we create a correctly typed schema ourselves and load chunks of data provided by SqlAlchemy
Which source is this for example: sql_database
Describe the data you'd like, be specific, link the docs Instead of yielding database rows as lists of dictionaries (which go through standard relational normalizer in dlt) we can yield arrow tables on panda frames.
pandas.io.sqlmodule. mind that this method does not preserve certain types ie. decimals