pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
19 stars 3 forks source link

mssql: Fast materialization/dematerialization based on bcpandas and similar bulk load techniques #169

Open windiana42 opened 6 months ago

windiana42 commented 6 months ago

Bulk loads are significantly faster than ODBC/JDBC connections for simple transport of data to SQL Server. We prepared the possibility to use bcpandas for bulk uploads. However, it seems we never fully integrated it into the respective table hook.