search

databricks / databricks-sql-python

Databricks SQL Connector for Python
Apache License 2.0
155 stars 90 forks source link

ENH: Add LOAD DATA LOCAL capability #132

Open dmoore247 opened 1 year ago

dmoore247 commented 1 year ago

Add a LOAD DATA LOCAL type capability to take files local to client, load to dbfs/volumes/bucket/storage account staging area, then COPY INTO a delta lake table, optionally remove data from staging area after COPY INTO. The benefit is to support bulk data loads from any client.

dmoore247 commented 1 year ago

I have code for upload and COPY INTO, would like to figure out right design (if appropriate) to add to api.

susodapop commented 1 year ago

Is this not already supported with the staging ingestion features? See the example here: https://github.com/databricks/databricks-sql-python/blob/main/examples/staging_ingestion.py

kravets-levko commented 5 months ago

@dmoore247 did you have a chance to try the staging ingestion feature? Does it solve your issue?