search

cody-scott / dagster-mssql-bcp

MIT License
1 stars 0 forks source link

Folder based load #9

Open cody-scott opened 4 days ago

cody-scott commented 4 days ago

Implement a folder based load where a list of csv files are loaded in bulk.

Method would be more chunk based to ingest to sql, with final io loading everything in as a single final import.

cody-scott commented 4 days ago

Should this be a separate io or am extension of the current one.

Separate then calling the normal one seems cleaner. Makes it clear the intent of the io