microsoft / go-sqlcmd

The new sqlcmd, CLI for SQL Server and Azure SQL (winget install sqlcmd / sqlcmd create mssql / sqlcmd open ads)
https://learn.microsoft.com/sql/tools/sqlcmd/go-sqlcmd-utility
MIT License
323 stars 56 forks source link

Add ability to import/export data to/from Parquet files with SQLCMD #486

Open dlevy-msft opened 6 months ago

dlevy-msft commented 6 months ago

Creating this placeholder for the investigation of adding import/export functionality to go-sqlcmd.

Main use cases would include:

  1. Output SQL Server query results to parquet files
  2. Read data from parquet files for display of some or all rows on screen to check contents of file
  3. Read data from parquet files and insert into table in SQL Server database
  4. Be able to read both standalone parquet files as well as delta
  5. Perform basic filtering via SQL-like where clause or regex by column

Additional possibilities include:

  1. Output SQL Server query results to json, csv, xls files
  2. Convert json, csv, xls to parquet
  3. Convert parquet, including delta, to json, csv, xls
  4. Convert subset of rows between formats - ex: "Take customer data from delta store and export all customers in Chicago to xls"
flarco commented 15 hours ago

You can do this easily with sling, check out: https://github.com/slingdata-io/sling-cli

# set connection via env var
export mssql='sqlserver://...'

# test connection
sling conns test mssql

# run export for many tables
sling run --src-conn mssql --src-stream 'my_schema.*' --tgt-object 'file://{stream_schema}/{stream_table}.parquet'

# run export for one table
sling run --src-conn mssql --src-stream 'my_schema.my_table' --tgt-object 'file://my_folder/my_table.parquet'

# run export for custom sql
sling run --src-conn mssql --src-stream 'select col1, col2 from my_schema.my_table where col3 > 0' --tgt-object 'file://my_folder/my_table.parquet'