turbot / steampipe-plugin-csv

Use SQL to instantly query data from CSV files. Open source CLI. No DB required.
https://hub.steampipe.io/plugins/turbot/csv
Apache License 2.0
19 stars 4 forks source link

support for gzipped csv #41

Closed daeho-ro closed 2 years ago

daeho-ro commented 2 years ago

I have added the gzip support, #39. Let say the file path is .../test.csv.gz.

  1. I include the match string condition like "*.csv.gz" in the config file.
  2. Since the table name is wrong, cut the base when it ends with .gz. As a result, the base is now test.csv and make the table name as test same as before.
  3. Check if the path ends with .gz when the file open, then add one more step to unzip the gzipped file.

Example query results

Results Here is the sample csv file. ``` > cat test.csv a,b,c,d,e 1,2,3,4,3 1,2,1,1,2 5,4,3,2,1 a,a,a,b,c ``` Let us gzip the file, ``` > gzip test.csv.gz ``` and then you can find the file is now gzipped. ``` > ls test.csv.gz ``` Now, do `steampipe query` and the result is here. ``` Welcome to Steampipe v0.16.4 For more information, type .help > select * from test +---+---+---+---+---+---------------------------+ | a | b | c | d | e | _ctx | +---+---+---+---+---+---------------------------+ | a | a | a | b | c | {"connection_name":"csv"} | | 5 | 4 | 3 | 2 | 1 | {"connection_name":"csv"} | | 1 | 2 | 1 | 1 | 2 | {"connection_name":"csv"} | | 1 | 2 | 3 | 4 | 3 | {"connection_name":"csv"} | +---+---+---+---+---+---------------------------+ ```
daeho-ro commented 2 years ago

I have updated my code to deal with the BOM skip part.

cbruno10 commented 2 years ago

Hey @daeho-ro , thanks for raising this PR! Just commenting to let you know this is on my radar to review tomorrow, sorry for the delay, I've been working through my backlog for the moment.

cbruno10 commented 2 years ago

@daeho-ro Sorry for the long wait for the review, thanks for adding this new feature! We'll plan for a release in a new plugin version early next week.

daeho-ro commented 2 years ago

No problem!