search

greymd / teip

Masking tape to help commands "do one thing well"
MIT License
571 stars 19 forks source link

Load CSV file comply with RFC 4180 #34

Closed greymd closed 1 year ago

greymd commented 2 years ago

Example

$ cat file.csv
"AAA","BBB,CCC",DDD
$ cat file.csv | teip --csv -f 2
"AAA","[BBB,CCC]",DDD
$ cat file.csv | teip --csv-ex -f 2
"AAA",["BBB,CCC"],DDD
$ cat file2.csv | teip --csv -f 2
"AAA","[BBB
CCC]",DDD

=> If a record includes the newline, solid mode will automatically be enabled.

$ cat file2.csv | teip --csv --csv-escape-newline -f 2
"AAA","[BBB\nCCC]",DDD
greymd commented 1 year ago

As of https://github.com/greymd/teip/commit/2aa840a35e062d4342ff842496b5dd536cd136a9, It works.

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time zcat /home/ubuntu/repos/greymd/test_files/xsv/1000000_Sales_Records.csv.gz  > /dev/null

real    0m1.091s
user    0m1.007s
sys     0m0.050s

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time zcat /home/ubuntu/repos/greymd/test_files/xsv/1000000_Sales_Records.csv.gz | ./target/release/teip --csv -f 12-13 -- sed 's/./@/g' > /dev/null

real    0m7.430s
user    0m7.575s
sys     0m0.263s

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time zcat /home/ubuntu/repos/greymd/test_files/xsv/1000000_Sales_Records.csv.gz | ./target/release/teip -d, -f 12-13 -- sed 's/./@/g' > /dev/null

real    0m8.701s
user    0m10.564s
sys     0m0.313s
greymd commented 1 year ago

Released at v2.1.0 https://github.com/greymd/teip/releases/tag/v2.1.0