search

mikefarah / yq

yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor
https://mikefarah.gitbook.io/yq/
MIT License
12.36k stars 602 forks source link

tsv file with value containing double quote in non-quoted field #2156

Open kermorgant opened 2 months ago

kermorgant commented 2 months ago

Describe the bug

A tsv file with non quoted value including quotes like Caramel "Tradition" makes yq fail with parse error ...: bare " in non-quoted-field

Version of yq: 4.44.3 Operating system: linux Installed via: binary release

Input tsv

test.csv:

Code    Name
1010    Caramel "Tradition"

Command The command you ran:

yq -ptsv --tsv-auto-parse=f  -ojson test.csv

Error: bad file 'test.csv': parse error on line 2, column 14: bare " in non-quoted-field

Additional context This error originally came from a tsv file downloaded from google spreadsheet using ?format=csv query param.

mikefarah commented 1 month ago

This is an error coming from golang's CSV parsing library (https://pkg.go.dev/encoding/csv) , seems like you need to do:

Code    Name
1010    "a ""quoted-field"""

To get it to work.

kermorgant commented 1 month ago

Indeed, that worked for my test file but unfortunately, I don't have control of the input file in my work environment (it's a direct download from a google spreadsheet). There is a LazyQuote setting in https://pkg.go.dev/encoding/csv that would maybe offer a workaround. Would it make sense for yq to support that ?

kermorgant commented 1 month ago

For what it's worth, I managed to workaround this issue by downloading the file as csv instead of tsv.