search

mikefarah / yq

yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor
https://mikefarah.gitbook.io/yq/
MIT License
11.28k stars 564 forks source link

--from-file does not accept \r\n line separators on Windows #1860

Closed boekhold closed 6 months ago

boekhold commented 8 months ago

Describe the bug

PS > Get-Content -Raw .\helmfile.yaml | yq --from-file patch.yq
Error: 7:2: invalid input text "\r\n|\nselect(.kind..."

where patch.yq has Windows Style CrLf line endings.

Also note that yq seems to be very sensitive of the Encoding of the file specified via --from-file. It does not accept UTF-16 files (which my PowerShell produces by default), and it does not accept a UTF-8 file with BOM.

Version of yq: 4.35.2 Operating system: Windows 10 22H2 Installed via: binary release

Command The command you ran:

Get-Content -Raw .\helmfile.yaml | yq --from-file patch.yq
mikefarah commented 8 months ago

Thanks for raising. The line ending issue is pretty straight forward to fix, and will be included in the next release.

I'm having trouble with the UTF encoding - theres quite a few different UTF16 encoding standards, and I haven't found a decent go library that handles them properly.

boekhold commented 7 months ago

@mikefarah I'm not sure if I should be reporting this here, as it seems to be related to the CrLf handling, or if I should open a new issue for this.... Let me know if I need to move this to a new issue.

Platform: Windows Input file:

---
# document 1 start comment
test:
  sub:
   key: value
---
# document 2 start comment
foo:
  bar:
   key: value

Line Ending: CrLf

yq '.' test-in.yaml:

---
# document 1 start comment
test:
  sub:
    key: value
# document 2 start comment
---
foo:
  bar:
    key: value

Note how the "document 2 start comment" has moved from the top of the second document to the bottom of the first document!

This is causing havoc with the Kubernetes manifest that I'm manipulating with yq...

If I explicitly change the line endings of test-in.yaml to UNIX-style "Lf", the issue does not appear.

mikefarah commented 7 months ago

Hey, thanks for raising, this is a separate issue. I've created another card :)

mikefarah commented 7 months ago

CRLF issue has been fixed in 4.40.2

mikefarah commented 6 months ago

Closing as the main issue has been fixed. Feel free to raise a separate issue for UTF coding if desired.