monogon-dev / intellij-cue

IntelliJ support for the CUE language.
Apache License 2.0
36 stars 3 forks source link

Verify UTF-8 encoding #3

Open jansorg opened 3 years ago

jansorg commented 3 years ago

https://github.com/cuelang/cue/blob/master/doc/ref/spec.md#source-code-representation

UTF-8 seems to be the only valid encoding. We could:

leoluk commented 3 years ago

The scanner seems to be quite forgiving:

https://github.com/cuelang/cue/blob/d9d048725a6d2e69c16533ddf87be7368f199740/cue/scanner/scanner.go#L81

Even clearly invalid Unicode is parsed and survives the round-trip through cue eval:

echo -ne 'foo: "\xf0\x28\x8c\x28\x7F\xfe\xff"' >> imports.cue

Not sure if this is a bug or a feature.

orpharion commented 3 years ago

I have had issues where UTF-16 json files on windows fail to read.