golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
122.95k stars 17.53k forks source link

x/text/encoding/unicode: add examples #14133

Open TomOnTime opened 8 years ago

TomOnTime commented 8 years ago

golang.org/x/text/encoding/unicode's documentation is a bit unclear to the newbie like myself that just wants to read files that Windows folks send them without having to become an expert in UTF-16. I think the docs make sense to someone that already understands transforms, unicode, etc, but not everyone should need to know such plumbing. All I wanted was something like ioutil.ReadFile() that automagically read MS-Windows UTF-16 files and gave me UTF-8.

An example like this would have helped:

func NewReader(rd io.Reader) io.Reader {
    // Make an tranformer that decodes MS-Windows (16LE) UTF files:
    winutf := unicode.UTF16(unicode.LittleEndian, unicode.IgnoreBOM)
    // Make a transformer that is like winutf, but abides by BOM if found:
    decoder := winutf.NewDecoder()
    // Make a Reader that uses decoder:
    return transform.NewReader(rd, unicode.BOMOverride(decoder))
}

fd, _ := os.Open(filename)
r := NewReader(fd)  // Read from "r" to get UTF-8.
utf8, _ := ioutil.ReadAll(r)
text := string(utf8)

It would be awesome if that example (or one like it) was added to the documentation. I'd be glad to submit a PR.

Furthermore, it would be useful to have a library similar to ioutil but automagically detects UTF-16 is found. I've made an example here: https://github.com/TomOnTime/utfutil

TomOnTime commented 8 years ago

P.S. I think the example might be clearer if the func was inlined.