proposal: encoding/xml: add iterators Decoder.Tokens, Decoder.RawTokens

[dolmen]

Proposal Details

Add iter.Seq2 iterator methods on encoding/xml.Decoder to iter over tokens and raw tokens.

func (d *Decoder) Tokens() iter.Seq2[Token, error] {
    return func(yield func(Token, error) bool) {
        for {
            token, err := d.Token()
            if err == io.EOF {
                break
            }
            if !yield(token, err) {
                break
            }
            // What should we do here if err != nil?
        }
    }
}

func (d *Decoder) RawTokens() iter.Seq2[Token, error] {
    return func(yield func(Token, error) bool) {
        for {
            token, err := d.RawToken()
            if err == io.EOF {
                break
            }
            if !yield(token, err) {
                break
            }
            // What should we do here if err != nil?
        }
    }
}

Note: I'm not yet sure to like an iterator that handles a stream error this way.

Related:

• 66626

[gabyhelp]

Related Issues

• encoding/xml: add decode from TokenReader, to enable stream transformers #19480 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dolmen]

Note that generic functions can also be written outside the package so the Decoder methods would be purely to allow "native" range-over-func support:

func IterTokens[Token any, Tokenizer interface{ Token() (Token, error) }](tok Tokenizer) iter.Seq2[Token, error] {
    return func(yield func(Token, error) bool) {
        for {
            token, err := tok.Token()
            if err == io.EOF {
                break
            }
            if !yield(token, err) {
                break
            }
        }
    }
}

func IterRawTokens[Token any, Tokenizer interface{ RawToken() (Token, error) }](tok Tokenizer) iter.Seq2[Token, error] {
    return func(yield func(Token, error) bool) {
        for {
            token, err := tok.RawToken()
            if err == io.EOF {
                break
            }
            if !yield(token, err) {
                break
            }
        }
    }
}