proposal: bufio: introduce a NewWriterBuffer

[ahmedabdou14]

Proposal Details

Background

The bufio package currently only provides 2 APIs to create a new Writer

• NewWriter(w io.Writer) *Writer
• NewWriterSize(w io.Writer, size int) *Writer

Problem

Both functions internally allocate a buffer, which may not be ideal for all use cases.

Motivation

There are scenarios where developers may want to manage buffer allocation themselves, such as using a sync.Pool of buffers or byte slices. In such cases, the current bufio APIs force a new buffer allocation each time a bufio.Writer is created, thus limiting the benefits of reusing buffers through a pool.

Proposal

Add a new function to the bufio package:

func NewWriterBuffer(w io.Writer, buf []byte) *Writer

[gabyhelp]

Related Issues and Documentation

• Package bufio > type Writer ¶ > func NewWriter ¶
• Package bufio > type Writer ¶ > func NewWriterSize ¶
• bufio: add Writer.AvailableBuffer #47527 (closed)
• bufio: add Writer.AvailableBuffer
• bufio: Reader and Writer Support add a method use caller's buf? #13650 (closed)
• Package textproto > type Writer ¶ > func NewWriter ¶
• proposal: encoding/csv add a new constructor with a configurable buffer size #67128 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

Note that this was rejected in #13650 by saying to put the entire bufio.Writer in the pool. Why is this not sufficient?

[ahmedabdou14]

Ya i only saw that issue after i posted this issue. However i left it open as there are still use cases where using a pool of []byte instead of a pool of bufio.Writer would come in handy.

example: what if you need a pool of byte slices to perform some logic and also want to reuse the same pool to create bufio writers? Currently, you will have to have 2 separate pools, one for bufio writers and the other for byte slices.

[adonovan]

example: what if you need a pool of byte slices to perform some logic and also want to reuse the same pool to create bufio writers? Currently, you will have to have 2 separate pools, one for bufio writers and the other for byte slices.

Yes, you would need two pools. But the change proposed here would break the encapsulation of the buffer internal to bufio.Writer, which seems like a high price to pay to avoid the need for two pools in rare cases.

[ahmedabdou14]

tbf, it is scary what could happen if devs reuse the pointer to the buffer outside the pool cycle. Maybe this feature can be an external package? I would be down to do one if it was decided not to move forward with this request.

[adonovan]

Maybe this feature can be an external package?

I'm not sure what you are suggesting. Are you proposing to make a fork of bufio that has the new functions? If so, you can do that in your own repo without asking for permission here.

[ahmedabdou14]

Yes exactly it can be a fork if it was decided not be added in the standard library

[ahmedabdou14]

For those looking for this feature, I have created the package here with the proposed interface and tests https://pkg.go.dev/github.com/ahmedabdou14/bufio