Capacity of slice does not match in use by bytes.Buffer between Go and generated Haxe

[elliott5]

This issue springs from issue #120

Code:

package main

import (
    "bytes"
    "fmt"
)

func main() {
    const japanese = "日本語日本語日本語日"
    var b bytes.Buffer
    for i := 0; b.Len() < 5_000; i++ {
        if i%100 == 0 {
            b.WriteString(japanese)
        } else {
            b.WriteString("0123456789")
        }
    }
    fmt.Println("cap:", b.Cap())
}

Results:

Go:
cap: 8192

--interp:
create compiler instance
listening on local port: 6254
nodejs server started
accepted connection
Generated: golibs//Main.hx - 0.763kb
haxe -lib go2hx -cp golibs -m Main --run Main
cap: 7292

--hl:
create compiler instance
listening on local port: 6280
nodejs server started
accepted connection
Generated: golibs//Main.hx - 0.763kb
haxe -lib go2hx -cp golibs --hl run.hl  -m Main
hl run.hl
cap: 7292

[PXshadow]

@elliott5 do by chance know the formula for growing capacity?

[elliott5]

There is a very good blog post on slices here that explains how it works in general.

To summarise:

By default, unless told otherwise (slicing or new capacity), the length of the underlying array is the length of the contents.

The capacity of the underlying array of a slice can be set by the user on creation. This is what is done by the Go bytes.Buffer code (used by the example code for this issue) - see func AppendByte() in the blog for similar logic.

The bigest complexity comes when using append:

• if the data to append will fit inside the target slice capacity, put it there; otherwise...
• ...a new underlying array is required, and there is a need to replicate the Go code that implements the size of the new underlying array for the new slice, which is here. func growslice(et *_type, old slice, cap int) slice is where the logic sits. Key part copied in the code block below:

    newcap := old.cap
    doublecap := newcap + newcap
    if cap > doublecap {
        newcap = cap
    } else {
        const threshold = 256
        if old.cap < threshold {
            newcap = doublecap
        } else {
            // Check 0 < newcap to detect overflow
            // and prevent an infinite loop.
            for 0 < newcap && newcap < cap {
                // Transition from growing 2x for small slices
                // to growing 1.25x for large slices. This formula
                // gives a smooth-ish transition between the two.
                newcap += (newcap + 3*threshold) / 4
            }
            // Set newcap to the requested cap when
            // the newcap calculation overflowed.
            if newcap <= 0 {
                newcap = cap
            }
        }
    }

[elliott5]

I think this may be a 2nd order problem, as the system still (mostly) works with a different slice growing strategy from Go.

[PXshadow]

I think this may be a 2nd order problem, as the system still (mostly) works with a different slice growing strategy from Go.

I agree, I did try writing up a system to exactly mimic go, but I had trouble with data transfer of the append because it doesn't keep track of the potential slice getting added into append, which I think go does keep track of?

Either way not very high on the priority list.

[PXshadow]

This issue seems to have regressed as the output now is: cap: 6303

[PXshadow]

Todo: implement this later and also look into using internally optimized functions such as Vector.fromArrayCopy in GoArray __append__

    newcap := old.cap
    doublecap := newcap + newcap
    if cap > doublecap {
        newcap = cap
    } else {
        const threshold = 256
        if old.cap < threshold {
            newcap = doublecap
        } else {
            // Check 0 < newcap to detect overflow
            // and prevent an infinite loop.
            for 0 < newcap && newcap < cap {
                // Transition from growing 2x for small slices
                // to growing 1.25x for large slices. This formula
                // gives a smooth-ish transition between the two.
                newcap += (newcap + 3*threshold) / 4
            }
            // Set newcap to the requested cap when
            // the newcap calculation overflowed.
            if newcap <= 0 {
                newcap = cap
            }
        }
    }