perf: optimize encoder buffer size

Previously, for carrying out an encoding process, we defined array as buffer with size of 130051 (130Kb). The size is calculated from maximum bytes need from a message:

# Header + ((max n Fields) * (n value)) + ((max n DeveloperFields) * (n value))
# 1 + (255*255) + (255*255)  = 130051

If we look at the Global Profile (Profile.xlsx), The message that has the most fields is Session having 153 fields, this is 102 less fields than the maximum spec and not all field will be used since some of them are sport-specific making it more less that 153 fields and not all manufacturers use developer fields, even if they do only small number that being utilized. So having 130051 upfront does not make sense here.

In real world, I found out that most of FIT activity files produced from various brands, such as Garmin, Bryton, Coros, etc, are only using less than 256 bytes.

I think using array is less flexible and having the previous size is waste of memory. So, we better use slice instead of array and grow as needed. But we need to define the appropriate starting point to reduce the chance of growing so we don't allocate so often.

I did a simple experiment to see the maximum size of a single message from messages defined by Global Profile (Profile.xlsx). This test uses generalisation as we treat all array and string as 255 bytes. From the experiment we got 5900.

// Here is only a pseudo-code as the real experiment code is 
// a bit messy as I use fitgen's mesgdef builder.
var maxSize int
var mesgName string
for _, mesg := range messages {
    var size int
    for _, field := range mesg.Field {
        if field.Array || field.BaseType == basetype.String {
            size += 255
        } else {
            size += int(field.BaseType.Size())
        }
    }
    if size > maxSize {
        maxSize = size
        mesgName = mesg.Name
    } 
}
fmt.Println(mesgName, maxSize)

# Output :
session 5900

The resulting size 5900 from session message, once again this test uses generalisation and not all session fields might be utilized. I think we need less than 5900, and when I think about it, the size of the maximum message definition is 1537. This value is big enough but still a small value (even less than default buffer 4096, right?) and we know that only message's MarshalAppend can grow the size of the slice. Additionally, we only need to change small code.

muktihari / fit

perf: optimize encoder buffer size #415

Codecov Report