feature: Support for compilation via tinygo

dereulenspiegel commented 2 years ago

I tried to build a project using this library via tinygo (because this project would also make sense in microcontrollers), but unfortunately tinygo doesn't support reflect (or at least certain parts of it. The error reported is:

../../../../pkg/mod/github.com/fxamacker/cbor/v2@v2.2.0/decode.go:1019:17: MakeMapWithSize not declared by package reflect
../../../../pkg/mod/github.com/fxamacker/cbor/v2@v2.2.0/tag.go:196:17: contentType.PkgPath undefined (type reflect.Type has no field or method PkgPath)

It would be awesome if we could get this library to compile via tinygo to support microcontrollers and smaller WASM modules. Unfortunately I am not deep enough into the architecture of cbor to understand how hard the dependency to these reflect methods is and how to resolve this.

Is this possible at all?

fxamacker commented 2 years ago

Supporting tinygo seems like a worthwhile feature. I would need to look into it to understand how much work this would take.

dereulenspiegel commented 2 years ago

@fxamacker Awesome thanks. Really looking forward to using your cbor module on something like a NRF52 series microcontroller.

mkungla commented 2 years ago

Supporting tinygo seems like a worthwhile feature. I would need to look into it to understand how much work this would take.

Seems that only following in decoder.go would need some other approach, since tinygo reflect package as for now does not have reflect.MakeMapWithSize method implemented

https://github.com/fxamacker/cbor/blob/9e37184acacd7330c79de271370b26d57c228718/decode.go#L1256-L1262

so for instance following compiles with tinygo

if v.IsNil() {
  v.Set(reflect.MakeMap(tInfo.nonPtrType))
}

fxamacker commented 2 years ago

Hey @mkungla thanks for looking into this! I'll take a closer look this weekend.

fxamacker commented 2 years ago

TL;DR master branch requires extensive changes to support tinygo but features/stream-mode branch (superset of master branch) would require fewer changes to allow tinygo to use StreamEncoder and StreamDecoder (which don't exist in master yet).

@dereulenspiegel @mkungla I've been looking into tinygo and it's a really cool project! I experimented with this version of tinygo:

tinygo version 0.23.0 linux/amd64 (using go version go1.18.2 and LLVM version 14.0.0)

tinygo's support for reflect is still very much a work in progress and it uses stubs that allow compiling which would panic at runtime.

For example, tinygo has many stubs like these:

In tinygo's value.go, there are 30 matches for panic("unimplemented: which allows us to compile but they panic at runtime:

fa@tinygo-amd64:~/go/src/github.com/fxamacker/cbor_main$ ./cbor_main
panic: unimplemented: (reflect.Type).FieldByName()
Aborted (core dumped)

Based on these findings, adding support for tinygo 0.23 would require more extensive changes than anticipated.

I'm going to keep this issue open because tinygo is making progress incrementally adding more support for reflect. In the meantime, I will:

look into updating features/stream-mode branch so tinygo can use StreamEncoder and StreamDecoder
look into adding support for 32-bit arch and revisit tinygo when they tag new releases

Thanks again for suggesting tinygo!

mkungla commented 1 year ago

@fxamacker in brief look of your code I see that reflect package is primarily used to get underlying builtin type of value and value it self to be encoded or decoded?

If so, it might be possible to get rid of the `reflect' package as a dependency altogether and problem solved 😄. For example, you can create an "internal" package to get types and values that contains something as follows. The following is just a bit of experimental code I have, which of course is not a out of box solution.

package internal

import "unsafe"

// interface for the header of builtin value
type typeiface struct {
    kind *kindinfo
    ptr  unsafe.Pointer
}

// minimal builtin kind info struct needed to get that info
type typeinfo struct {
    size       uintptr
    ptrdata    uintptr // number of bytes in the kinde that can contain pointers
    hash       uint32  // hash of type; avoids computation in hash tables
    tflag      uint8   // extra type information flags
    align      uint8   // alignment of variable with this type
    fieldAlign uint8   // alignment of struct field with this type
    kind       uint8   // enumeration for C
}

type Kind uint

const (
    KindInvalid Kind = iota
    KindBool
    KindInt
    KindInt8
    KindInt16
    KindInt32
    KindInt64
    KindUint
    KindUint8
    KindUint16
    KindUint32
    KindUint64
    KindUintptr
    KindFloat32
    KindFloat64
    KindComplex64
    KindComplex128
    KindArray
    KindChan
    KindFunc
    KindInterface
    KindMap
    KindPointer
    KindSlice
    KindString
    KindStruct
    KindUnsafePointer
)

var kindNames = []string{
    KindInvalid:       "invalid",
    KindBool:          "bool",
    KindInt:           "int",
    KindInt8:          "int8",
    KindInt16:         "int16",
    KindInt32:         "int32",
    KindInt64:         "int64",
    KindUint:          "uint",
    KindUint8:         "uint8",
    KindUint16:        "uint16",
    KindUint32:        "uint32",
    KindUint64:        "uint64",
    KindUintptr:       "uintptr",
    KindFloat32:       "float32",
    KindFloat64:       "float64",
    KindComplex64:     "complex64",
    KindComplex128:    "complex128",
    KindArray:         "array",
    KindChan:          "chan",
    KindFunc:          "func",
    KindInterface:     "interface",
    KindMap:           "map",
    KindPointer:       "ptr",
    KindSlice:         "slice",
    KindString:        "string",
    KindStruct:        "struct",
    KindUnsafePointer: "unsafe.Pointer",
}

func (t Kind) String() (str string) {
    if uint(t) < uint(len(kindNames)) {
        str = kindNames[uint(t)]
    }
    return
}

// Kill "reflect"
func GetUnderlyingValueAndKind(in any, withvalue bool) (value any, kind Kind) {
    e := (*typeiface)(unsafe.Pointer(&in))

    // check whether it is really a pointer or not.
    t := e.kind
    if in == nil || t == nil {
        return nil, KindInvalid
    }

    // there are 27 kinds.
    // check whether t is stored indirectly in an interface value.
    f := uintptr(Kind(t.kind & ((1 << 5) - 1)))
    if t.kind&(1<<5) == 0 {
        f |= uintptr(1 << 7)
        kind = Kind(f & (1<<5 - 1))
    } else {
        kind = Kind(t.kind & ((1 << 5) - 1))
    }

        // return early if you only need to know underlying kind
    if !withvalue {
        return nil, kind
    }
    switch kind {
    case KindBool:
        value = *(*bool)(e.ptr)
    case KindInt:
        value = *(*int)(e.ptr)
    case KindInt8:
        value = *(*int8)(e.ptr)
    case KindInt16:
        value = *(*int16)(e.ptr)
    case KindInt32:
        value = *(*int32)(e.ptr)
    case KindInt64:
        value = *(*int64)(e.ptr)
    case KindUint:
        value = *(*uint)(e.ptr)
    case KindUint8:
        value = *(*uint8)(e.ptr)
    case KindUint16:
        value = *(*uint16)(e.ptr)
    case KindUint32:
        value = *(*uint32)(e.ptr)
    case KindUint64:
        value = *(*uint64)(e.ptr)
    case KindUintptr, KindPointer, KindUnsafePointer:
        value = *(*uintptr)(e.ptr)
    case KindFloat32:
        value = *(*float32)(e.ptr)
    case KindFloat64:
        value = *(*float64)(e.ptr)
    case KindComplex64:
        value = *(*complex64)(e.ptr)
    case KindComplex128:
        value = *(*complex128)(e.ptr)
    case KindString:
        value = *(*string)(e.ptr)
        // ... other supported kinds
    }
    return value, kind
}

mkungla commented 1 year ago

@fxamacker has that issue become obsolete