[Question]tinygo build wasm error when use protobuf

zhenjunMa commented 3 years ago

Question

when i use struct compiled by protobuf, and use tinygo build to a wasm file, i got an error.

How to reproduce

model.proto

syntax = "proto3";

package model;

message Request {
    string name = 1;
}

build command protoc -I. --go_out=. model.proto

hello.go

package main

import (
    "TinyGoProtoBuf/model"
)

func main() {
    _ = &model.Request{
        Name: "a",
    }
}

build command: tinygo build -o wasm.wasm -target wasm ./hello.go

error:

i have uploaded the total project: https://github.com/zhenjunMa/TinyGoProtoBuf

any idea about how to solve this? or tinygo can not support protobuf now.

zhenjunMa commented 3 years ago

hello.go

package main

import (
    "encoding/json"
    "fmt"
)

type request struct {
    name string `json:"name"`
}

func main() {
    req := &request{
        name: "hello",
    }

    b, _ := json.Marshal(req)

    fmt.Println(len(b))

}

i tried to use json like above, but still got an error:

so, if i can use any other way to serialize a struct ?

dengliangjun commented 3 years ago

I met those compiled errors as the same with yours You should update tinygo version to 0.18. However, Maybe you will meet new errors as the same with mine. i.e. xxxx/protobuf/proto/clone.go : AddpendSlice not declared by package reflect xxxx/protobuf/proto/discard.go : t.FieldByName undefined ( type reflect.Type has no field or method FieldByName) xxxx...... xxxx.....

If you solve this problem, Please tell me. we can solve this problem jointly. email: dengliangjun@skycto.com

aykevl commented 3 years ago

Some of these issues are fixed in the latest release. Can you try again with the latest release?

dengliangjun commented 3 years ago

Some of these issues are fixed in the latest release. Can you try again with the latest release?

Hi, I need your help, My Env : tinygo=0.18.0 go=1.13 target=wasm my source code like ： struct Order { Name string ‘json:name’ Price int 'json:price' } bytes:=[b'{',b‘}’]] // empty json "{}" json.Unmarshal(bytes,&order)

get a runtime error: panic: unimplemented : (reflect.Type).Name()

Whether My go version is too low. My source code can run within 'go build'

zhenjunMa commented 3 years ago

@aykevl thank you for you reply.

as @dengliangjun said, after i upgrade tinygo to 1.18.0

when i used protobuf to serialize a struct, i got an error:

github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect

when i used json to serialize a struct, i got an another error:

panic: unimplemented: (reflect.Type).Name()

the difference is for protobuf, the error is happened at compiled time, but for json, the error is happened at runtime 😂

dengliangjun commented 3 years ago

the same errors as you, me too! I want to solve/contribute this issue. Whether there is anyone else can direct me.

来自skycto.com企业邮箱------------------------------------------------------------------ @.> 日　期：2021年05月20日 11:10:12 @.> @.>; @.> 主　题：Re: [tinygo-org/tinygo] [Question]tinygo build wasm error when use protobuf (#1879)

@aykevl thank you for you reply. as @deadprogram said, after i upgrade tinygo to 1.18.0 when i used protobuf to serialize a struct, i got an error: github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect when i used json to serialize a struct, i got an another error: panic: unimplemented: (reflect.Type).Name() the difference is for protobuf, the error is happened at compiled time, but for json, the error is happened at runtime 😂 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

hamidrezakks commented 3 years ago

Well I had the same problem, i highly suggest to use go go proto instead. https://github.com/gogo/protobuf

Make sure you remove extra methods from generated .pb.go files and just keep marshal and unmarshal methods

aykevl commented 3 years ago

when i used json to serialize a struct, i got an another error:

panic: unimplemented: (reflect.Type).Name()

Yes, I'm aware of this issue. It is something that I want to fix in the reflect package. Unfortunately there is no easy fix.

aykevl commented 3 years ago

when i used protobuf to serialize a struct, i got an error:

github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect

This should be fixed with this PR: https://github.com/tinygo-org/tinygo/pull/1902

dengliangjun commented 3 years ago

Well I had the same problem, i highly suggest to use go go proto instead. https://github.com/gogo/protobuf

Make sure you remove extra methods from generated .pb.go files and just keep marshal and unmarshal methods thanks for your suggestions and help. have you test your approach? can you share your code within gogo/protobuf, which can be compiled with tinygo.

zhenjunMa commented 3 years ago

when i used protobuf to serialize a struct, i got an error: github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect

This should be fixed with this PR: #1902

@aykevl thank you for your attention about this issue.

I have tried to compile the latest dev branch which maybe fixed my question, the compiled result is below:

It's my bad that I didn't describe the problem clearly before, so let me redo this.

use tinyGo 0.18.0 to compile a go file that used protobuf to serialize a struct, the full error stack is:

➜  wasm git:(wasm) ✗ tinygo build -o wasm.wasm -scheduler=none -target=wasi -wasm-abi=generic -tags 'abi_010' wasm.go
# github.com/golang/protobuf/proto
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/clone.go:222:20: AppendSlice not declared by package reflect
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/discard.go:244:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/pointer_unsafe.go:287:17: NewAt not declared by package reflect
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/table_merge.go:646:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/table_unmarshal.go:406:39: reflect.Zero(reflect.PtrTo(t)).MethodByName undefined (type reflect.Value has no field o

use tinyGo 0.19.0-dev to compile the same go file, the full error stack is:

➜  wasm git:(wasm) ✗ tinygo build -o wasm.wasm -scheduler=none -target=wasi -wasm-abi=generic -tags 'abi_010' wasm.go
# github.com/golang/protobuf/proto
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/discard.go:244:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/pointer_unsafe.go:287:17: NewAt not declared by package reflect
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/table_merge.go:646:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../../../../../pkg/mod/github.com/golang/protobuf@v1.3.5/proto/table_unmarshal.go:406:39: reflect.Zero(reflect.PtrTo(t)).MethodByName undefined (type reflect.Value has no field o

PR: #1902 really solve the AppendSlice problem, but still can't compile protobuf to wasm.

@hamidrezakks

I also tried to use gogoprotobuf, but no matter 0.18.0 or 0.19.0-dev, it can't work yet, the error is same with above.

could you share your code?

hamidrezakks commented 3 years ago

Hi @zhenjunMa

Here is one of our protos

// MessageEnvelope
// This type of message will be used to contain another ProtoBuffer Message inside
message MessageEnvelope {
    int64 Constructor = 1;
    fixed64 RequestID = 2;
    bytes Message = 4;
    bytes Auth = 8;
    repeated KeyValue Header = 10;
}

the generated gogo file has many functions just keep these

func (m *MessageEnvelope) Marshal() (dAtA []byte, err error)
func (m *MessageEnvelope) MarshalTo(dAtA []byte) (int, error)
func (m *MessageEnvelope) MarshalToSizedBuffer(dAtA []byte) (int, error)

and the main struct which is

type MessageEnvelope struct {
    Constructor int64       `protobuf:"varint,1,opt,name=Constructor,proto3" json:"Constructor,omitempty"`
    RequestID   uint64      `protobuf:"fixed64,2,opt,name=RequestID,proto3" json:"RequestID,omitempty"`
    Message     []byte      `protobuf:"bytes,4,opt,name=Message,proto3" json:"Message,omitempty"`
    Auth        []byte      `protobuf:"bytes,8,opt,name=Auth,proto3" json:"Auth,omitempty"`
    Header      []*KeyValue `protobuf:"bytes,10,rep,name=Header,proto3" json:"Header,omitempty"`
}

in my case.

for generating proto.bp.go files use this command protoc -I=$GOPATH/src -I=./msg --gogofaster_out=./msg ./msg/*.proto

alanpoon commented 3 years ago

Here is my script to aggressively strip lines https://gist.github.com/alanpoon/541ceea2129412b5a5ae046ad90fccd6 to make it compile into wasm.

aykevl commented 3 years ago

I am working on refactoring the reflect package to be more powerful. This is still a work in progress, but once completed I think it will be feasible to implement the remaining features necessary for protobuf.

fredgoya commented 2 years ago

Is this (the reflect package work and the protobuf support) still WIP? I'd love for the proxy-wasm-go issue to move forward but it's depending on this ticket.

andresperezl commented 1 year ago

Using tinygo 0.28.1 to compile a wasm/wasi binary that uses protobuf, and using wasmedge to run it, I see the following error:

panic: unimplemented: reflect.New()

Which was strange as reflect.New() seems to be implemented. But looking for that text in the code further down below I found it at reflect.NewAt():

func NewAt(typ Type, p unsafe.Pointer) Value {
    panic("unimplemented: reflect.New()")
}

And looking at the protobuf-go code you can find a call to it here: https://github.com/protocolbuffers/protobuf-go/blob/master/internal/impl/pointer_unsafe.go#L78

tinygo-org / tinygo

[Question]tinygo build wasm error when use protobuf #1879

Question

How to reproduce