reflect: MakeInterface

[bradfitz]

Feature request for MakeInterface, to go along with the newly-arrived MakeFunc (Issue
1765).

Good goal would be gomock without source code generation.

[jimmyfrasche]

If something like that was added to the reflect package itself, then fmt.Errorf could be rewritten to something like:

// Errorf formats according to a format specifier and returns the string
// as a value that satisfies error.
//
// The methods of the first error in the format args are still there
// [editor's note: I shouldn't write this docstring]
func Errorf(format string, a ...interface{}) error {
  msg := errors.New(Sprintf(format, a...))
  if err := firstError(a); err != nil {
    return reflect.Combine(err, msg)
  }
  return msg
}
func firstError(a []interface{}) error {
  for _, i := range a {
    if e, ok := i.(error); ok {
      return e
    }
  }
  return nil
}

That's not a light change to consider, it could have some weird consequences, but it would mean that a lot of code that hides optional methods on errors would stop doing so with no modification.

[jimmyfrasche]

Another issue: (should have tested this earlier) StructOf can't create an unexported field, though maybe that's not an issue, since you could only get to the fields with reflect anyway and they're fairly arbitrary. If you use it to created exported fields with an unexported type from another package you can set that field, which is mildly surprising, but that is what we want to do here: https://play.golang.org/p/wQ8dC45dmI

[jimmyfrasche]

Let's say for the moment that reflect can create new types with methods, something equivalent to what I wrote above is added to the stdlib, and that it becomes the blessed way, by convention, of composing errors.

This approach works well for errors with additional methods but not so well for errors with additional fields or sentinel errors, like io.EOF. IsX style funcs, like os.IsPermission, that dispatch on the identity of the type don't play well with this scheme.

Errors with additional fields can be made to work with this approach by adding an unexported method and an exported helper. For example, let's examine os.PathError. If it adds a method like

func (p *PathError) pathError() *PathError {
  return p
}

and the OS package adds a func like

func IsPathError(err error) *PathError { // result == nil implies false
  if p, ok := err.(interface{pathError() *PathError}); ok {
    return p.pathError()
  }
  return nil
}

then an error, under any number of combinations (with non-PathErrors), is still fully inspectable. Helpers like IsPermission could use similar tactics behind the scenes to extract the correct underlying error for their tests.

There's a small increase in API and implementation but everything works and is Go1 compatible.

Sentinel errors are a bit more annoying. They could follow the same basic approach, but each one would need its own type, like

type eof struct{}
func (e eof) Error() string { return "EOF" }
func (e eof) isEOF() {}
var EOF = eof{}
func IsEOF(err error) bool {
  _, eof := err.(interface{isEOF()})
  return eof
}

I don't think that's strictly Go1 compatible, but the only code I can think of that that would break is something using reflect. Perhaps it would be permissible as part of the transition to Go2?

[jimmyfrasche]

It looks like the specific approach taken in the gist I posted relied on a bug ( #22073 ).

If it wants to expose unexported methods, it would need to use embedding to handle the method promotion (once that is in place) and only create proxy methods as tie-breakers for exported methods.

However that has several unhappy implications:

1. it cannot remove any "useless" values as it cannot detect when they do not contribute to the resultant method set
2. it cannot create tie-breakers for unexported methods which implicitly removes the method from the method set due to the ambiguity

1 is unfortunate but non-essential.

However, 2 means it just cannot handle unexported methods correctly. It needs to stick to only exported methods and always hide all unexported methods.

So the approach in the gist is correct and it can discard values that don't add to the method set, but it's less useful than I'd hoped because it can't handle unexported methods.

In order to handle unexported methods in the same manner as exported methods, it would need to either be a builtin or a function defined in the reflect package that circumvents the usual protections.

[dsnet]

I hit a use case for this while working on google/go-cmp#88. The gist of the problem is that TypeOf and ValueOf can never directly return a type that is an interface. I understand why it is the case, but it goes against user expectation:

b := new(bytes.Reader)
reflect.TypeOf(b)            // *bytes.Reader
reflect.TypeOf(io.Reader(b)) // io.Reader expected, but got *bytes.Reader

This causes an issue when doing something like:

equateNotExist := cmp.Comparer(func(x, y error) bool { ... })
var e1, e2 error = syscall.ENOENT, os.ErrNotExist
cmp.Equal(e1, e2, equateNotExist) // false

The user expected this to be true since there is a custom comparer that knows how to compare error interfaces. However, this returns false because under the hood, cmp.Equal uses reflect.ValueOf and loses the fact that the user intended to pass in two error interfaces. cmp.Equal see two different types syscall.Errno and *errors.errorString and immediately returns false without even giving the custom comparer a chance to work.

If you so much as pass even a pointer to the interface, then it works as expected:

cmp.Equal(&e1, &e2, equateNotExist) // true

However, doing so is not intuitive at all.

In an effort to make this more user-friendly, my approach was to try to promote the initial top-level value into an interface with the common set of methods to both values:

// Determine the set of common methods.
type method struct {
    Name string
    Type reflect.Type
}
mm := make(map[method]int)
for i := 0; i < vx.NumMethod(); i++ {
    mm[method{vx.Type().Method(i).Name, vx.Method(i).Type()}]++
}
for i := 0; i < vy.NumMethod(); i++ {
    mm[method{vy.Type().Method(i).Name, vy.Method(i).Type()}]++
}
var ms []reflect.Method
for m, n := range mm {
    if n == 2 {
        ms = append(ms, reflect.Method{Name: m.Name, Type: m.Type})
    }
}

// Promote to interface type with common method set.
iface := reflect.InterfaceOf(ms)
return vx.Convert(iface), vy.Convert(iface)

but this does not work since there is no reflect.InterfaceOf method.

[Merovius]

@dsnet I might misread this, but the documentation says that

The comparer f must be a function "func(T, T) bool" and is implicitly filtered to input values assignable to T. If T is an interface, it is possible that f is called with two values of different concrete types that both implement T.

That would seem to cover the case you mention? In particular this works.

[dsnet]

@Merovius, so as to keep this thread about InterfaceOf, I'll comment on google/go-cmp#88 regarding your comment. My point in my previous comment is that I had use-case for InterfaceOf to create a dynamic interface that contains the common set of methods for 2 concrete types.

[matttproud]

I don't want to be a party pooper, but adding a feature like reflect.MakeInterface — or any other way of creating arbitrary types to fulfill interfaces at runtime — pushes the Go complexity clock quite close to midnight. My fear hearkens from building, maintaining, and diagnosing production Java servers that pushed the JRE through things like cglib and similar machinations. I am really not trying to sound like reactionary, but these systems were an absolute nightmare and enabled bounds of complexity in Inversion of Control harnesses and other obtuse stacks of indirection. The only thing that could — barely — tame JRE dynamic code generation is Java's superb runtime debugger ecosystem, which we don't have.

I skimmed over the issue's feedback and wondered: is there value in having a discussion 1. whether we should have this feature at all and 2. whether the benefits disproportionately outweigh the likely costs.

[ianlancetaylor]

@matttproud I assume you are talking about the case of adding methods to a type created by using the reflect package. E.g., adding methods to a type created via reflect.StructOf. I clarify because I think that the function that does that operation should not be called reflect.MakeInterface. I think that reflect.MakeInterface, possibly better named reflect.InterfaceOf, is a function that creates a new interface type, just as reflect.StructOf creates a new struct type. I don't see any particular harm in being able to do that.

Personally I'm also not too worried about being able to add methods to a type created by reflect. That is nothing like Java's cglib. In Go, the code for the actual methods will be, essentially, a function literal. Adding methods to a type will amount to a different way of calling a function literal. Yes, the stack trace will be complex, but this is still much less powerful and much less complex than cglib.

[march1993]

I tried to imitate reflect.StructOf by creating reflect.StructOf but I got the following error,

unexpected fault address 0xc000198090
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0xc000198090 pc=0xc000198090]

Here's the file added to reflect package,

package reflect

import (
    "sort"
    "unsafe"
)

type InterfaceMethod struct {
    Name string
    Func Value
}

func InterfaceOf(funcs []InterfaceMethod) Value {
    if len(funcs) == 0 {
        return New(TypeOf("Interface {}"))
    }

    var (
        repr     = []byte("struct { /* non-invertible */ }")
        hash     = fnv1(0, []byte("struct { /* non-invertible */ }")...)
        size     uintptr
        typalign uint8
        methods  []method
    )

    // add sorted methods here
    {
        for idx := range funcs {
            if funcs[idx].Func.Type().Kind() != Func {
                panic("funcs contains non-func")
            }
            c := funcs[idx].Name[0]
            if !('A' <= c && c <= 'Z') {
                panic("funcs contains unexported func")
            }
        }

        sort.Slice(funcs, func(i, j int) bool {
            return funcs[i].Name < funcs[j].Name
        })

        for idx := range funcs {
            ft := funcs[idx].Func.Type().common()
            ft.uncommon()

            ifn := funcs[idx].Func
            methods = append(methods, method{
                name: resolveReflectName(newName(funcs[idx].Name, "", true)),
                mtyp: resolveReflectType(ft),
                ifn:  resolveReflectText(unsafe.Pointer(&ifn)),
                tfn:  resolveReflectText(unsafe.Pointer(&ifn)),
            })
        }
    }

    var typ *structType
    var ut *uncommonType

    {
        // A *rtype representing a struct is followed directly in memory by an
        // array of method objects representing the methods attached to the
        // struct. To get the same layout for a run time generated type, we
        // need an array directly following the uncommonType memory.
        // A similar strategy is used for funcTypeFixed4, ...funcTypeFixedN.
        tt := New(StructOf([]StructField{
            {Name: "S", Type: TypeOf(structType{})},
            {Name: "U", Type: TypeOf(uncommonType{})},
            {Name: "M", Type: ArrayOf(len(methods), TypeOf(methods[0]))},
        }))

        typ = (*structType)(tt.Elem().Field(0).Addr().UnsafePointer())
        ut = (*uncommonType)(tt.Elem().Field(1).Addr().UnsafePointer())

        copy(tt.Elem().Field(2).Slice(0, len(methods)).Interface().([]method), methods)
    }

    ut.mcount = uint16(len(methods))
    ut.xcount = ut.mcount // we only have exported methods
    ut.moff = uint32(unsafe.Sizeof(uncommonType{}))
    str := string(repr)

    // Round the size up to be a multiple of the alignment.
    size = align(size, uintptr(typalign))

    // Make the struct type.
    var istruct any = struct{}{}
    prototype := *(**structType)(unsafe.Pointer(&istruct))
    *typ = *prototype

    typ.str = resolveReflectName(newName(str, "", false))
    typ.tflag = 0 // TODO: set tflagRegularMemory
    typ.hash = hash
    typ.size = size
    typ.ptrdata = typeptrdata(typ.common())
    typ.align = typalign
    typ.fieldAlign = typalign
    typ.ptrToThis = 0
    typ.tflag |= tflagUncommon

    {
        typ.kind &^= kindGCProg
        bv := new(bitVector)
        addTypeBits(bv, 0, typ.common())
        if len(bv.data) > 0 {
            typ.gcdata = &bv.data[0]
        }
    }

    typ.equal = nil

    typ.kind &^= kindDirectIface

    return Zero(&typ.rtype)
}

Here's the testing code,

package main

import (
    "fmt"
    "reflect"
    "testing"
)

func TestInterfaceOf(t *testing.T) {
    type AliceInterface interface {
        BobMethod() (error, string)
    }
    Alice := reflect.TypeOf((*AliceInterface)(nil)).Elem()
    Bob := Alice.Method(0)

    Carol := reflect.MakeFunc(Bob.Type, func(args []reflect.Value) []reflect.Value {
        fmt.Println("[wow]")
        nilError := reflect.Zero(reflect.TypeOf((*error)(nil)).Elem())
        return []reflect.Value{nilError, reflect.ValueOf("haha")}
    })

    Dan := reflect.InterfaceOf([]reflect.InterfaceMethod{
        {
            Name: "BobMethod",
            Func: Carol,
        },
    })

    Dan.Method(0).Call([]reflect.Value{}) // panic

    dan := Dan.Interface().(AliceInterface)
    dan.BobMethod() // also panic

}

Does anybody know why?

[march1993]

Why was my comment marked as off-topic? 😠

[DeedleFake]

I would guess that it's because the Go issue tracker is explicitly not for asking questions. I'd try either golang-nuts or /r/golang.