[RFC][GOLANG] TVM Golang Runtime Interface Discussion

[srkreddy1238]

Here briefing the golang interface design aspects.

Few golang concepts

• Usage of packages C and unsafe golang packages to go across golang boundary while accessing functions, types and pointer ..etc.
• Slices: Usage if golang slice where ever is possible to simplify application programming. Example being use slice of string instead of argument pair containing argc, argv

Please refer to golang tutorials for details of cgo, unsafe, slices ..etc.

Objective

• Initial version is expected to cover basic inference API listed below.
• Similar to many programming languages a package which can be imported and used is the best option here too.
• Hence the objective of this effort is a golang package gotvm which is wrapped over TVM c_runtime_api.

API

This initial version expected to cover the below API

func FuncListGlobalNames() (retVal []string, err error)
func GetGlobalFunction(funcname string) (retVal func(args ...interface{}) (*Value, error), err error)
func RegisterFunction(args ...interface{}) (err error)
type Array
    func Empty(shape []int64, args ...interface{}) (parray *Array, err error)
    func (parray Array) AsSlice() (retVal interface{}, err error)
    func (parray Array) CopyFrom(val interface{}) (err error)
    func (parray Array) GetCtx() (retVal Context)
    func (parray Array) GetDType() (retVal string)
    func (parray Array) GetNdim() (retVal int32)
    func (parray Array) GetShape() (retVal []int64)
type ByteArray
    func NewByteArray(val []uint8) (retVal *ByteArray)
    func (tbytearray ByteArray) GetData() (retVal string)
type Context
    func CPU(index int32) Context
    func CPUPinned(index int32) Context
    func GPU(index int32) Context
    func Metal(index int32) Context
    func OpenCL(index int32) Context
    func OpenGL(index int32) Context
    func ROCM(index int32) Context
    func SDAccel(index int32) Context
    func VPI(index int32) Context
    func Vulkan(index int32) Context
type Function
    func ConvertFunction(args ...interface{}) (fhandle Function, err error)
    func (tvmfunction Function) Invoke(args ...interface{}) (retVal *Value, err error)
type Module
    func LoadModuleFromFile(modpath string, args ...interface{}) (retVal *Module, err error)
    func (tvmmodule *Module) GetFunction(funcname string, args ...interface{}) (retVal func(args ...interface{}) (*Value, error), err error)
type Value
    func (tvmval Value) AsFloat64() (retVal float64)
    func (tvmval Value) AsFunction() (retVal *Function)
    func (tvmval Value) AsInt64() (retVal int64)
    func (tvmval Value) AsModule() (retVal *Module)
    func (tvmval Value) AsStr() (retVal string)

Sample Code

/*!
 *  Copyright (c) 2018 by Contributors
 * \brief Sample golang application deployment over tvm.
 * \file simple.go
 */

package main

import (
    "fmt"
    "runtime"
    "./gotvm"
)

// NNVM compiled model paths.
const (
    modLib    = "./deploy.so"
)

// main
func main() {
    // Welcome
    defer runtime.GC()
    fmt.Printf("TVM Go Interface : v%v\n", gotvm.GoTVMVersion)
    fmt.Printf("TVM Version   : v%v\n", gotvm.TVMVersion)
    fmt.Printf("DLPACK Version: v%v\n\n", gotvm.DLPackVersion)

    // Import tvm module (so)
    modp, _ := gotvm.LoadModuleFromFile(modLib)
    fmt.Printf("Module Imported\n")

    // Allocate Array for inputs and outputs.

    // Allocation by explicit type and context.
    tshapeIn  := []int64{4}
    inX, _ := gotvm.Empty(tshapeIn, "float32", gotvm.CPU(0))

    // Default allocation on CPU
    inY, _ := gotvm.Empty(tshapeIn, "float32")

    // Default allocation to type "float32" and on CPU
    out, _ := gotvm.Empty(tshapeIn)

    fmt.Printf("Input and Output Arrays allocated\n")

    // Fill Input Data : inX , inY
    inXSlice := []float32 {1, 2, 3, 4}
    inYSlice := []float32 {5, 6, 7, 8}

    // Copy the data on target memory through runtime CopyFrom api.
    inX.CopyFrom(inXSlice)
    inY.CopyFrom(inYSlice)

    fmt.Printf("X: %v\n", inXSlice)
    fmt.Printf("Y: %v\n", inYSlice)

    // Get function "myadd"
    funp, _ := modp.GetFunction("myadd")

    // Call function
    funp(inX, inY, out)

    fmt.Printf("Module function myadd executed\n")

    // Get the output tensor as an interface holding a slice through runtime CopyTo api.
    outSlice, _ := out.AsSlice()

    // Print results
    fmt.Printf("Result:%v\n", outSlice.([]float32))
}

@dmlc/tvm-team welcome to comment on this info.

Please refer to #1470 to have a look at initial effort.

[tqchen]

I hope we can have a healthy discussion on what high-level API that goes beyond C FFI style API can be provided. Specifically, if we look at tvm runtime API of java, js, python, we have specific object class for Module, Array, and Function which automatically manages their lifecycle and when necessary overloads the ops. I know little about go but from what I learnt, this should be possible

[tqchen]

Some things I think that are related. In python and js we use variadic interface to implement function invocation, it seems to be possible to implement a dynamically dispatched variadic function via func function_name(a ...interface{})

• see https://blog.learngoprogramming.com/golang-variadic-funcs-how-to-patterns-369408f19085

[tqchen]

Point2: ideally we want to remove Free functions and automatically manage the objects, and always use Finalizer to trigger the free of the objects https://golang.org/pkg/runtime/#SetFinalizer

[tqchen]

Point3: from user's point of view, we may not need to have a TVMFunction object at all, the handle can be captured, but module.GetFunction can directly return a closure with signature:

func function_name(a ...interface{})

This way the function returned by module.GetFunction can be directly invoked as a function

[tqchen]

Please update the examples accordingly, once we have a clear interface proposal, we can proceed on implementation again

[srkreddy1238]

Updated the interface and example code above. This version is based on

• variadic args for variable parameters
• Finalizers for automatic garbage collection
• Interfaces to send and receive arbitrary datatypes

The PR #1470 also updated accordingly for review.

[tqchen]

Three things:

• Try to (hide) the C APIs and remove them from API list
• Use return value style

  mod = gotvm.LoadModuleFromFile("xx.dso");

• Please check if there is a common convention of code style guide and have a consistent code style for function naming etc.

[srkreddy1238]

@tqchen Sorry, I forgot to submit the edited sample code and final API interface above :)

Have a look now.

• Only above listed API and Types are exposed on golang side.
• Yes, used return value style as ret, err. Optional second return value is golang style of reporting error. Here I am returning TVMGetLastErr as golang error object.
• Golang goes with CamelCase and PascalCase based on exported and un exported types.

[tqchen]

• TVMType might not be necessary, most cases people are good with strings e.g. "float32"
• Need to pass in TVMContext in array creation instead of flag
• GetData is dangerous as it only works on CPU, instead, do copy API(backed by copy(from/to)bytes
  • Checkout js api and java api https://docs.tvm.ai/jsdoc/tvm.NDArray.html
  • AsArray
  • CopyFrom

[tqchen]

maybe @yzhliu can also chime in to comment on api design

[srkreddy1238]

Addressed above changes.

• Strings instead of TVMType
• TVMContext exposed to user
• Copy API's used instead of direct access to TVMArray data.

[tqchen]

@srkreddy1238 can you please also update the sample code on the first post?

[srkreddy1238]

Updated the sample code.

[tqchen]

Some of the suggestions to make it more consistent with existing APIs in other languages(js, java)

• If there is concept of namespace(gotvm), we can consider remove TVM prefix, i.e. TVMContext -> Context
• ModLoadFromFile -> LoadModuleFromFile
• EmptyArray -> Empty
• SetData-> CopyFrom(slice)
• GetData -> AsSlice

[tqchen]

• Let us add gotvm.CPU(0) and gotvm.GPU(0) as helper constructors. The example now looks good to me.

We still need to flesh out the details of the PackedFunc system, specifically

• Please also add simple packed function example that handles return values(so we can see the usage of TVMValue)
• We need to handle passing go callbacks as PackedFuncs(see other language APIs)

Please create a simple example to demonstrate cases in https://github.com/dmlc/tvm/blob/master/tests/web/test_packed_func.js

[srkreddy1238]

Ref below from complex.go example

    // Call function
    graphrt, err := funp(jsonStr, modp, (int64)(gotvm.KDLCPU), (int64)(0))
    if err != nil {
        fmt.Print(err)
        return
    }

    graphmod := graphrt.(*gotvm.TVMModule)

    fmt.Printf("Graph runtime Created\n")

Packed Function return is handled and it return an interface holding the value based on the ret_type_code from TVM runtime.

• I will work on amending the target helpers and go callbacks.

[tqchen]

according to java's wrapping, we can introduce TVMValue object as an intermediate variant type, and use AsXX function to do the explicit conversion with type checking, and the call will look like

graphrt, err:= ..
graphmod = graphrt.AsModule()

[srkreddy1238]

Sure, I will refer java wrappers and add them in golang too.

[tqchen]

Please update the examples once you have an updated API and I will check again

[srkreddy1238]

Updated API interface above with wrappers on TVMValue and TVMContext wrappers API.

Please find below complex example which use these wrappers.

/*!
 *  Copyright (c) 2018 by Contributors
 * \brief Sample golang application deployment over tvm.
 * \file complex.go
 */

package main

import (
    "fmt"
    "io/ioutil"
    "math/rand"
    "./gotvm"
    "runtime"
)

// NNVM compiled model paths.
const (
    modLib    = "./mobilenet.so"
    modJSON   = "./mobilenet.json"
    modParams = "./mobilenet.params"
)

// main
func main() {
    defer runtime.GC()
    // Welcome
    fmt.Printf("TVM Go Interface : v%v\n", gotvm.GoTVMVersion)
    fmt.Printf("TVM Version   : v%v\n", gotvm.TVMVersion)
    fmt.Printf("DLPACK Version: v%v\n\n", gotvm.DLPackVersion)

    // Query global functions available
    funcNames, err := gotvm.FuncListGlobalNames()
    if err != nil {
        fmt.Print(err)
        return
    }

    fmt.Printf("Global Functions:%v\n", funcNames)

    // Import tvm module (so)
    modp, err := gotvm.LoadModuleFromFile(modLib)
    if err != nil {
        fmt.Print(err)
        fmt.Printf("Please copy tvm compiled modules here and update the sample.go accordingly.\n")
        fmt.Printf("You may need to update modLib, modJSON, modParams, tshapeIn, tshapeOut\n")
        return
    }
    fmt.Printf("Module Imported:%p\n", modp)

    bytes, err := ioutil.ReadFile(modJSON)
    if err != nil {
        fmt.Print(err)
        return
    }
    jsonStr := string(bytes)

    // Load module on tvm runtime - call tvm.graph_runtime.create
    funp, err := gotvm.GetGlobalFunction("tvm.graph_runtime.create")
    if err != nil {
        fmt.Print(err)
        return
    }

    // Call function
    graphrt, err := funp(jsonStr, modp, (int64)(gotvm.KDLCPU), (int64)(0))
    if err != nil {
        fmt.Print(err)
        return
    }

    graphmod := graphrt.AsModule()

    fmt.Printf("Graph runtime Created\n")

    // Array allocation attributes
    tshapeIn  := []int64{1, 224, 224, 3}
    tshapeOut := []int64{1, 1001}

    // Allocate input Array
    inX, err := gotvm.Empty(tshapeIn, "float32", gotvm.CPU(0))
    if err != nil {
        fmt.Print(err)
        return
    }

    // Allocate output Array
    out, err := gotvm.Empty(tshapeOut)
    if err != nil {
        fmt.Print(err)
        return
    }
    fmt.Printf("Input and Output Arrays allocated\n")

    // Get module function from graph runtime : load_params
    // Read params
    bytes, err = ioutil.ReadFile(modParams)
    if err != nil {
        fmt.Print(err)
    }
    paramsByteArray := gotvm.NewByteArray(bytes)

    // Load Params
    funp, err = graphmod.GetFunction("load_params")
    if err != nil {
        fmt.Print(err)
        return
    }

    fmt.Printf("Func load_params:%p\n", funp)

    // Call function
    _, err = funp(paramsByteArray)
    if err != nil {
        fmt.Print(err)
        return
    }

    fmt.Printf("Module params loaded\n")

    // Set some data in input Array
    inSlice := make([]float32, (244 * 244 * 3))

    rand.Seed(10)
    rand.Shuffle(len(inSlice), func(i, j int) {inSlice[i],
                                               inSlice[j] = rand.Float32(),
                                               rand.Float32() })

    inX.CopyFrom(inSlice)

    // Set Input
    funp, err = graphmod.GetFunction("set_input")
    if err != nil {
        fmt.Print(err)
        return
    }

    // Call function
    _, err = funp("input", inX)
    if err != nil {
        fmt.Print(err)
        return
    }

    fmt.Printf("Module input is set\n")

    // Run
    funp, err = graphmod.GetFunction("run")
    if err != nil {
        fmt.Print(err)
        return
    }

    // Call function
    _, err = funp()
    if err != nil {
        fmt.Print(err)
        return
    }

    fmt.Printf("Module Executed \n")

    // Call runtime function get_output
    funp, err = graphmod.GetFunction("get_output")
    if err != nil {
        fmt.Print(err)
        return
    }

    // Call function
    _, err = funp(int64(0), out)
    if err != nil {
        fmt.Print(err)
        return
    }
    fmt.Printf("Got Module Output \n")

    // Print results
    outIntf, _ := out.AsSlice()
    outSlice := outIntf.([]float32)
    fmt.Printf("Result:%v\n", outSlice[:10])
}

[tqchen]

The API looks good, please also include examples for how can we use callbacks

[srkreddy1238]

@tqchen

One situation while designing function callbacks for discussion.

The function closure is defined as func (args ...interface{}) (*Value, error) which takes golang types as variadic args and packs them as TVMValue array internally. The return value is TVMValue and AsXXX wrappers are provided to access the underlying values.

I am looking at function callbacks definition as func (args ...interface{}) (interface{}) which takes variadic args and returns an interface.

The signature should be same for both cases.

Reason behind proposing interface{} instead of TVMValue is user don't' need to explicitly embed golang types into TVMValue with wrappers.

Samples:

    // Call function
    graphrt, err := funp(jsonStr, modp, (int64)(gotvm.KDLCPU), (int64)(0))
    if err != nil {
        fmt.Print(err)
        return
    }

    graphmod := graphrt.(*gotvm.TVMModule)

else user need to handle embedding jsonStr, modp ..etc. into TVMValue with functions.

Please advice.

[tqchen]

OK, I think it is fine to use the interface if go users are familiar with that. One second thought on the callbacks, we do need a way to convert functions into handles and pass PackedFunc handle back when there is a function argument. We can do it either by attaching a field to the closure(if go permit that, as in https://github.com/dmlc/tvm/blob/master/web/tvm_runtime.js#L466), or we do need a customized Function struct, and use a special method Call to call the function

[srkreddy1238]

@tqchen

Here is the final sample with packed function system support for golang. Please comment on the //TODO below to finalize.

Also the API above is updated with addition of RegisterFunction , ConvertFunction and Invoke

/*!
 *  Copyright (c) 2018 by Contributors
 * \brief Sample golang application to demonstrate function callbacks in go
 * \file funccb.go
 */

package main

import (
    "fmt"
    "runtime"
    "./gotvm"
    "strings"
)

// sampleCb is a simple golang callback function like C = A + B.
func sampleCb(args ...interface{}) (retVal interface{}, err error) {
    for i, v := range args {
        fmt.Printf("ARGS:%T: %v --- %T : %v\n",i, i, v, v)
    }

    val1 := args[0].(int64)
    val2 := args[1].(int64)

    retVal = int64(val1+val2)

    return
}

// sampleErrorCb is a callback function which returns a golang error.
func sampleErrorCb(args ...interface{}) (retVal interface{}, err error) {

    err = fmt.Errorf("Callback function returns an error\n")

    return
}

// sampleFunctionCb returns a function closure which is embed as packed function in TVMValue.
func sampleFunctionCb(args ...interface{}) (retVal interface{}, err error) {
    funccall := func (cargs ...interface{}) (interface{}, error) {
        return sampleCb(cargs...)
    }

    retVal = funccall

    return
}

// main
func main() {
    // Welcome
    defer runtime.GC()
    fmt.Printf("TVM Go Interface : v%v\n", gotvm.GoTVMVersion)
    fmt.Printf("TVM Version   : v%v\n", gotvm.TVMVersion)
    fmt.Printf("DLPACK Version: v%v\n\n", gotvm.DLPackVersion)

    // Verify callback function by direct call.
    fmt.Printf("\n\n ------ Direct call Test ------ \n")
    retVal, err := sampleCb(int64(10), int64(20))
    fmt.Printf("simpleCb: %v\n", retVal)

    _, err = sampleErrorCb()
    if err == nil {
        fmt.Printf("Expected err but not received\n")
        return
    }

    retVal, err = sampleFunctionCb(int64(15), int64(25))
    fmt.Printf("sampleFunctionCb:%v", retVal)

    fmt.Printf("\n\n ------ Register Function With TVM ------ \n")
    // Register sampleCb with TVM packed function system and call and check Global Function List.
    gotvm.RegisterFunction(sampleCb, "sampleCb");

    // Query global functions available
    funcNames, err := gotvm.FuncListGlobalNames()
    if err != nil {
        fmt.Print(err)
        return
    }

    found := 0
    for ii := range (funcNames) {
        if strings.Compare(funcNames[ii], "sampleCb") == 0 {
            found = 1
        }
    }

    if found == 0 {
        fmt.Printf("Function registerd but, not listed\n")
        return
    }

    // Get "sampleCb" and verify the call.
    funp, err := gotvm.GetGlobalFunction("sampleCb")
    if err != nil {
        fmt.Print(err)
        return
    }

    //TODO: funp here is a function closure.
    //      Do we convert this to Function handle to keep it common across.
    //      New method Invoke can be used to call.

    // Call function
    result, err := funp((int64)(10), (int64)(20))
    if err != nil {
        fmt.Print(err)
        return
    }

    fmt.Printf("sampleCb result: %v\n", result.AsInt64())
    //TODO: Do we still need AsXXX wrappers?
    //      As packed function is defined to return interface{} instead.

    fmt.Printf("\n\n ------ Convert Function With TVM ------ \n")
    // Simple convert to a packed function
    fhandle, err := gotvm.ConvertFunction(sampleErrorCb)
    retVal, err = fhandle.Invoke()

    if err == nil {
        fmt.Printf("Expected err but not received via packed function\n")
    }

    fmt.Printf("Error received as expected as :###%v###\n ", err.Error())

    fmt.Printf("\n\n ------ Function Closure Return Type With TVM ------ \n")
    // Check function closure through packed function system.

    // Not passing a function name implicitely
    // picks the name from reflection as "main.sampleDunctionCb"
    gotvm.RegisterFunction(sampleFunctionCb);

    funp, err = gotvm.GetGlobalFunction("main.sampleFunctionCb")
    if err != nil {
        fmt.Print(err)
        return
    }

    // Call function
    result, err = funp()
    if err != nil {
        fmt.Print(err)
        return
    }

    pfunc := result.AsFunction()
    pfuncRet, err := pfunc.Invoke((int64)(30), (int64)(40))
    fmt.Printf("sampleFunctionCb result:%v\n", pfuncRet.AsInt64())
}

[tqchen]

• Let us keeps the PackedFunc(Function) interface consistent, i.e. GetGlobalFunction should also return PackedFunc, as they can be passed as arguments
• For the same reason, let us either always enforce PackedFunc to return Value or always return the interface, both are fine, but we need to be consistent.
• Please provide examples that involve closure when Function
  • PackedFunc(Function) is passed as argument of a function
  • go-closure is passed as argument of a function(get automatically converted to PackedFunc)
  • go-closure is returned by a PackedFunc callback

[tqchen]

One possible problem of the current callback interface is that the value types seem to be pretty strict, for example, we do pfunc.Invoke((int64)(0)), but ideally, we want to support pfunc.Invoke(0).

One option is to Value as a signature for the implementation of callback could be helpful. So callback's signature takes in Value and call AsXXX during to get the argument and construct Value during return.

The invoke expose the signature (args ...interface{}) Value, and we allow user to directly pass in int/closure/Value/float/Array etc and do the automated conversion to Value under the hood

[srkreddy1238]

To summarize the TODO

• GetGlobalFunction should return TVMFunction (Packed Function) which will be called via Invoke method to be consistent across.

• Packed function implementation signature would be (args ...Value) (interface{}) where AsXXX functions will be helpful to receive arguments and user can return any type.

• Packed Function invoke signature would be (args ...interface{}) (*Value) where user can pass any types as arguments and receives Value where AsXXX helpers will be helpful.

sampleFunctionCb in above example demonstrate go-closure turning into PackedFunc under the hood automatically. I will add more samples for the same.

[tqchen]

One thing to keep in mind is that in the cases where interface{} is used, we also need to support Value as a argument/return

[srkreddy1238]

Yes, Value is handled.

Also covering []Value which is possible when a callback function calling another callback function with slice of it's own arguments.