Bindings for Golang

[reyammer]

Make magika available to Go applications.

[LeslieLeung]

I think I can help with this.

[reyammer]

Awesome! should we then assign this to you so that people know someone is already on it? :-)

[reyammer]

If you actually want to pick this up, great, and let's discuss briefly how you would approach this. There are several ways to deal with this, let's make sure we align first on the approach :-)

[LeslieLeung]

Here are some of my ideas:

• the python version's main logic is extract features from files, then predict with onnx. I think I can rewrite the whole thing in Go, with onnx-go
• the Go version would be available in both package and executable(CLI). With Go, the CLI can be distributed to more platforms more conveniently
• the config and models directory should move to a parent folder, so that other languages(like Rust or PHP) can use them in a similar way

[weebney]

Would love to participate in this.

[reyammer]

Thanks for the interest!

Here are my thoughts:

• ideally, we would like to not reimplement the feature extraction in every language, because we may change that for future versions... and in that case we may need to update the implementation in all these ports. If it's necessary, then we'll need to do it. The question is: is it really necessary? is there a way to have a lower-level implementation of things (e.g., a shared object, .so) and then the different ports rely on the same? Not sure.
• the idea would be to have n bindings for n languages/environments (so to simplify using integration with existing languages), but we would have only one CLI. The current CLI is in python, but we are investigating whether we can have a new CLI in Rust (see #93). This new CLI would replace the existing python one (as we expect it to be 8x/9x/10x faster; tests pending). So, I'm not sure about how a CLI in go would fit in the grand plan?
• config/models are currently copied in the different python/, js/ folders because 1) they may be in different formats, 2) it makes the packaging much easier if all the files are part of the same dir). But, ideally, we would have "the raw model/config" in one place only, and then have all python/js model/config generated starting from that raw data. We didn't have time to implement this. I'm going to open an issue for tracking; but for now I wouldn't worry too much about this and you can assume to have model/config in the same dir?

Feedback is welcome :-)

[LeslieLeung]

I am not very keen on the idea of using shared library on creating bindings for go (just go, not other languages). Using shared library means using CGO(I am not very sure about this, correct me if I am wrong), and CGO has at least these drawbacks compared to native go:

• cross-compiling could be a nightmare
• worse performance

[LeslieLeung]

Note for external contributors: this may become the future de-facto magika client, so its design will need extra care. We are reaching out internally as well, so please let us know if you are starting working on this so that we can coordinate better and/or avoid duplicated work. Thank you!

I wasn't aware of this earlier. A rust implementation would be enough, apologies for any inconvenience caused.

[reyammer]

Hey, no worries! and sorry for delay in following up. Interesting you mentioned the rust impl would be enough... how would a golang app call out magika inference without shelling out to the CLI?

[LeslieLeung]

Hey, no worries! and sorry for delay in following up. Interesting you mentioned the rust impl would be enough... how would a golang app call out magika inference without shelling out to the CLI?

I mean a go CLI would not be necessary if you already have plans for a Rust one.

But for the go package, there are a few different approaches.

1. implement both feature extraction and inference with native go
2. call the Rust CLI in go, executing CLI commands(is this what you meant by "shelling out to the CLI"?)

Personally I prefer the first one because a) the feature extraction logic is not too complicated to implement, b) dependency management would be easier, and c) better performance.

The second one is easier to implement and more agnostic to changes though. But the performance overhead is not ideal since the package is more commonly used in online processing instead of offline CLI/script.

[reyammer]

Hello @LeslieLeung, sorry for the long delay in replying! and thanks for your input! We are preparing a new release, and we are now actively thinking about this go bindings aspect as well; your input is very useful! We will update this issue when we wrap up our thoughts on the topic. Thanks again!