JS bindings (GopherJS)

[gavinmcfarland]

Hi,

I'm trying to create a wrapper for a node js support using the go-jsonnet version of jsonnet rather than the the C++ version. I'm trying to use gopherjs to transpile go-jsonnet which can be used for node.js applications. I'm hoping by doing this, it will prove faster than the C++ version using emscripten and 🤞will also support imports.

I have an example go app setup following some advice from this article

package main

//go:generate gopherjs build --minify

// This is an experiment to see if gopherjs can reasonably generate js code from go source
// so that we can have a single-source solution for keys and addresses.
// Use "go generate" to build this.

import (
    "crypto/sha256"

    "github.com/gopherjs/gopherjs/js"
)

func main() {
    js.Module.Get("exports").Set("hashit", hashit)
}

func hashit(s string) []byte {
    h := sha256.New()
    h.Write([]byte(s))
    return h.Sum(nil)
}

However I'm unsure what I need to reference from jsonnet after following these instructions:

git clone github.com/google/go-jsonnet
cd go-jsonnet
go build ./cmd/jsonnet

Is there any chance anyone can point me in the right direction?

[sbarzowski]

This may be very helpful. We wanted to do that to use on the website (emscripten version was often problematic).

I'm assuming by "what I need to reference from Jsonnet" you mean the instructions for using the Jsonnet interpreter s a library to run some Jsonnet programs. So, to run Jsonnet program you need to create the VM using MakeVM, then optionally configure it using ExtVar, TlaVar etc. and then you can run some Jsonnet program (passed as string) using EvaluateSnippet.

Let me know if you have any other questions.

[gavinmcfarland]

I'm completely out of my depth here as I've never built anything in Go. Looks like I might have to learn how to build applications in Go and maybe I can have an attempt at it. Your instructions make complete sense high level, although no sense to me at the moment. It might be sometime before I get round to making this work. Thank you though. Maybe these instructions will be useful to someone else as well.

[buremba]

There is more info about GopherJS in this thread: https://groups.google.com/forum/#!topic/jsonnet/FjZoPFvdTaA As we now have the formatter in the Go version, it would be great if we can add GopherJS support to the library and publish the compiled JS as part of the package.

[buremba]

We have been using GopherJS compiled version so far but as we have more files and codebase, it started to suffer from default recursion limits JS runtime in Chrome. It looks like GopherJS is not actively maintained so we switched from GopherJS to WebAssembly. The performance is close to the Go version now but the browser you want to run Jsonnet needs to support Webassembly.

In case anyone wants to try it out, here is the proof-of-concept snippet for the bridge:

// MemoryImporter "imports" data from an in-memory map.
type RecipeMemoryImporter struct {
    files               map[string]string
    foundAtVerification map[string]jsonnet.Contents
    libraries           map[string]string
}

// Import fetches data from a map entry.
// All paths are treated as absolute keys.
func (importer *RecipeMemoryImporter) Import(importedFrom, importedPath string) (contents jsonnet.Contents, foundAt string, err error) {
    fileRootPath := filepath.Dir(fmt.Sprintf("%s", importedFrom))
    fullFilePath := filepath.Clean(fmt.Sprintf("%s/%s", fileRootPath, importedPath))
    // Return from cache
    if cached, importedBefore := importer.foundAtVerification[fullFilePath]; importedBefore {
        return cached, importedPath, nil
    }

    fileContent, exists := importer.libraries[fullFilePath]
    if !exists {
        a, b := importer.files[fullFilePath]
        fileContent = a
        exists = b
    }

    if exists {
        jsonnetContent := jsonnet.MakeContents(fileContent)
        importer.foundAtVerification[fullFilePath] = jsonnetContent // Cache the contents
        return jsonnetContent, importedPath, nil
    } else {
        return jsonnet.Contents{}, "", fmt.Errorf("import not available %v", fullFilePath)
    }
}

func objectToMap(value js.Value) map[string]string {
    fileNames := objectKeysInvoker.Invoke(value)
    fileNamesLength := fileNames.Get("length").Int()
    files := make(map[string]string)
    for ; fileNamesLength > 0; fileNamesLength-- {
        filename := fileNames.Index(fileNamesLength - 1).String()
        files[filename] = value.Get(filename).String()
    }
    return files
}

func buildRange(beginLocation ast.Location, endLocation ast.Location) map[string]interface{} {
    jsLine := make(map[string]interface{})

    start := make(map[string]interface{})
    end := make(map[string]interface{})

    jsLine["start"] = start
    jsLine["end"] = end

    start["character"] = beginLocation.Column - 1
    start["line"] = beginLocation.Line - 1

    end["character"] = endLocation.Column - 1
    end["line"] = endLocation.Line - 1

    return jsLine
}

var objectKeysInvoker = js.Global().Get("Object").Get("keys")

func compile(this js.Value, p []js.Value) interface{} {
    filename := p[0].String()
    files := objectToMap(p[1].JSValue())
    extCodes := p[2].JSValue()
    tlaVars := p[3].JSValue()
    libraries := objectToMap(p[4].JSValue())

    value := make(map[string]interface{})
    node, err := jsonnet.SnippetToAST(filename, files[filename])

    if err != nil {
        value["error"] = err.Error()

        switch err := err.(type) {
        case errors.StaticError:
            loc := err.Loc()

            value["line"] = buildRange(loc.Begin, loc.End)
        }
    } else {
        var vm = jsonnet.MakeVM()

        memoryImporter := RecipeMemoryImporter{}
        memoryImporter.files = files
        memoryImporter.libraries = libraries
        memoryImporter.foundAtVerification = make(map[string]jsonnet.Contents)
        vm.Importer(&memoryImporter)

        extCodeList := objectKeysInvoker.Invoke(extCodes)
        tlaVarsList := objectKeysInvoker.Invoke(tlaVars)
        extCodeListLength := extCodeList.Get("length").Int()
        tlaVarsListLength := tlaVarsList.Get("length").Int()

        for ; extCodeListLength > 0; extCodeListLength-- {
            key := extCodeList.Index(extCodeListLength - 1).String()
            vm.ExtCode(key, extCodes.Get(key).String())
        }

        for ; tlaVarsListLength > 0; tlaVarsListLength-- {
            key := tlaVarsList.Index(tlaVarsListLength - 1).String()
            vm.ExtCode(key, tlaVars.Get(key).String())
        }

        evaluate, err := vm.Evaluate(node)
        if err != nil {
            value["error"] = err.Error()
        } else {
            value["result"] = evaluate
        }
    }

    return js.ValueOf(value)
}

func main() {
    c := make(chan struct{}, 0)
    js.Global().Set("compile", js.FuncOf(compile))

    <-c
}

Once you compile the Webassembly, you can simply run the following JS function:

compile('test.json', {'test.json': '{a:1}'}, {}, {}, {})

The output should be as follows:

> {result: "{  "a": 1 } "}

[sbarzowski]

@buremba This is really cool. I think with some minor changes we could start using it for the Jsonnet website. The implementation doesn't seem complete though. For example I don't see Import method for the RecipeMemoryImporter. Anyway, it is a great starting point.

[buremba]

@sbarzowski just updated the snippet, sorry for missing that part.

[iwoloschin]

@buremba I was wondering if you ever got this working? I see there's https://github.com/rakam-io/monaco-jsonnet, but I'm not actually really sure how to use it (or the above snippet for the matter). I am specifically interested in getting jsonnet support for monaco, so your repo sounds perfect, if I just knew how to use it 😄.

[buremba]

Hey @iwoloschin. Yes, I got it working. :) There is still a lot to do but you can see the editor on https://app.rakam.io. Here is a video from the editor: https://share.vidyard.com/watch/qLA884ibjpXW6dhzTWFe6Y

[iwoloschin]

Awesome! Any chance you can share some instructions, or even just some code? I tried pulling down the monaco-jsonnet repo but when I try to run npm run compile it spews a few errors, and I might have missed something but it was not entirely clear how to integrate that with monaco. Thanks!

[buremba]

@iwoloschin you can use the npm package (https://www.npmjs.com/package/monaco-jsonnet) in your app via Webpack (https://www.npmjs.com/package/monaco-editor-webpack-plugin-with-jsonnet)

I didn't write any documentation but the integration is the same as using monaco-editor-webpack-plugin as stated here: https://github.com/microsoft/monaco-editor/blob/master/docs/integrate-esm.md

[iwoloschin]

@buremba thanks for the pointers. I took a look at this some more yesterday and still can't figure it out, the issue is probably me not being very familiar with frontend development.

I'm trying to make this work in an existing Angular application, which probably is just making everything more complicated. I think I'm failing to understand how to actually set up the editor instance to use the jsonnet mode/workers. Any other tips you can think of?

I'll try to set up a simplified demo if I can, might be easier to work against that?

[iwoloschin]

Here's a quick StackBlitz I threw together:

https://stackblitz.com/edit/materia-ngx-monaco-editor-example-p7q2z8?devtoolsheight=33&file=src/app/app.component.ts

I'm clearly not linking the monaco editor & jsonnet package together, but I'm not really sure how to do that in my case.

[AlJohri]

now that there is a wasm build here: https://github.com/google/go-jsonnet/pull/561

would it be possible to publish an official package to npm with js bindings?