Golang that uses stdio and are piped , like Unix

[gedw99]

hey @brendanburns @brendandburns

I lots of golang packages that use std io .

they are piped together on the shell and the last one pipes out to file or shell .

So I was wondering is this is possible to support ? I am not sure if it’s part of the wasi standard . But it’s very standard since every is can do piping and stdio .

the reason for WASM is:

sandboxing

schema evolution . The raw data ed into the WASM could be v1 or v2, so by inspecting the first line of the data being fed in, I can load up the right version of the WASM that was compiled for that version of the data .

[gedw99]

I can add an example if the code as an example if it helps for testing too . Just 20 lines of code for each of the 3 that are piped together

[brendandburns]

wasi does support stdio and stdout. It's a little outside of the scope of this repository, but we could add an example to the https://github.com/dev-wasm/dev-wasm-go repository.

There's a simple example that just uses stdout here: https://github.com/dev-wasm/dev-wasm-go/tree/main/examples/component_main

[gedw99]

This is great. I use stdio for all golang entry points to make it reusable.

But that https://github.com/dev-wasm/dev-wasm-go/blob/main/examples/component_main/Makefile looks scary still. :)

Would be great if this repo using wazero could do it, because then it will just work everywhere; even SBC computers and even in a browser.

I like how I can run it on any OS and pipe things and debug easily. My stuff is all pure golang using only std lib. No http. No file stuff. Just pure take data in from stdio, process it, and send it out over stdio.

So then I could compile the golang to wasm ( or wasi ), and then have caddy wasm server so the same.

what follows is some golang quasi code for this pattern I do. It does have file output though.

unix piping of cm01 to cmd02

´´´sh

cmd01 $(TEST_NAME).dsl | cmd02 -stdout

cmd01 $(TEST_NAME).dsl | cmd02 -stdout - > filename.jpg

cmd01:

// $ cmd01                    # input from stdin, output to stdout
// $ cmd01 -o foo.xml         # input from stdin, output to foo.xml
// $ cmd01 foo.dsl            # input from foo.dsl output to stdout
// $ cmd01 -o foo.xml foo.dsl # input from foo.dsl output to foo.xml
// $ cmd01 -version           # show cmd01 version
func main() {

    dest := flag.String("o", "", "output destination")
    version := flag.Bool("version", false, "show version")
    var input io.ReadCloser = os.Stdin
    var output io.WriteCloser = os.Stdout
    var rerr, werr error

    flag.Parse()

    if *version {
        l := len(Version)
        fmt.Fprintln(os.Stderr, Version[1:l-1])
        os.Exit(1)
    }

    if len(flag.Args()) > 0 {
        input, rerr = os.Open(flag.Args()[0])
        if rerr != nil {
            fmt.Fprintf(os.Stderr, "%v\n", rerr)
            os.Exit(1)
        }
    }

    if len(*dest) > 0 {
        output, werr = os.Create(*dest)
        if werr != nil {
            fmt.Fprintf(os.Stderr, "%v\n", werr)
            os.Exit(2)
        }
    }

    // do the compute on the data...
    err := Process(output, input)
    if err != nil {
        fmt.Fprintf(os.Stderr, "%v\n", err)
        os.Exit(3)
    }

    input.Close()
    output.Close()
    os.Exit(0)
}

[gedw99]

was thinking that making a cmd01 and cmd02 example might be useful to test against ?

could eventually just make a generator to make it easy for golang devs to make mini programs. It woudl just gen the entry point skeleton and import and call some golang module that just does compute with no side effects.

[brendandburns]

Caddy is specifically about serving HTTP. It sounds like you want to just to pipes in and out from the WASM binary.

I would take a look at this example using wazero:

https://github.com/tetratelabs/wazero/tree/main/examples/basic

I think that you could use that as the basis for the tool that you want to build.

[gedw99]

Thank I will take a look

I got the cmd’s piping through benthos a day ago. Then out to http

benthos has a WASM plugin too so I might try that.

it’s a stateless runner that can be changed at runtime .

[gedw99]

The whole reason I am using WASM is so that I can inspect the schema of the DSL, and use the golang / WASM that was compiled for that schema . Shema evolution cheating.

I store the WASM in nats . So it gets distributed in real time to servers . I use a web hook off GitHub to update the WASM in nats .

another pipe eh :)

[gedw99]

I might use nats as well as http / see as the outer transport . So then the clients all stay in sync when a new DSL or DSL fragment is sent into the system .

this is for the EU Open Science project . Code will be open eventually . The scientist run all this stuff on their own on premise boxes , so use nats leaf nodes for their boxes , then pipe to cloud bats and out to the world