Generated wasm too large

[iarkaroy]

Thank you for your contribution. This looks pretty awesome to work with. I just started playing around with it.

I started with minimal example:

package main

import (
    "log"
    "net/http"

    "github.com/maxence-charriere/go-app/v8/pkg/app"
)

type home struct {
    app.Compo
}

func (h *home) Render() app.UI {
    return app.H1().Text("Hello World!")
}

func main() {
    app.Route("/", &home{})
    app.RunWhenOnBrowser()
    http.Handle("/", &app.Handler{
        Name:        "Hello",
        Description: "An Hello World! example",
    })

    if err := http.ListenAndServe(":8000", nil); err != nil {
        log.Fatal(err)
    }
}

Generated wasm with:

GOARCH=wasm GOOS=js go build -o web/app.wasm

But the generated wasm is a whooping 11.5 MB. Any suggestion to reduce the file size?

Go version: 1.16.4

[maxence-charriere]

You can serve it gzipped.

[iarkaroy]

Yes. Still it will be around 3 MB. I will experiment with tinygo to see if that helps.

[maxence-charriere]

If you dont use http you can separate client code in another package. It the http package that add a lot.

[iarkaroy]

Thanks for the suggestion. I have been trying to build with tinygo but it seems to have some issues. I have added the issue https://github.com/tinygo-org/tinygo/issues/1886.

[maxence-charriere]

Tinygo is not supported by this package for now.

[thepudds]

FWIW, there has been some progress on the tinygo side of this, e.g.:

https://github.com/tinygo-org/tinygo/issues/1886#issuecomment-846629254

[ignishub]

Just make two files that contains "serve" function. In GOOS=js target this function will be empty.

Without it : 12 Mb With it(without http package): 5 Mb

Then use brotli to reduce it again:

Result: 985 Kb

[oderwat]

I am experimenting with go-app right now. To get the sizes down I made two targets for the HTTP server and one is a stub. In addition, I experimenting in using the chi router and with that, I have a compression middleware for the wasm. This brings the size to 1.3 MB (from >16 MB) and lets me add the backend code using chi. I also experiment with "live reloading" (through a javascript HEAD checking and using reflex for recompilation). This is a bit rough about the edges but I am actually pretty happy with how everything works so far. You may want to look at https://github.com/oderwat/go-guess-the-number-app (this is not meant to be public for real, but I thought I want to share this as example and maybe get comments)

[mar1n3r0]

@oderwat You are right. We might end up needing to separate the http package and still use Tinygo to achieve js world bundle sizes. Thanks for the example. Splitting into two files doesn't seem to reduce the size though. Can you elaborate on your results?

[oderwat]

13.3 MB compressed? My testing (a lot of stuff) app has currently 13.8 MB which ends up as 3.2 MB with on-the-fly compression middleware. I also use my #631 PR to show people a loading indicator, which makes its loading time "more accepted" as it shows the progress to the user.

But I hope that we get TinyGo to work. Sadly I just came from my vacation and my day job has different priorities at the moment.

[mar1n3r0]

Hey not a problem. I don't see any difference in the wasm size after splitting into two files actually it's 13.3 MB not compressed and around 3 MB when compressed so no improvement over the original inclusion of the http package. I think when building the wasm the http package is still included.

In the end I think we will need both this technique and Tinygo for best results.

[oderwat]

Well, I use HTTP clients in the frontend anyways. But there are other packages that could be used (maybe https://github.com/golangee/wasm-net). But I did not have time to try it yet.

[oderwat]

That is as a wasm server, not an HTTP client library to use in the frontend? The Server is fine in go-app as it is just the server binary where size matters less.

[oderwat]

Sure. But when you access APIs from the frontend you need to use one or more HTTP clients in the code.

[mar1n3r0]

Yes, my misconception. So we can't save size on that front in an easy straight forward way. It remains a mystery though why the tinygo produced files are still big. Also the http client in react is 2.5 KB in Go it's 7 MB, quite a big difference. Can we find an alternative tiny http client let's say developed for embedded devices?

[oderwat]

Never mind: https://github.com/golangee/wasm-net is not what I meant. This also includes net/http. I found something else but can not dig up the link right now. Basically, it works by calling the functions in the browser as js does it.

[mar1n3r0]

A gRPC client? We already have an example of using that in go-app: https://github.com/maxence-charriere/go-app/issues/447 Even if we use that we still serve requests from net/http and need a lightweight alternative to it.

[oderwat]

It is about "net/http" in the frontend part and you can not just use gRPC to access an HTTP API. I am not talking only about the client/server communication of the app itself.

Even if we use that we still serve requests from net/http and need a lightweight alternative to it.

No, we don't? It does not change the WASM size at all that you use net/http for the server-side.

[mar1n3r0]

It is about "net/http" in the frontend part and you can not just use gRPC to access an HTTP API. I am not talking only about the client/server communication of the app itself.

Even if we use that we still serve requests from net/http and need a lightweight alternative to it.

No, we don't? It does not change the WASM size at all that you use net/http for the server-side.

I meant trying to use gRPC API instead of HTTP API in order to avoid including the net/http package. But we still need something to serve the wasm file tothe browser.

I think it's precisely the net/http package that adds size, actually more than half of it.

[oderwat]

And btw. gRPC uses HTTP/2. I would rather use something like https://github.com/twitchtv/twirp

[mar1n3r0]

And btw. gRPC uses HTTP/2. I would rather use something like https://github.com/twitchtv/twirp

Yeah I am not speculating about that, anything as an alternative to a smaller final size.

[mar1n3r0]

Without it : 12 Mb With it(without http package): 5 Mb

It was stated previously that the net/http package adds 7 MB.

[oderwat]

You can not use gRPC for an HTTP API. We need to access a lot of different APIs and not all of them are made by us. The key to a smaller size will be TinyGo until the Go compiler gets optimized for WASM.

[mar1n3r0]

You can not use gRPC for an HTTP API. We need to access a lot of different APIs and not all of them are made by us. The key to a smaller size will be TinyGo until the Go compiler gets optimized for WASM.

Please read the pull request discussion. Tinygo successfully compiled but the final size was 6.3 MB. So Tinygo doesn't solve the problem with size. It's the net/http package that mainly contributes to the big wasm size.

[oderwat]

It was stated previously that the net/http package adds 7 MB.

And I say you can probably use the browsers XMLHttpRequest() for this and don't need net/http in the frontend. But I don't have time to try it.

[mar1n3r0]

It was stated previously that the net/http package adds 7 MB.

And I say you can probably use the browsers XMLHttpRequest() for this and don't need net/http in the frontend. But I don't have time to try it.

How are we going to serve the wasm file to the browser without http.ListenAndServe?

[mar1n3r0]

If you dont use http you can separate client code in another package. It the http package that add a lot.

I hope @maxence-charriere can clarify the basics here.

[oderwat]

@mar1n3r0 you simply build two times. One for the WASM not containing the server and one for the server itself (which simply uses net/http).

[mar1n3r0]

@mar1n3r0 you simply build two times. One for the WASM not containing the server and one for the server itself (which simply uses net/http).

They are the same package how do you split them? In your example the final size remains unchanged from before the split.

[oderwat]

main.go

func main() {
    // Frontend routing
    app.RouteWithRegexp("/.*", &appControl{})

    // this concludes the part which goes into the front-end
    app.RunWhenOnBrowser()

    // this will depend on the target (wasm or not wasm) and
    // it starts the servers if it is not the wasm target.
    AppServer()
}

wasmstub.go

// Our empty version of the httpServer for usage with the wasm target
// this way we will not include any of the related code
//go:build wasm

package main

func AppServer() {
}

server.go

// Our empty version of the httpServer for usage with the wasm target
// this way we will not include any of the related code
//go:build !wasm

package main

import (
....
)

func AppServer() {
    // the actuall server code
......

[oderwat]

I reverted my sample since I posted this here as it contained too much "tech" which I did for the company.

[mar1n3r0]

main.go

func main() {
  // Frontend routing
  app.RouteWithRegexp("/.*", &appControl{})

  // this concludes the part which goes into the front-end
  app.RunWhenOnBrowser()

  // this will depend on the target (wasm or not wasm) and
  // it starts the servers if it is not the wasm target.
  AppServer()
}

wasmstub.go

// Our empty version of the httpServer for usage with the wasm target
// this way we will not include any of the related code
//go:build wasm

package main

func AppServer() {
}

server.go

// Our empty version of the httpServer for usage with the wasm target
// this way we will not include any of the related code
//go:build !wasm

package main

import (
....
)

func AppServer() {
  // the actuall server code
......

Will give it a try.

[mar1n3r0]

Thanks, that worked like a charm. Here are the results and the example code based on the hello example: https://github.com/mar1n3r0/hello-wasm

1. Original size - 12 MB After multi-stage and omitting net/http in wasm
2. Go compiler - 5 MB
3. TinyGo compiler - 1.8 MB
4. TinyGo + gzipped - 574KB

Still the reduction from TinyGo is not on par with what vecty has achieved but at least we are under the 1 MB threshold. For comparison hellovecty.wasm with TinyGo + gzip is 5 times smaller sitting at 97K. An error occurred of course after removing the env variables but it shows what would be possible once SetEnv is included in the next TinyGo release.

[oderwat]

I don't think that "setenv" has anything to do with the error (besides that I already wondered what "setenv" in WASM may even mean). If you compile with TinyGo you also need to use the TinyGo WASM framework variant. You can not just use this WASM with the WASM of the standard Go compiler.

[mar1n3r0]

Yeah you are right: https://tinygo.org/docs/guides/webassembly/ It needs the wasm_exec.js version from TinyGo.

[oderwat]

I already have a wasm module loader in my go-app tests which can load additional wasm modules generated by the original or the TinyGo compiler. This is why I know that :)

[mar1n3r0]

That's fixed now here: https://github.com/maxence-charriere/go-app/blob/master/pkg/app/gen/scripts.go#L31 No further errors from wasm. It's now the envs that are remaining.

[mar1n3r0]

I don't think that "setenv" has anything to do with the error (besides that I already wondered what "setenv" in WASM may even mean). If you compile with TinyGo you also need to use the TinyGo WASM framework variant. You can not just use this WASM with the WASM of the standard Go compiler.

SetEnv has nothing to do with WASM but it has to be implemented in TinyGo in order to be used.

[oderwat]

If that uses my "template" replacement it may need {{.Env }} changed to {{.Env}} because I just replace later one.

[oderwat]

SetEnv has nothing to do with WASM but it has to be implemented in TinyGo in order to be used.

But I only want to use TinyGo for the WASM part and that hardly ever needs to use "SetEnv" as (afaik) there is nothing like "env variables" in WASM. I didn't had time what this "SetEnv" actually is used for.

The error you get is also template-related.

[mar1n3r0]

This is the original go-app code. Haven't touched anything there.

[oderwat]

How can you compile using TinyGo without a replacement for text/template? ... if you removed that part it will surely not work :)

You need #680 or something similar.

[mar1n3r0]

I used your hard-coding from the PR and also commented this:

`func (h *Handler) makeAppJS() []byte { if h.Env == nil { h.Env = make(map[string]string) } // internalURLs, _ := json.Marshal(h.InternalURLs) // h.Env["GOAPP_INTERNAL_URLS"] = string(internalURLs) // h.Env["GOAPP_VERSION"] = h.Version // h.Env["GOAPP_STATIC_RESOURCES_URL"] = h.Resources.Static() // h.Env["GOAPP_ROOT_PREFIX"] = h.Resources.Package()

// for k, v := range h.Env {
//  if err := os.Setenv(k, v); err != nil {
//      Log(errors.New("setting app env variable failed").
//          Tag("name", k).
//          Tag("value", v).
//          Wrap(err))
//  }
// }

env, err := json.Marshal(h.Env)
if err != nil {
    panic(errors.New("encoding pwa env failed").
        Tag("env", h.Env).
        Wrap(err),
    )
}

s := appJS
s = strings.ReplaceAll(s, "{{.Env}}", btos(env))
s = strings.ReplaceAll(s, "{{.Wasm}}", h.Resources.AppWASM())
s = strings.ReplaceAll(s, "{{.WorkerJS}}", h.resolvePackagePath("/app-worker.js"))
return []byte(s)

}`

So yeah it's normal that it's not working since the env variables are set here by SetEnv.

[oderwat]

No there is a {{.Env }} in the output, you show and that is the "syntax error" and related to the templates not correctly translated.

s = strings.ReplaceAll(s, "{{.Env}}", btos(env))

Looks to me as if generate did not run or that somewhere a {{.Env }} is hiding or reintrocuded.

[mar1n3r0]

Yeah because {{.Env }} was not replaced by the env variable which is not set. If i remove the comments it starts working so it's certainly there.

[oderwat]

And please don't comment out:

// internalURLs, _ := json.Marshal(h.InternalURLs)
// h.Env["GOAPP_INTERNAL_URLS"] = string(internalURLs)
// h.Env["GOAPP_VERSION"] = h.Version
// h.Env["GOAPP_STATIC_RESOURCES_URL"] = h.Resources.Static()
// h.Env["GOAPP_ROOT_PREFIX"] = h.Resources.Package()

It is just os.SetEnv() which does not work (and looks useless to me. I don't understand what this is good for when the target is WASM. But maybe it is compiled twice. I don't know (yet)

[mar1n3r0]

// Getenv retrieves the value of the environment variable named by the key. It
// returns the value, which will be empty if the variable is not present.
func Getenv(k string) string {
    if IsServer {
        return os.Getenv(k)
    }

    env := Window().Call("goappGetenv", k)
    if !env.Truthy() {
        return ""
    }
    return env.String()
}

[oderwat]

{{.Env}} is not replaced by an Env variable but with the value of the map which is defined about.

[mar1n3r0]

That's that for now until SetEnv is implemented in TinyGo.

[oderwat]

The IsServer part should never run when wasm is the target so maybe put this in an extra file similar to the server code itself, so it will not include the call at all.