search

cue-lang / cue

The home of the CUE language! Validate and define text-based and dynamic configuration
https://cuelang.org
Apache License 2.0
4.91k stars 278 forks source link

Cannot run the WASM demos #2924

Closed fgm closed 1 month ago

fgm commented 4 months ago

What version of CUE are you using (cue version)?

$ cue version
cue version v0.0.0-20240307083748-cdcb6fa19bf3

go version go1.22.0
      -buildmode exe
       -compiler gc
  DefaultGODEBUG httplaxcontentlength=1,httpmuxgo121=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1
     CGO_ENABLED 1
          GOARCH arm64
            GOOS darwin
             vcs git
    vcs.revision cdcb6fa19bf332283500711ce5381df1cc1935e7
        vcs.time 2024-03-07T08:37:48Z
    vcs.modified true

Does this issue reproduce with the latest stable release?

Yes, but slightly differently.

What did you do?

[dependencies]

[lib] crate-type = ["cdylib"]

- I built in two different ways, generating the `hello.wasm` both times, and tried the commands after that one with both version, with identical results
  - `cargo build --target wasm32-wasi`
  - `cargo build --target wasm32-unknown-unknown`
- copy the `hello.wasm` file to the project directory containing the `wasmer.cue` file:
```cuelang
@extern("wasm")
package p

add: _ @extern("hello.wasm", abi=c, sig="func(int64, int64): int64")
isPrime: _ @extern("hello.wasm", abi=c, name=is_prime, sig="func(uint64): bool")

x0: add(1, 2)
x1: add(-1, 2)
x2: add(100, 1)
b1: isPrime(127)
b2: isPrime(128)

What did you expect to see?

What did you see instead?

Showing there was no attempt to stat or open the hello.wasm file.

What more did you do ?

I ran cue master HEAD under debugger, and found out:

Anything else ?

myitcv commented 4 months ago

Thanks, @fgm - I've copied in @4ad

4ad commented 4 months ago

Hmm, can you try without using CUE modules?

4ad commented 4 months ago

To answer your ancillary questions more specifically:

I am following this Rust path to evaluate WASM integration, but if I have to use WASM, I would prefer to use gc or tinygo because the functions I need to use are written in Go, and it would avoid having to rewrite them in Rust

Gc is impossible to use because it can't export functions yet (see golang/go/#65199).

Tinygo is not guaranteed to work because it doesn't have a stable, or a standard ABI. Realistically only C and Rust can be expected to work at this point.

Additionally, there is no support for strings yet, which I imagine you might need.

  • I am exploring this WASM solution because the current mechanism is to
    • fork cue
    • add our functions in a net2 Go package
    • add the package to pkg/packages.txt
    • go generate ./pkg
    • add that package to pkg/register.go until registration becomes automatic autogenerated as in the TODO in gen.go#24
    • go build -o cue08 cuelang.org/go/cmd/cue
  • We would very much prefer a dynamic solution that does not require maintaining a Cue fork

Do you need to use the CUE command line tool, or would a Go API to inject additional functions suffice? We could potentially make it easy to add functions via the API, but of course, if you need to use the command line tool rather than the API this would not help you.

fgm commented 4 months ago

@4ad Thanks for answering so fast. I get the same behavior if I remove the module (cue.mod/), that was how I initially did it, actually : I only added the cue.mod/ to check if that fixed things.

Using the CUE CLI would be best, but using a Go API to inject additional functions might probably be sufficient: the devs in that team mostly use the CLI to debug their configurations, but the actual logic using the validations is in custom Go code, so that should not be a major issue.

I noticed you mention C and Rust: AIUI gccgo can work with the C ABI : is that correct and could that be a solution ?

4ad commented 4 months ago

I might be wrong, but I don't believe gccgo supports Wasm at all.

Additionally, the "C ABI" for Wasm is not defined anywhere really, so every compiler does something slightly different. Sometimes even the same compiler can do something different (see wasm32-wasi vs. wasm32-unknown-unknown above). That's the problem with Wasm, really, and the reason why we want to move to the Component Model as soon as it is feasible.

fgm commented 4 months ago

OK, so it would be either the existing fork-based solution (short term), or a way to add custom functions in Go for custom apps (second best, if you think that is something which is easily doable), or WASM from Rust once we figure out why I cannot run it yet, then.

Is there an existing issue I could look-at/help-with for the second solution ?

fgm commented 4 months ago

OK, so I found the problem: i was running, as shown: cue eval wasmer.cue or cue08 eval wasmer.cue, and got the errrors above.

Running simply cue eval, cue08 eval, cue eval ., or cue08 eval . there is no problem, the code works as expected. That was similar to running a single Go file vs running a package.

I found the info by dissecting what basic.txtar actually did.

4ad commented 4 months ago

Ah, yes, that makes perfect sense.

fgm commented 4 months ago

FWIW, -buildmode=c-shared i was just accepted in https://github.com/golang/go/issues/65199 so it's now only a matter of time before Go can export functions. So maybe that will mean the ability to use Go-sourced WASM for functions in some weeks/months.

4ad commented 4 months ago

Kind of. The Go compiler does not use the C ABI, so we will need to add a Go-specific ABI, it wouldn't Just Work. But yes.

fgm commented 4 months ago

Hypothetically, any idea of the amount of work it would take should I try to build this to Cue ?

4ad commented 4 months ago

The problem is that the only way to get the memory layout of Go structs is via the -asmhdr flag of the Go compiler. Since we need this information it would most likely mean we need to compile the Go code on the user's behalf, since we can't expect the user to provide us with this information.

We also need a way for this information to find its way into CUE. Perhaps we could add it to CUE struct fields as an attribute. Of course we don't want users to have to write this by hand, so we would need some sort of converter from the -asmhdr output to CUE.

On a technical side of things, making use of this information (once we have it in CUE) is probably quite easy. The more difficult part is making an ergonomic UI/UX around it for users, and plumbing all the pieces together.

In other words the difficulty lies in designing the feature rather than implementing it.

fgm commented 1 month ago

@4ad you just marked this as completed : is it just routine maintenance, or any specific change ? Related to #2035 AFAICS.

4ad commented 1 month ago

Apologies, I wrote a message but it somehow got erased when I closed the issue.

This wasn't really a bug as Wasm was intended to be used with full packages and not with individual files.

But we have removed Wasm support from the CUE command itself (API support still present), so this issue doesn't even serve as a feature request for the forseeable future. We're waiting for Component Model support and when (if) that will arrive it will necessarily have a different UX in CUE.