[question] How do I expose function to wasm?

[konsumer]

I assume it is something with m3_LinkRawFunction, but I can't quite work out how I would expose a nim function to the wasm. I am happy to add examples to the README.

Also, this may be way out of scope for this project, but if you know, it would be very helpful: How do I pass string-pointers in/out of wasm?

There are 4 kinds of string things I might want to do:

• get the value of a string-pointer returned from wasm-function
• pass a string-pointer as an argument to a wasm-function (and get the value, in wasm)
• allow wasm to call a nim-function with a string pointer (and get the value, in nim)
• return a string-pointer from a nim-function to wasm (and get the value, in wasm)

I could not figure out how to do these in C, but I feel like it must be possible (I can do them in js host.)

[beef331]

I have not presently got around to making it simpler to do, but in this project I do show how one accesses data at a pointer from the host.

To expose a function you can refer to https://github.com/beef331/wasm3/blob/master/tests/test1.nim#L100 and https://github.com/beef331/wasm3/blob/master/tests/test1.nim#L47

The callWasm macro presently is very simple so only works in the most simple cases, but I do intend on making it much more usable macro. Otherwise you have to manually move the stack pointer that the host procedure gets. Where each value is stored in a int64 that's stored contiguously so a proc that is (int32, int32): int32 would have the first value on the stack pointer be the return address, the next two would be the input parameters.

[konsumer]

Ooh, I like how you have easy access to mem with wasmHostProc. This seems pretty cool. I will try to make more examples, and add to README, if you like.

[konsumer]

Still trying to work through this. Trying to make a simple string-log function, exposed to wasm.

I have this, wasm-side:

proc null0_log(msgPtr: string) {.importC.}

And I do this, in host:

import unittest
import wasm3
import wasm3/wasm3c

proc null0_log(runtime: PRuntime; ctx: PImportContext; sp: ptr uint64; mem: pointer): pointer {.cdecl.} =
  proc null0_log_inside(message: string) = echo(message)
  callWasm(null0_log_inside, sp, mem)

let env = loadWasmEnv(readFile("example.wasm"), hostProcs = [wasmHostProc("*", "null0_log", "i(i)", null0_log)])

/usr/local/Cellar/nim/1.6.10/nim/lib/std/genasts.nim(87, 13) Error: illformed AST:
       Tip: 5 messages have been suppressed, use --verbose to show them.
     Error: Build failed for package: null0
        ... Execution failed with exit code 256
        ... Command: /usr/local/Cellar/nim/1.6.10/nim/bin/nim c --colors:on --noNimblePath -d:NimblePkgVersion=0.0.1 --path:/Users/konsumer/.nimble/pkgs/wasm3-0.1.1 --path:/Users/konsumer/.nimble/pkgs/micros-0.1.7 --hints:off -o:/Users/konsumer/Desktop/null0/null0 /Users/konsumer/Desktop/null0/src/null0.nim

Am I still missing a step?

[beef331]

Like i said callWasm is presently limited, also you will not be able to just pass a string like that. Wasm uses 32bit integers for capacity and length but your host is likely 64bit. Not to mention that the automatic memory management of both programs might get in the way. it's best to make a proc that takes a ptr UncheckedArray[char], uint32 so that it can work with all languages easily.

[konsumer]

ptr UncheckedArray[char], uint32 seems like a good direction, thanks. I had similar confusion in C/wasm3. I think I need to work out the C/nim equivalent of these functions. If I understand it, __lowerBuffer is calling __new to make a pointer in wasm, then copying data into it. If I just used a buffer pointer + length for all strings, that would probly be more universal.

My overall goal is a game engine that can run user's wasm. I made this but am now turning it into a libretro core and adding lots of language-features (make it a bit more like love2d.) I originally got around all these concerns by mostly just passing numbers, so things like string readFile(name: string) are making me have to dig into lots more of this. I appreciate the help.

[beef331]

My overall goal is a game engine that can run user's wasm.

Yea my goal with wasm is similarly related, I want it for game modding. A language agnostic scripting backend is fantastic in my view. You did get me to expand some of the planned ideas. callWasm is now callHost and now there is a whole mixin'd generic system for calling host functions (should be simpler now to add more types).

so things like string readFile(name: string) are making me have to dig into lots more of this. I appreciate the help.

Yea it's generally best to just think of the ABI as the same as the C ABI, so you lower types down to their lowest level. A set[YourEnum] would be like a uint8 or whatever the smallest int that matches. Any dynamic allocated collection is a ptr T, len and so fourth. It's just properly agnostic that way.

[konsumer]

It's just properly agnostic that way.

I like this direction.

would be like a uint8 or whatever the smallest int that matches

but like that is a max string-size of 255 bytes, right? why not uint32? I think I will need more bytes for http and file-reading and stuff.

I think for returns I would need to "bundle" the pointer/length, somehow though, right? Like you can only return one thing.

In assemblyscript, they encode strings by making pointer-1 the length, so if you have the wasm-pointer, you also have the length (take the byte before it.) They also use WTF16, but they have a method to encode a buffer as UTF8, so I was returning a pointer to the UTF8 buffer, and using null-endings (for better interop with C.) I didn't work out how to pull strings out of wasm memory, C-side (for like a return) but it works in a wrapped function. A similar approach might be useful here. If I just had 1 way to encode strings across all languages, I could make lil adapters for all of them, in my game-header. Maybe something like this:

• first bytes length (u32)
• wasm-pointer (u32, wasm-side) to UTF8 bytes

[beef331]

I was saying for a set[Enum] it'd be uint8 or the smallest integer it fits in. Anyway I think I can close this issue now.

[konsumer]

Originally I was doing al this with C and assemblyscript, so I have some examples of how I did it there:

• C-side log function
• Example sending the UTF8 string to host, in aassemblyscript

This hits the usecase of "allow wasm to call a nim-function with a string pointer (and get the value, in nim)"

[konsumer]

I was saying for a set[Enum] it'd be uint8 or the smallest integer it fits in

Ah, ok, I gotcha.

[konsumer]

I think I am going to go back to C, and try to work out this string protocol, then "port" it to nim. It may be a bit simpler to see it working with a few different wasm-side languages, and have a working host to compare against.

[beef331]

Looking at your other code, and yea you could use a cstring in Nim to the same effect, but that means it has to be null terminated and you dont have O(1) length. So if that's fine for your use case it's fine to use.

[konsumer]

I finally worked out some usecases around this in C. It came down to this:

// HOST -> WASM via return
static m3ApiRawFunction (test_string_get) {
  printf("test_string_get was called.\n");
  m3ApiReturnType (uint32_t);
  uint32_t wPointer;

  // example of what you want to return
  char* buffer = "Hello from C host.";

  // lowerBuffer
  size_t s = strlen(buffer) + 1;
  null0_check_wasm3(m3_CallV (wmalloc, s));
  m3_GetResultsV(wmalloc, &wPointer);
  char* wBuffer = m3ApiOffsetToPtr(wPointer);
  memcpy(wBuffer, buffer, s);

  m3ApiReturn(wPointer);
  m3ApiSuccess();
}

I am going to try to flesh out the rest of my string-usecases, and then work on porting them to nim. Thank you for your help!

Maybe after I have a clearer idea of how it all fits together, I can make some c/nim macros to make it easier? It occurs to me that if the returntype was string or char* or whatever, this could be auto-generated like this, and it would look more like it does when returning an int. I think similar macros could be done for params, and it would be very easy to follow.

Actually, this already works for args:

static m3ApiRawFunction (null0_log) {
  m3ApiGetArgMem(const char*, message);
  printf("Log from WASM: %s\n", message);
  m3ApiSuccess();
}

so maybe just:

static m3ApiRawFunction (test_string_get) {
  m3ApiReturnType (const char*);
  const char* buffer = "Hello from C host.";
  m3ApiReturn(buffer);
  m3ApiSuccess();
}

If m3ApiGetArgMem/m3ApiGetArg were also merged (based on if it's a pointer-type) I think it would also be easier, like your callback just defines ins/outs and they just work, but maybe alternately, I could make a m3ApiReturnTypeMem to denote that's a wasm-side pointer, and then the API would match, at least.

I mean to take it further, a macro could guess all the m3ApiGetArgMem/m3ApiGetArg//m3ApiReturnTypeMem/m3ApiReturnType calls from the function itself, right?:

const char* test_string_get() {
  const char* buffer = "Hello from C host.";
  return buffer;
}
wasmFunction (test_string_get);

where wasmFunction would figure out args and return-type mappings. Like in this case, it would say "const char* is a pointer to a type that is not supported in wasm, so I am going to have to lowerBuffer, and return the wasm-pointer."

I am not good with macros in nim or C, so I will look into it, but if this seems sane, it could make things much easier to work with.

[beef331]

Based off of your questioning I've implemented a test program using callHost and one done manually to showcase how one would do it in Nim.

https://github.com/beef331/wasm3/blob/master/tests/test1.nim#L144-L167

For custom types you may want to implement your own fromWasm hook equal to the ones found here: https://github.com/beef331/wasm3/blob/master/src/wasm3/wasmconversions.nim#L15-L37

The issue with automatically allocating things is you do not have the Wasm environment and cannot call functions of it, so I do not know if one can allocate easily inside a host function without global variables.

[konsumer]

I love this. Nim is such a cool language!

[beef331]

I should warn that the string/ seq / openarray variations of fromWasm probably do not work, so be cautious :P

[konsumer]

An update on the C-side:

I have tests for all of these use-cases, that work in JS host here:

1. WASM -> HOST via param (null0_log)
2. HOST -> WASM via return (test_string_get)
3. WASM -> HOST via return (test_string_retstring)
4. HOST -> WASM via param (test_string_param)

The last 2 are not working in C, since I don't think m3ApiOffsetToPtr works outside of m3ApiRawFunction is there another way to get access to the wasm-memory?

Update: I think I got it with M3Memory* _mem = &runtime->memory; Sorry for all the questions.

[beef331]

Just to update, I've cleaned a bit of code up a bit and have expanded the tests for wasmconversions. Now one can do https://github.com/beef331/wasm3/blob/945317223568954f3699806f70386a7e70694d2b/tests/test1.nim#L112 assuming there are the required fromWasm procedures, and it'll just work.