Closed danec020 closed 3 years ago
Hi danec020,
The short answer to you question is that youprobably want to use
the new way to create standalone wasm modules: -s STANDALONE_WASM
.
Using SIDE_MODULE to achieve this is probably a bad idea because it generates relocation code which you probably don't need for your use case.
There is some documentation here: https://github.com/emscripten-core/emscripten/wiki/WebAssembly-Standalone
I should mention that emscripten is very aggressive a DCE (dead code
elimination) so
in -Os
mode it will already strip away any JS code or wasm code that you
don't explicitly
use/need.
cheers. sam
On Sun, Nov 24, 2019 at 11:45 AM danec020 notifications@github.com wrote:
I am following a book and trying to learn WebAssembly but things have obviously changed since the book was written. I am trying to create an exported C++ project that doesn't use the standard library and instead uses my own functions. That way we can strip the unneeded includes out. This is the method I am using to compile the code. I am also doing the JS code myselft to understand creating promises and importing the wasm files.
Here is the old example's generated WAT file compared to the newly generated WAT file. You can see that the global fields of an int and a struct are exported as functions for some reason. At the end is the cpp file that has these exports. Any help on how I can prevent this if it is not supposed to happen or how should I correctly import them in the Promise? At the very bottom is my import code in JS.
Compile script emcc side_module_system_functions.cpp validate.cpp -s WASM_ASYNC_COMPILATION=0 -s SIDE_MODULE=2 -O1 -o validate.wasm
Old Exported Wasm imports
(module (type (;0;) (func (param i32 i32 i32))) (type (;1;) (func (param i32) (result i32))) (type (;2;) (func (param i32))) (type (;3;) (func (param i32 i32) (result i32))) (type (;4;) (func (param i32 i32 i32 i32) (result i32))) (type (;5;) (func (param i32 i32 i32) (result i32))) (type (;6;) (func)) (import "env" "__memory_base" (global (;0;) i32)) (import "env" "memory" (memory (;0;) 256))
New Exported WASM imports
(module (type (;0;) (func)) (type (;1;) (func (param i32 i32 i32))) (type (;2;) (func (param i32) (result i32))) (type (;3;) (func (param i32))) (type (;4;) (func (param i32 i32) (result i32))) (type (;5;) (func (param i32 i32 i32) (result i32))) (type (;6;) (func (param i32 i32 i32 i32) (result i32))) (type (;7;) (func (result i32))) (import "env" "g$AllocatedMemoryChunks" (func $g$AllocatedMemoryChunks (type 7))) (import "env" "g$current_allocated_count" (func $g$current_allocated_count (type 7))) (import "env" "__memory_base" (global (;0;) i32)) (import "env" "memory" (memory (;0;) 0))
The CPP file with the problematic exports, side_module_system_functions.cpp
include
include
ifdef __cplusplus
extern "C" { // So that the C++ compiler does not rename our function names
endif
const int TOTAL_MEMORY = 65536; // We should always have at least 1 page of memory (1,024 bytes x 64 KiB). const int MAXIMUM_ALLOCATED_CHUNKS = 10; int current_allocated_count = 0;
struct MemoryAllocated { int offset; int length; };
struct MemoryAllocated AllocatedMemoryChunks[MAXIMUM_ALLOCATED_CHUNKS];// The array that will hold details about each memory allocation
// Helper when adding an item before existing items in the array void InsertIntoAllocatedArray(int new_item_index, int offset_start, int size_needed) { // Shift everything to the right by one if it is to the right of where the new item will be inserted for (int i = (MAXIMUM_ALLOCATED_CHUNKS - 1); i > new_item_index; i--) { AllocatedMemoryChunks[i] = AllocatedMemoryChunks[(i - 1)]; }
// Add the new item at the specified index AllocatedMemoryChunks[new_item_index].offset = offset_start; AllocatedMemoryChunks[new_item_index].length = size_needed; // Increment the count of blocks allocated current_allocated_count++;
}
// Our version of malloc EMSCRIPTEN_KEEPALIVE int create_buffer(int size_needed) { // If we are already at our limit of allocated memory blocks then exit now if (current_allocated_count == MAXIMUM_ALLOCATED_CHUNKS) { return 0; }
// Adjust the start position to give room for items that will be copied into memory when the module is instantiated int offset_start = 1024; int current_offset = 0; int found_room = 0; int memory_size = size_needed; while (memory_size % 8 != 0) { memory_size++; }// Increase the size so that the next offset will be a multiple of 8 // Loop through the currently allocated memory... for (int index = 0; index < current_allocated_count; index++) { // If there is space between the previous offset and the current one for the memory that is wanted then... current_offset = AllocatedMemoryChunks[index].offset; if ((current_offset - offset_start) >= memory_size) { // Add the current item to the current index of the array (bump the rest of the items to the right by one) InsertIntoAllocatedArray(index, offset_start, memory_size); found_room = 1; break; } // OffsetStart for the next loop will be the end of the current array item's memory block offset_start = (current_offset + AllocatedMemoryChunks[index].length); } // Room wasn't found in between the existing allocated memory blocks if (found_room == 0) { // If there is room between the end of the last memory block and the end of memory then... if (((TOTAL_MEMORY - 1) - offset_start) >= size_needed) { // Add the item to the array AllocatedMemoryChunks[current_allocated_count].offset = offset_start; AllocatedMemoryChunks[current_allocated_count].length = size_needed; current_allocated_count++; found_room = 1; } } // If there was room for the memory needed then return the offset if (found_room == 1) { return offset_start; } // Otherwise, tell the caller there was no room (NULL pointer) return 0;
}
JavaScript Import Promise
let moduleMemory = null; let moduleExports = null;
function initializePage() { document.getElementById("name").value = initialData.name;
const category = document.getElementById("category"); const count = category.length; for (let index = 0; index < count; index++) { if (category[index].value === initialData.categoryId) { category.selectedIndex = index; break; } }
moduleMemory = new WebAssembly.Memory({initial: 256});
const importObject = { env: { __memory_base: 0, memory: moduleMemory, } };
WebAssembly.instantiateStreaming(fetch("validate.wasm"), importObject).then(result => { moduleExports = result.instance.exports; }); }
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/emscripten-core/emscripten/issues/9893?email_source=notifications&email_token=AAD55ZJEO7ROMEKKUUV557DQVLKVFA5CNFSM4JRAUMZ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H3UYXDA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD55ZLW5JBHYUXB3FTKRMDQVLKVFANCNFSM4JRAUMZQ .
Thanks for the information @sbc100 , I will try and use the STANDALONE_WASM and see what happens. But with the new changes, should it be exporting the Fields as functions? One of them is just an int but it thinks it is a function for some reason.
No, the behaviour you are seeing is the dynamic linking ABI and it should not appear with STANDALONE_WASM.
In the example you sent the global data symbols is actually being
imported as g$current_allocated_count
, not exported. The fp$
and g$
prefixes are part of the dynamic linking ABI in emscripten.
They are used by the module to find out the address or table location
of a symbol at runtime. This is because when you compile a position
independent module (i.e. fPIC or SIDE_MODULE) the location of a given
global variable in memory is not known until runtime so the module has
to call out the embedded / dynamic loader to get the address of its
own global variable (this is only true for global variable with
external linkage).
On Sun, Nov 24, 2019 at 5:00 PM danec020 notifications@github.com wrote:
Thanks for the information @sbc100 , I will try and use the STANDALONE_WASM and see what happens. But with the new changes, should it be exporting the Fields as functions? One of them is just an int but it thinks it is a function for some reason.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
@sbc100 Thanks so much for all the help! I really love what you guys are doing and I hope one day I can understand it as well as you. I feel like WASM is going to be a huge part of the future of Web Development so I am really trying my best to understand the inner workings and how things work so I can get in early on the technology. Your last answer was very insightful and I really appreciate that you took the time to explain to a new comer like myself. Unfortunately if I remove SIDE_MODULE and replace it with STANDALONE_WASM I lose my function name that need to be included. It ignores my EMSCRIPTEN_KEEPALIVE attributes and the names are lost.
So if I was doing dynamic linking how does one set the promise up for this type of import? I tried looking at emscripten's js file that it exports but it looks like it is doing it's promise differently than I was doing? Since the JS side of things doesn't need to interact with current_allocated_count, the "function" would not need to do anything.
Sorry if it is a basic question, you don't need to answer it if it is a pain. I am just curious as how I could have made it work for learning purposes. :) Anyhow, would you like me to keep this open or close this issue?
I would not try to do dynamic linking without emcripten's JS code.
Lets try to fix your use case correctly and instead get STANDALONE_WASM
working for you. You say that EMSCRIPTEN_KEEPALIVE
is not working for you? How about -s EXPORTED_FUNCTIONS
? Which functions are exported from the resulting module?
I confirmed that EMSCRIPTEN_KEEPALIVE
does work correctly with -s STANDALONE_WASM
.
@sbc100 You are correct, the functions are exported still but they are no longer prefixed with an _ it seems. So I got that working but it appears the memory imports are now gone so when I try to pass in memory from the JS side to the C++ side, nothing gets copied over. I looked in the wat version of the new output and it appears to be missing this section.
(import "env" "__memory_base" (global (;0;) i32))
(import "env" "memory" (memory (;0;) 256))
The instructions I have is unclear how the export options knew that I would be importing memory. I think it mentioned something about when using string literals emscripten knew to import memory, but I could be wrong.
With STANDALONE_WASM the memory is always exported from the module, so you don't need to create it yourself.
Before I was creating a JS array and setting an import object where it would assign the "memory" import to the JS array. Like this:
moduleMemory = new WebAssembly.Memory({initial: 256});
const importObject = {
env: {
__memory_base: 0,
memory: moduleMemory,
}
};
WebAssembly.instantiateStreaming(fetch("validate.wasm"), importObject).then(result => {
moduleExports = result.instance.exports;
});
I believe you are saying this is no longer necessary? So how can I access the memory from the JS side with this new export option? Also is there somewhere that documents these details? I hate bothering you with these ,probably obvious, questions.
The "memory" will be available to you as moduleExports['memory'] after instantiation.
There is only a little documentation available on these as until recently the wasm<->JS interface has really been an implementation detail. What documentation we have have is at https://github.com/emscripten-core/emscripten/wiki/WebAssembly-Standalone. You can also try look at WASI for documentation since for the low level interfaces we are trying to match what is happening in WASI.
Still what are you trying to do is fairly advanced and we always recommend you use the emscripten-generated JS to load the wasm file.. which avoids all these details.
@ sbc100 I understand this might not be the easy way to do things, I am trying to get a better understanding how things work before I just start using the emscripten generated js. I am able to manipulate the memory now and get some of my functionality working but it isn't ideal. Before I was creating a new memory module with a size of 256. I then assigned that to the module to use as its chunk of memory. I was leaving the first 1024 bits alone in case the module had to write something to the memory. With the new export options I can get the same results if I make a function that takes in a memory module and stores it as a reference to use internally but it is not as elegant as before.
Anyhow, I really appreciate the help as I have stated many times :D. I am going to try and dig through the github and try to understand what is going on. Also it seems like debuging C++ code is quite hard at the moment. The only thing I can do is send stuff to the console it seems. I am not a web developer so maybe there are some better ways.
You can control the size of the memory the wasm module uses by settings -s TOTAL_MEMORY
. I don't understand what you man by memory of size 256. Wasm memory is measured in pages of 64k each. Do you mean 256 pages of memory? That would be 16Mb?
Rather than hardcoding 1024 as the offset to use for you data you should really either statically allocate your memory (as a C global) or dynamically allocate it with malloc. Otherwise there is no way to safely use any of the C library. If you don't want to use malloc or the C library then you can put your memory wherever you want after __memory_base
. The linker provides the __memory_base
global for this purpose. It tells you where the module's static data ends. Normally malloc/sbk are in charge of allocating memory after that point.
If you are really not using libc or malloc or sbrk then I'm not sure what benefit you are getting from using emscripten and you might want to look into just using clang on its own. Or maybe I'm missing something?
@sbc100 Well the reason I am doing all this is for learning purposes. I am like learning how things are done within an API so I have a better understanding what happens when I do something in that API. I am also probably not experienced enough to explain what I "think" is happening, thus the confusion.
Anyhow, thanks again! You've gotten me far enough and I appreciate that. In the future I will probably just use emscriptens output.
Btw, here's a nice example of hand-written JS code for standalone wasm output: https://github.com/zeux/meshoptimizer/blob/bdc3006532dd29b03d83dc819e5fa7683815b88e/js/meshopt_decoder.js
This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.
I am following a book and trying to learn WebAssembly but things have obviously changed since the book was written. The old WASM file works fine but when exporting with the latest version it does not. The new version wants to import two field as functions for some reason.
I am trying to create an exported C++ project that doesn't use the standard library and instead uses my own functions. That way we can strip the unneeded includes out. I am running into a problem where AllocatedMemoryChunks field is being exported as a function and the browser console is saying in the promise I am missing a callable. I am creating the JS code myself as part of the learning process. I am creating an import object with my memory import information but I am not sure how to set it up to work with the AllocatedMemoryChunks.
Here is the old example's generated WAT file compared to the newly generated WAT file. You can see that the global fields of an int and a struct are exported as functions for some reason. At the end is the cpp file that has these exports. Any help on how I can prevent this if it is not supposed to happen or how should I correctly import them in the Promise? At the very bottom is my import code in JS.
Compile script
emcc side_module_system_functions.cpp validate.cpp -s SIDE_MODULE=2 -O1 -o validate.wasm
Old Exported Wasm imports
New Exported WASM imports
The CPP file with the problematic exports, side_module_system_functions.cpp
JavaScript Import Promise