search

openframeworks / openFrameworks

openFrameworks is a community-developed cross platform toolkit for creative coding in C++.
http://openframeworks.cc
Other
9.94k stars 2.55k forks source link

Latest Emscripten 2.0.33 works well, except file loading. [solved] #6781

Closed Jonathhhan closed 1 year ago

Jonathhhan commented 2 years ago

It was possible to compile with the latest Emscripten, for that I had to recompile the OF libs (FreeImage or freetype for example) with the current Emscripten and made some other small changes like using c++17 and std filesystem instead of boost (boost had an undefined symbol). I guess the issue now has to do with the glm qualifier glm::qualifier)0>&, this is the Java Script error message for the imageSubsectionExample (everything compiles fine): imageSubsectionExample.html:1 emscripten_set_main_loop_timing: Cannot set timing mode for main loop since a main loop does not exist! Call emscripten_set_main_loop first to set one up. printErr @ imageSubsectionExample.html:1 imageSubsectionExample.html:1 Aborted(native code called abort()) printErr @ imageSubsectionExample.html:1 imageSubsectionExample.js:1 Uncaught RuntimeError: Aborted(native code called abort()) at abort (imageSubsectionExample.js:1:34496) at _abort (imageSubsectionExample.js:1:119651) at std::__2::__throw_failure(char const*) (imageSubsectionExample.wasm:0x362df2) at std::__2::__fs::filesystem::__canonical(std::__2::__fs::filesystem::path const&, std::__2::error_code*) (imageSubsectionExample.wasm:0x380f6f) at ofToDataPath(std::__2::__fs::filesystem::path const&, bool) (imageSubsectionExample.wasm:0x55a2d) at ofApp::setup() (imageSubsectionExample.wasm:0x22653) at ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) (imageSubsectionExample.wasm:0x4638a) at std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) (imageSubsectionExample.wasm:0x4c0d6) at ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) (imageSubsectionExample.wasm:0x2a046) at ofCoreEvents::notifySetup() (imageSubsectionExample.wasm:0x4cb73) Is there a way to solve whether the boost, the glm or the std filesystem issue, so that is possible to load data with OF and the latest Emscripten (everything else seems to work fine)?

Jonathhhan commented 2 years ago

This is the Java Script error message if I compile with boost (even if there is no file to load):

advanced3dExample.html:1 missing function: 
_ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_
printErr @ advanced3dExample.html:1
advanced3dExample.html:1 Aborted(-1)
printErr @ advanced3dExample.html:1
advanced3dExample.js:1 Uncaught RuntimeError: Aborted(-1)
    at abort (advanced3dExample.js:1:34320)
    at __ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_ (advanced3dExample.js:1:47694)
    at boost::filesystem::absolute(boost::filesystem::path const&, boost::filesystem::path const&) (advanced3dExample.wasm:0x9716b)
    at main (advanced3dExample.wasm:0x1c5b5)
    at advanced3dExample.js:1:35039
    at callMain (advanced3dExample.js:1:277462)
    at doRun (advanced3dExample.js:1:278046)
    at advanced3dExample.js:1:278201

And interestingly, if I compile with std filesystem and -O1 instead of -O3 the glm error is gone (which enables for example the polygonExample to work):

imageSubsectionExample.html:1 emscripten_set_main_loop_timing: Cannot set timing mode for main loop since a main loop does not exist! Call emscripten_set_main_loop first to set one up. printErr @ imageSubsectionExample.html:1 imageSubsectionExample.html:1 Aborted(native code called abort()) printErr @ imageSubsectionExample.html:1 imageSubsectionExample.js:201 Uncaught RuntimeError: Aborted(native code called abort()) at abort (imageSubsectionExample.js:1596:10) at _abort (imageSubsectionExample.js:5377:2) at void std::__2::__fs::filesystem::__throw_filesystem_error<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >&, std::__2::__fs::filesystem::path const&, std::__2::__fs::filesystem::path const&, std::__2::error_code const&>(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> >&, std::__2::__fs::filesystem::path const&, std::__2::__fs::filesystem::path const&, std::__2::error_code const&) (imageSubsectionExample.wasm:0x3d5671) at std::__2::__fs::filesystem::detail::(anonymous namespace)::ErrorHandler<std::__2::__fs::filesystem::path>::report(std::__2::error_code const&) const (imageSubsectionExample.wasm:0x3d561a) at std::__2::__fs::filesystem::__canonical(std::__2::__fs::filesystem::path const&, std::__2::error_code*) (imageSubsectionExample.wasm:0x3d54df) at ofToDataPath(std::__2::__fs::filesystem::path const&, bool) (imageSubsectionExample.wasm:0x4d9d7) at ofLoadImage(ofPixels_<unsigned char>&, std::__2::__fs::filesystem::path const&, ofImageLoadSettings const&) (imageSubsectionExample.wasm:0x10179) at ofImage_<unsigned char>::load(std::__2::__fs::filesystem::path const&, ofImageLoadSettings const&) (imageSubsectionExample.wasm:0x10a8c) at ofApp::setup() (imageSubsectionExample.wasm:0x9248) at ofBaseApp::setup(ofEventArgs&) (imageSubsectionExample.wasm:0x330e2) So my question is: Is there is a way to make any of the filesystems work with this Emscripten version?

Jonathhhan commented 2 years ago

I comiled boost again, and now OF works with boost and Emscripten 2.0.33... Can add what I changed exactly for that, but basically I recompiled the necessary OF libs with Apothecary Emscripten 2.0.33. Edit: Also had to change some other stuff, but now it seems to work like the last versions...

Jonathhhan commented 2 years ago

Those are the additional changes (these changes need to be added, too: https://github.com/openframeworks/openFrameworks/issues/6758):

library_html5video.js (remove type)

html5video_player_pixel_format: function(id){
    return allocate(intArrayFromString(VIDEO.players[id].pixelFormat), ALLOC_STACK);
},

html5video_grabber_pixel_format: function(id){
    return allocate(intArrayFromString(VIDEO.grabbers[id].pixelFormat), ALLOC_STACK);
},

index.html (keyevents)

<canvas class=emscripten id=canvas oncontextmenu=event.preventDefault() tabindex=-1></canvas>

library_html5.js(emsdk remove __ from getBoundingClientRect)

    #if !DISABLE_DEPRECATED_FIND_EVENT_TARGET_BEHAVIOR
    if (Module['canvas']) {
      var rect = getBoundingClientRect(Module['canvas']);
      HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.canvasX / 4 }}}] = (e.clientX - rect.left) * (Module['canvas'].width / rect.width);
      HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.canvasY / 4 }}}] = (e.clientY - rect.top) * (Module['canvas'].height / rect.height);
    } else { // Canvas is not initialized, return 0.
      HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.canvasX / 4 }}}] = 0;
      HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.canvasY / 4 }}}] = 0;
    }
#endif
    var rect = getBoundingClientRect(target);
    HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.targetX / 4 }}}] = (e.clientX - rect.left) * (target.width / rect.width);
    HEAP32[idx + {{{ C_STRUCTS.EmscriptenMouseEvent.targetY / 4 }}}] = (e.clientY - rect.top) * (target.height / rect.height);

config.emscripten.default.make (add DYNCALLS=1 and BINARYEN_EXTRA_PASSES=--one-caller-inline-max-function-size=1)

PLATFORM_LDFLAGS = -Wl,--gc-sections --preload-file bin/data@data --emrun -s DYNCALLS=1 -s MAX_WEBGL_VERSION=2 -s WEBGL2_BACKWARDS_COMPATIBILITY_EMULATION=1 -s BINARYEN_EXTRA_PASSES=--one-caller-inline-max-function-size=1

roymacdonald commented 2 years ago

hey @Jonathhhan Thanks for all this. Can you push to your OF fork all this changes so it is easier to check it and not having to go through all this issue and manually changing stuff.?

Jonathhhan commented 2 years ago

@roymacdonald yes, of course. I will do that the next days.

roymacdonald commented 2 years ago

after installing emscripten 2.0.33 I cant compile as it complains the following:

error: undefined symbol: _ZN24ofxEmscriptenSoundStreamC1Ev (referenced by top-level compiled C/C++ code)
error: undefined symbol: tessDeleteTess (referenced by top-level compiled C/C++ code)
error: undefined symbol: tessNewTess (referenced by top-level compiled C/C++ code)

@Jonathhhan it is not seeing ofxEmscriptenSoundStream. did you add it somewhere? the other errors are for tess2. thanks

roymacdonald commented 2 years ago

I was able to get it to compile. I mostly needed to recompile the core lib dependencies.

Jonathhhan commented 2 years ago

@roymacdonald thats nice. It would be great if you can help me to get the ofxPd examples to run with Emscripten (for using it with AudioWorklets). ofxPd works for me as part of ofxOfelia with Emscripten and standalone, but somehow not with Emscripten without ofxOfelia...

roymacdonald commented 2 years ago

Sure, no problem. Let me get it to run the window properly. Did you manage to get GLES 2 shaders to work? I have not. I am modifying ofxAppEmscriptenWindow to make it work properly based on some examples that emscripten has. once I get that to work it my guess is that it shouldnt be much trouble to get audio to work as I know very well how audio works within OF.

Jonathhhan commented 2 years ago

Yes, GLES2 is working now (my example https://gameoflife3d.handmadeproductions.de/ actually uses shaders for the grid). I think its part of my changes. https://github.com/openframeworks/openFrameworks/issues/6758

changing this:

EGLint contextAttribs[] = { EGL_CONTEXT_MAJOR_VERSION, 3, EGL_NONE, EGL_NONE }; std::vector attribList = { EGL_RED_SIZE, EGL_DONT_CARE, EGL_GREEN_SIZE, EGL_DONT_CARE, EGL_BLUE_SIZE, EGL_DONT_CARE, EGL_ALPHA_SIZE, EGL_DONT_CARE, EGL_DEPTH_SIZE, EGL_DONT_CARE, EGL_STENCIL_SIZE, EGL_DONT_CARE, EGL_SAMPLE_BUFFERS, EGL_DONT_CARE, EGL_RENDERABLE_TYPE, EGL_OPENGL_ES3_BIT_KHR, EGL_NONE };

in https://github.com/Jonathhhan/openFrameworks/blob/master/addons/ofxEmscripten/src/ofxAppEmscriptenWindow.cpp and adding the flag -s USE_WEBGL2=1 to config.emscripten.default.mk enables webgl2 without editing any Emscripten file (in addition I use -s WEBGL2_BACKWARDS_COMPATIBILITY_EMULATION=1 for backwards compatibility with webgl1 https://emscripten.org/docs/optimizing/Optimizing-WebGL.html).

Jonathhhan commented 2 years ago

For AudioWorklets and SharedArraybuffer all used libs (for most things boost is enough) and also OF need to be compiled with -s USE_PTHREAD=1 or multi thread support. The pthread compiled libs work also if pthread isnt used.

roymacdonald commented 2 years ago

@Jonathhhan I tried out your changes and cloned your OF repo, but it was not working. Maybe I missed something. Although I got it to work by changing a lot of other things and making it rely more on the emscripten functions. Maybe I messed up something when trying to get it to compile. I will go back and start over now that I have the necesary libs compiled and working. I will let you know how it goes. c cheers

Jonathhhan commented 2 years ago

@roymacdonald sorry, I dont have all the changes in my repo yet. I will tell you, if so. This is the mentioned and used Emscripten AudioWorklet branch: https://github.com/tklajnscek/emscripten/tree/worklet_support.

Jonathhhan commented 2 years ago

@roymacdonald I made an OF AudioWorklet branch (I hope, its fine like that) that includes all updates for Emscripten 2.0.34 (and some other fixes) and for the Emscripten AudioWorklet branch: https://github.com/Jonathhhan/openFrameworks/tree/AudioWorklet Only change that needs to be done in Emscripten is replacing file-packager.py with this file: https://github.com/Jonathhhan/emscripten/blob/master/tools/file_packager.py (basically I just added 

if (typeof window === "object")

at line 291)

roymacdonald commented 2 years ago

@Jonathhhan awesome! I'll check it and get back with comments (and see how to make all the other audio stuff to work)

Jonathhhan commented 2 years ago

This is the error message with the pdExample from ofxPd:

pdExample.html:1 emscripten_set_main_loop_timing: Cannot set timing mode for main loop since a main loop does not exist! Call emscripten_set_main_loop first to set one up.
printErr @ pdExample.html:1
err @ pdExample.js:1
_emscripten_set_main_loop_timing @ pdExample.js:1
_eglSwapInterval @ pdExample.js:1
$ofxAppEmscriptenWindow::setVerticalSync(bool) @ pdExample.wasm:0x1ca18
$ofSetVerticalSync(bool) @ pdExample.wasm:0xfd379
$ofApp::setup() @ pdExample.wasm:0x13f35
$ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) @ pdExample.wasm:0xfa4ea
$std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) @ pdExample.wasm:0x101ac5
$ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) @ pdExample.wasm:0x1d2c6
$ofCoreEvents::notifySetup() @ pdExample.wasm:0x102dbd
$ofxAppEmscriptenWindow::loop() @ pdExample.wasm:0x1c299
$std::__2::__function::__func<void (*)(), std::__2::allocator<void (*)()>, void ()>::operator()() @ pdExample.wasm:0x101255
$ofMainLoop::loop() @ pdExample.wasm:0x100d22
$ofRunApp(ofBaseApp*) @ pdExample.wasm:0xfd010
$__original_main @ pdExample.wasm:0x13e0e
$main @ pdExample.wasm:0x13eba
(anonymous) @ pdExample.js:1
callMain @ pdExample.js:1
doRun @ pdExample.js:1
(anonymous) @ pdExample.js:1
setTimeout (async)
run @ pdExample.js:1
runCaller @ pdExample.js:1
removeRunDependency @ pdExample.js:1
receiveInstance @ pdExample.js:1
receiveInstantiationResult @ pdExample.js:1
Promise.then (async)
(anonymous) @ pdExample.js:1
Promise.then (async)
instantiateAsync @ pdExample.js:1
createWasm @ pdExample.js:1
(anonymous) @ pdExample.js:1
pdExample.html:1 /
pdExample.html:1 Buffer size: 512
pdExample.html:1 
pdExample.html:1 BEGIN Patch Test
pdExample.js:1 Uncaught RuntimeError: null function or function signature mismatch
    at pd_typedmess (pdExample.wasm:0x58c64)
    at binbuf_eval (pdExample.wasm:0x30f3b)
    at canvas_objtext (pdExample.wasm:0xaea1c)
    at canvas_obj (pdExample.wasm:0xae8d1)
    at pd_typedmess (pdExample.wasm:0x58720)
    at binbuf_eval (pdExample.wasm:0x30f3b)
    at binbuf_evalfile (pdExample.wasm:0x341a3)
    at glob_evalfile (pdExample.wasm:0x2e003)
    at libpd_openfile (pdExample.wasm:0x280eb)
    at ofxPd::openPatch(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) (pdExample.wasm:0x206b3)
$pd_typedmess @ pdExample.wasm:0x58c64
$binbuf_eval @ pdExample.wasm:0x30f3b
$canvas_objtext @ pdExample.wasm:0xaea1c
$canvas_obj @ pdExample.wasm:0xae8d1
$pd_typedmess @ pdExample.wasm:0x58720
$binbuf_eval @ pdExample.wasm:0x30f3b
$binbuf_evalfile @ pdExample.wasm:0x341a3
$glob_evalfile @ pdExample.wasm:0x2e003
$libpd_openfile @ pdExample.wasm:0x280eb
$ofxPd::openPatch(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) @ pdExample.wasm:0x206b3
$ofApp::setup() @ pdExample.wasm:0x14430
$ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) @ pdExample.wasm:0xfa4ea
$std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) @ pdExample.wasm:0x101ac5
$ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) @ pdExample.wasm:0x1d2c6
$ofCoreEvents::notifySetup() @ pdExample.wasm:0x102dbd
$ofxAppEmscriptenWindow::loop() @ pdExample.wasm:0x1c299
$std::__2::__function::__func<void (*)(), std::__2::allocator<void (*)()>, void ()>::operator()() @ pdExample.wasm:0x101255
$ofMainLoop::loop() @ pdExample.wasm:0x100d22
$ofRunApp(ofBaseApp*) @ pdExample.wasm:0xfd010
$__original_main @ pdExample.wasm:0x13e0e
$main @ pdExample.wasm:0x13eba
(anonymous) @ pdExample.js:1
callMain @ pdExample.js:1
doRun @ pdExample.js:1
(anonymous) @ pdExample.js:1
setTimeout (async)
run @ pdExample.js:1
runCaller @ pdExample.js:1
removeRunDependency @ pdExample.js:1
receiveInstance @ pdExample.js:1
receiveInstantiationResult @ pdExample.js:1
Promise.then (async)
(anonymous) @ pdExample.js:1
Promise.then (async)
instantiateAsync @ pdExample.js:1
createWasm @ pdExample.js:1
(anonymous) @ pdExample.js:1
pdExample.js:1 Uncaught TypeError: Cannot read properties of undefined (reading 'slice')
    at pdExample.js:1:5077
(anonymous) @ pdExample.js:1
roymacdonald commented 2 years ago

I got that error before and fixed it. And I ended up doing some massive changes to ofxAppEmscriptenWindow. I think that now it looks better. :) https://github.com/roymacdonald/openFrameworks/blob/emscriptenUpdate/addons/ofxEmscripten/src/ofxAppEmscriptenWindow.cpp https://github.com/roymacdonald/openFrameworks/blob/emscriptenUpdate/addons/ofxEmscripten/src/ofxAppEmscriptenWindow.h

Jonathhhan commented 2 years ago

@roymacdonald with your changes I get kind of a similar error as in the pdExample (but only one of them):

EmscriptenExample.js:1 Uncaught TypeError: Cannot read properties of undefined (reading 'slice')
    at EmscriptenExample.js:1:5966
(anonymous) @ EmscriptenExample.js:1

while undefined is Module["preRun"]:

var necessaryPreJSTasks = Module["preRun"].slice();
if (typeof window === "object") {
    Module["arguments"] = window.location.search.substr(1).trim().split("&");
    if (!Module["arguments"][0]) {
        Module["arguments"] = []
    }
}
if (!Module["preRun"])
    throw "Module.preRun should exist because file support used it; did a pre-js delete it?";
necessaryPreJSTasks.forEach(function(task) {
    if (Module["preRun"].indexOf(task) < 0)
        throw "All preRun tasks that exist before user pre-js code should remain after; did you replace Module or modify Module.preRun?"
});
var moduleOverrides = {};
var key;
for (key in Module) {
    if (Module.hasOwnProperty(key)) {
        moduleOverrides[key] = Module[key]
    }
}
var arguments_ = [];
var thisProgram = "./this.program";
var quit_ = function(status, toThrow) {
    throw toThrow
};

The patch loads fine, but without the audioWorklet... But maybe it is possible to use -s PROXY_TO_PTHREAD=1 -s OFFSCREENCANVAS_SUPPORT=1 -s OFFSCREEN_FRAMEBUFFER=1 with your changes... :)

roymacdonald commented 2 years ago

I am not even able to run the pd example as it fails compiling it, although the OF audio examples work fine. There are some issues with the ofxPd addon. Are you using a branch other than ofxPd's master?

roymacdonald commented 2 years ago

btw, I am using macos. i understand you use linux, right?

Jonathhhan commented 2 years ago

yes, ubuntu 20.04. I forgot to mention, I think I use the ofxPd lib that is included in ofxOfelia and I set the inputs to 0... with a little bit of work it could be possible to use offscreen rendering with your changes (already have an image, but I cant resize it - and it conflicts with the audioworklet ;) )...

Jonathhhan commented 2 years ago

It works with -s PROXY_TO_PTHREAD=1 -s OFFSCREENCANVAS_SUPPORT=1 -s OFFSCREEN_FRAMEBUFFER=1, sadly not yet together with audioworklet yet, but it seems to be very efficient. But I have to set the canvas size in the html file, if I try to set it in java script I get this message:

Uncaught DOMException: Failed to set the 'width' property on 'HTMLCanvasElement': Cannot resize canvas after call to transferControlToOffscreen().
    at Object.updateCanvasDimensions (http://localhost:6931/EmscriptenExample.js:1:218824)
    at Object.setCanvasSize (http://localhost:6931/EmscriptenExample.js:1:217256)
    at _emscripten_set_canvas_size (http://localhost:6931/EmscriptenExample.js:1:223031)
    at _emscripten_receive_on_main_thread_js (http://localhost:6931/EmscriptenExample.js:1:220745)
    at _do_call (http://localhost:6931/EmscriptenExample.wasm:wasm-function[15107]:0x8a3324)
    at emscripten_current_thread_process_queued_calls (http://localhost:6931/EmscriptenExample.wasm:wasm-function[15105]:0x8a2b6b)
    at emscripten_main_thread_process_queued_calls (http://localhost:6931/EmscriptenExample.wasm:wasm-function[15117]:0x8a36c6)
    at http://localhost:6931/EmscriptenExample.js:1:29890
    at Worker.worker.onmessage (http://localhost:6931/EmscriptenExample.js:1:39637)

If I dont remove the audioworklet part with offscreen canvas I get this message (maybe because it is not possible to create a worker from a worker...):

EmscriptenExample.worker.js:1 worker.js onmessage() captured an uncaught exception: TypeError: Cannot read properties of undefined (reading 'audioWorklet')
self.onmessage @ EmscriptenExample.worker.js:1
EmscriptenExample.worker.js:1 TypeError: Cannot read properties of undefined (reading 'audioWorklet')
    at Object.initAudioWorkletPThread (EmscriptenExample.js:1484:27)
    at _html5audio_stream_create (EmscriptenExample.js:9687:13)
    at ofxEmscriptenSoundStream::setup(ofSoundStreamSettings const&) (EmscriptenExample.wasm:0x21b75)
    at ofxOfelia::init(int, int, int, int, bool, bool, int, int, bool, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) (EmscriptenExample.wasm:0x28ae6)
    at ofApp::setup() (EmscriptenExample.wasm:0x1d57a)
    at ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) (EmscriptenExample.wasm:0x523af3)
    at std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofEventArgs, std::__2::recursive_mutex> > ofEvent<ofEventArgs, std::__2::recursive_mutex>::make_function<ofEasyCam>(ofEasyCam*, void (ofEasyCam::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofEventArgs, std::__2::recursive_mutex> > ofEvent<ofEventArgs, std::__2::recursive_mutex>::make_function<ofEasyCam>(ofEasyCam*, void (ofEasyCam::*)(ofEventArgs&), int)::'lambda'(void const*, ofEventArgs&)>, bool (void const*, ofEventArgs&)>::operator()(void const*&&, ofEventArgs&) (EmscriptenExample.wasm:0x51c5d6)
    at ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) (EmscriptenExample.wasm:0x2076c)
    at ofCoreEvents::notifySetup() (EmscriptenExample.wasm:0x533088)
    at ofxAppEmscriptenWindow::loop() (EmscriptenExample.wasm:0x1f6ab)
Jonathhhan commented 2 years ago

And without -s PROXY_TO_PTHREAD=1 -s OFFSCREENCANVAS_SUPPORT=1 -s OFFSCREEN_FRAMEBUFFER=1 your updated files work with audioworklet if I remove:

var necessaryPreJSTasks = Module["preRun"].slice();
if (typeof window === "object") {
    Module["arguments"] = window.location.search.substr(1).trim().split("&");
    if (!Module["arguments"][0]) {
        Module["arguments"] = []
    }
}
if (!Module["preRun"])
    throw "Module.preRun should exist because file support used it; did a pre-js delete it?";
necessaryPreJSTasks.forEach(function(task) {
    if (Module["preRun"].indexOf(task) < 0)
        throw "All preRun tasks that exist before user pre-js code should remain after; did you replace Module or modify Module.preRun?"
});

from the generated java script file (at line 239).

roymacdonald commented 2 years ago

Great! it sounds that Module["preRun"] got deleted and thats is why it complains. I have not gotten ofxPd to run yet (haven't done much though).

Jonathhhan commented 2 years ago

Here is an example with -s PROXY_TO_PTHREAD=1 -s OFFSCREENCANVAS_SUPPORT=1 -s OFFSCREEN_FRAMEBUFFER=1: https://test2.handmadeproductions.de/ I would say it runs better than without (only the sound and resizing issues then). Here for comparison without offscreen rendering: https://arturocastro.net/files/of-emscripten/assimpExample/ I also put alpha off, because it results in a white background with offscreen canvas rendering (or whatever color the background has). Offscreen canvas does not seem to work with firefox, but chrome and edge (thats what I tested): pthread_create: failed to transfer control of canvas "canvas" to OffscreenCanvas! Error: [Exception... "Method not implemented" nsresult: "0x80004001 (NS_ERROR_NOT_IMPLEMENTED)" location: "JS frame :: https://test2.handmadeproductions.de/assimpExample.js :: ___pthread_create_js :: line 2209" data: no]

roymacdonald commented 2 years ago

So I am able to run the sound examples for input and output audio with graphics and all. One thing I noticed was that the files were not being able to load because of ofToDataPath was not giving the correct path. in ofUtils.cpp i modified the defaultDataPath function to look like this 


    string defaultDataPath(){
    #if defined TARGET_OSX
        try{
            return std::filesystem::canonical(ofFilePath::join(ofFilePath::getCurrentExeDir(),  "../../../data/")).string();
        }catch(...){
            return ofFilePath::join(ofFilePath::getCurrentExeDir(),  "../../../data/");
        }
    #elif defined TARGET_ANDROID
        return string("sdcard/");
    #elif defined(TARGET_EMSCRIPTEN) /// added this
        return string("data/"); // and this to make it work with emscripten
    #else
        try{
            return std::filesystem::canonical(ofFilePath::join(ofFilePath::getCurrentExeDir(),  "data/")).make_preferred().string();
        }catch(...){
            return ofFilePath::join(ofFilePath::getCurrentExeDir(),  "data/");
        }
    #endif
    }

Now I can load files yet still ofxPd gives me odd errors which dont have much info in order to debug. So, out of this I would be inclined to say that the problem lies in either ofxPd or libpd, rather than in OF as audio works well.

Jonathhhan commented 2 years ago

@roymacdonald thats great. thank you very much for your help. I discovered that my filesystem issue was caused by -s ASSERTIONS=2.

roymacdonald commented 2 years ago

No problem. It is you who I must thank. So you were having filesystem issues with the Debug version only (which has ASERTIONS=2)? What would be really useful now is to get ofThread working with Emscripten. I will keep you updated. cheers

Jonathhhan commented 2 years ago

Yes, only with ASERTIONS=2. ofThread sounds great, but not sure how to use them. If audioWorklet and offscreen rendering would work at the same time, both (audio and graphics) would have their own thread. Meanwhile ofxPd (the ofxOfelia lib) is initing, but not opening the patch:

[verbose] Pd: inited
pdExample.html:1 [verbose] Pd:  samplerate: 44100
pdExample.html:1 [verbose] Pd:  channels in: 0
pdExample.html:1 [verbose] Pd:  channels out: 2
pdExample.html:1 [verbose] Pd:  ticks: 8
pdExample.html:1 [verbose] Pd:  block size: 64
pdExample.html:1 [verbose] Pd:  calc buffer size: 512
pdExample.html:1 [verbose] Pd:  queued: yes
pdExample.html:1 [verbose] Pd: audio processing on
pdExample.html:1 
pdExample.html:1 BEGIN Patch Test
pdExample.html:1 [verbose] Pd: opening patch: test.pd path: /data/pd
pdExample.js:1 Uncaught RuntimeError: null function or function signature mismatch
    at pd_typedmess (pdExample.wasm:0x1a643)
    at binbuf_eval (pdExample.wasm:0x193b1)
    at binbuf_evalfile (pdExample.wasm:0x8b27f)
    at glob_evalfile (pdExample.wasm:0x42c92)
    at libpd_openfile (pdExample.wasm:0x7d5e3)
    at ofxPd::openPatch(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) (pdExample.wasm:0x72227)
    at ofApp::setup() (pdExample.wasm:0x105c21)
    at ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) (pdExample.wasm:0x77859)
    at std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) (pdExample.wasm:0x287a4)
    at ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) (pdExample.wasm:0x1c50b)
$pd_typedmess @ pdExample.wasm:0x1a643
$binbuf_eval @ pdExample.wasm:0x193b1
$binbuf_evalfile @ pdExample.wasm:0x8b27f
$glob_evalfile @ pdExample.wasm:0x42c92
$libpd_openfile @ pdExample.wasm:0x7d5e3
$ofxPd::openPatch(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) @ pdExample.wasm:0x72227
$ofApp::setup() @ pdExample.wasm:0x105c21
$ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) @ pdExample.wasm:0x77859
$std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) @ pdExample.wasm:0x287a4
$ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) @ pdExample.wasm:0x1c50b
$ofCoreEvents::notifySetup() @ pdExample.wasm:0x79d84
$ofxAppEmscriptenWindow::loop() @ pdExample.wasm:0xe229c
$std::__2::__function::__func<void (*)(), std::__2::allocator<void (*)()>, void ()>::operator()() @ pdExample.wasm:0xc2343
$ofMainLoop::loop() @ pdExample.wasm:0xface5
$ofRunApp(ofBaseApp*) @ pdExample.wasm:0xe948a
$__original_main @ pdExample.wasm:0x161e7d
$main @ pdExample.wasm:0x11340d
Module._main @ pdExample.js:1
callMain @ pdExample.js:1
doRun @ pdExample.js:1
(anonymous) @ pdExample.js:1
setTimeout (async)
run @ pdExample.js:1
runCaller @ pdExample.js:1
removeRunDependency @ pdExample.js:1
receiveInstance @ pdExample.js:1
receiveInstantiationResult @ pdExample.js:1
Promise.then (async)
(anonymous) @ pdExample.js:1
Promise.then (async)
instantiateAsync @ pdExample.js:1
createWasm @ pdExample.js:1
(anonymous) @ pdExample.js:1

Edit: And if I use a very simple patch instead of the included one, it even works (just a print message):

/
Buffer size: 512
[verbose] Pd: inited
[verbose] Pd:  samplerate: 44100
[verbose] Pd:  channels in: 0
[verbose] Pd:  channels out: 2
[verbose] Pd:  ticks: 8
[verbose] Pd:  block size: 64
[verbose] Pd:  calc buffer size: 512
[verbose] Pd:  queued: no
[verbose] Pd: audio processing on

BEGIN Patch Test
[verbose] Pd: opening patch: test.pd path: /data/pd
[verbose] Pd: print: print: This is a test!
print: This is a test!
Patch: "test.pd" $0: 1003 valid: 1
[verbose] Pd: closing patch: test.pd
Audio worklet PThread context initialized!
Audio worklet node created! Tap/click on the window if you don't hear audio!

Btw: Emscripten 3.0.1 was just released...

roymacdonald commented 2 years ago

Btw: Emscripten 3.0.1 was just released... lol. well. we will have to test it.

So the issues you have wiht ofxPd is because of its patch being to complex? is it even working when compiling without emscripten?

As of threads, yes the audio stuff runs on its own thread. I was reading emscriptens documentation and it seems to be quite straight forwards to implement ofThread for emscripten. (I use it a lot and one of the projects I wanted to port uses it and making it single threaded would make it super slow and unresponsive, besides having to get rid of the multithread model.

Jonathhhan commented 2 years ago

No, its just the included pdExample. Now it works, besides pd.closePatch. I had to delete it from the test and I had to delete the midi messages from the pd patch (which makes kind of sense, because there is no "normal" midi in the web). This is the pd.closePatch error message at the end of the test:

pdExample.js:1 Uncaught RuntimeError: null function or function signature mismatch
    at pd_free (pdExample.wasm:0x18c3e)
    at libpd_closefile (pdExample.wasm:0x42875)
    at ofxPd::closePatch(pd::Patch&) (pdExample.wasm:0x72698)
    at ofApp::setup() (pdExample.wasm:0x107551)
    at ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) (pdExample.wasm:0x77416)
    at std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) (pdExample.wasm:0x287a3)
    at ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) (pdExample.wasm:0x1c50a)
    at ofCoreEvents::notifySetup() (pdExample.wasm:0x79941)
    at ofxAppEmscriptenWindow::loop() (pdExample.wasm:0xe229b)
    at std::__2::__function::__func<void (*)(), std::__2::allocator<void (*)()>, void ()>::operator()() (pdExample.wasm:0xc20b2)
roymacdonald commented 2 years ago

I noticed that ofxPd extends PdBase and it occludes some of its virtual functions, such as closePatch. The compiler was warning about it. Try changing the name in ofxPd so this occlusion does not happen. It might be part of the problem.

Jonathhhan commented 2 years ago

Yes, it is: class ofxPd : public pd::PdBase, protected pd::PdReceiver, protected pd::PdMidiReceiver Should I try to change the name PdBase or pd to ofxPd? Thats about opening patches: 

    /// \section Opening Patches

        /// the pd::Patch class is a wrapper around a unique pointer to an open
        /// instance, giving you access to the $0 value for that instance
        ///
        /// if you are going to use multiple instances of a patch, you will need
        /// need to keep a Patch object in order to differentiate between them
        ///
        /// see src/pd/cpp/PdTypes.hpp for more info

        /// open a patch file, takes an absolute or relative path (in data folder)
        /// returns a Patch object
        pd::Patch openPatch(const std::string& patch);

        /// open a patch file using the filename and path of an existing patch
        ///
        /// // open a patch, don't need pd::Patch instance object
        /// pd.openPatch("apatch.pd", "/some/path");
        ///
        /// set the filename within the patch object or use a previously opened
        /// object and create a new instance
        ///
        /// // open an instance of "somefile.pd", save the instance in a pd::Patch
        /// Patch p2("somefile.pd", "/some/path");    // set file and path
        /// pd.openPatch(p2);
        ///
        /// // open a new instance of "somefile.pd"
        /// Patch p3 = pd.openPatch(p2);
        ///
        /// p2 and p3 now refer to 2 different instances of "somefile.pd" and 
        /// p2.dollarZero() & p3.dollarZero() should now return different ids
        ///
        pd::Patch openPatch(pd::Patch& patch);

        /// close a patch file, takes the patch's basename (filename without extension),
        /// use this function if you've only opened 1 instance of the given patch
        void closePatch(const std::string& patch);

        /// close a patch file using a Patch object, clears the given Patch object
        /// does not affect other open instances of the same patch
        void closePatch(pd::Patch& patch);
roymacdonald commented 2 years ago

I think it would be better to modify ofxPd. that way it will remain compatible with any update of libpd.

Jonathhhan commented 2 years ago

@roymacdonald that would definetely be better, but I suspect that I am not enough into it to do that. Its also what @danomatika says:

I prefer not to provide local hacks and would much rather accept changes made upstream, ie. libpd/pure-data, sonI suggest starting there and working back toward here. https://github.com/danomatika/ofxPd/issues/75

roymacdonald commented 2 years ago

@Jonathhhan Sure, although, does changing the function name fix that problem?

Jonathhhan commented 2 years ago

@roymacdonald not sure, if I understand it right (I guess not). But it didnt work the ways I tried (tried to rename closePatch to closePatch2 for example, or pd to ofxPd).

Jonathhhan commented 2 years ago

Btw, this is how I edited library_html5video.js for loading videos with drag and drop and a file browser (its quite messy), but the same way it should be possible to load audio or other stuff with Emscripten:

function _html5video_player_create() {
    var video = document.createElement("video");
    var player_id = VIDEO.getNewPlayerId();
    VIDEO.players[player_id] = video;
    var texId = GL.getNewId(GL.textures);
    var texture = GLctx.createTexture();
    texture.name = texId;
    GL.textures[texId] = texture;
    GLctx.bindTexture(GLctx.TEXTURE_2D, texture);
    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_MAG_FILTER, GLctx.LINEAR);
    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_MIN_FILTER, GLctx.LINEAR);
    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_WRAP_S, GLctx.CLAMP_TO_EDGE);
    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_WRAP_T, GLctx.CLAMP_TO_EDGE);
    VIDEO.players[player_id].textureId = texId;
    window.ondragover = function (e) {
        e.preventDefault()
    }
        ;
    window.ondrop = function (e) {
        e.preventDefault();
        console.log("Reading...");
        var length = e.dataTransfer.items.length;
        if (length > 1) {
            console.log("Please only drop 1 file.")
        } else {
            upload(e.dataTransfer.files[0])
        }
    }
        ;
    function upload(file) {
        if (file.type.match(/video\/*/)) {
            URL.revokeObjectURL(VIDEO.players[player_id].src);
            VIDEO.players[player_id].src = URL.createObjectURL(file);
            console.log("This file seems to be a video.")
        } else {
            console.log("This file does not seem to be a video.")
        }
    }
    video.onloadedmetadata = function (e) {
        console.log(this.videoWidth + "x" + this.videoHeight);
        VIDEO.players[player_id].returnWidth = this.videoWidth;
        VIDEO.players[player_id].returnHeight = this.videoHeight;
        VIDEO.players[player_id].width = 800;
        VIDEO.players[player_id].height = 600;
        var videoImage = document.createElement("canvas");
        videoImage.width = 800;
        videoImage.height = 600;
        var videoImageContext = videoImage.getContext("2d");
        videoImageContext.fillStyle = "#000000";
        videoImageContext.fillRect(0, 0, videoImage.width, videoImage.height);
        VIDEO.playersContexts[player_id] = videoImageContext;
        VIDEO.players[player_id].currentTime = 0
    };

    return player_id
}

https://github.com/Jonathhhan/Pure-Data-Ofelia-Emscripten/blob/main/videoPlayer/library_html5video.js It works like in this example: https://videoplayer.handmadeproductions.de/

Jonathhhan commented 2 years ago

Actually Emscripten 3.0.0 works with just recompiling the OF libs (with and without the Emscripten AudioWorklet branch).

Jonathhhan commented 2 years ago

And for the webMIDI implementation I used this file (some time ago, and it was only working with ofxOfelia): https://github.com/cwilso/midi-synth/blob/master/js/midi.js

Jonathhhan commented 2 years ago

Regarding the OF Libs and pthread compilation: I needed to compile boost, FreeImage and assimp with pthread support. For opencv I just needed to change a path in the shell file (also for single thread compilation). For boost I needed to keep the old files and just replace the new ones. FreeImage was the most complicated one, I had to edit the Makefile.gnu in the FreeImage build folder (which gets deleted during compilation, if the apothecary folder is not renamed). Thats the edited FreeImage Makefile.gnu:

# Linux makefile for FreeImage

# This file can be generated by ./gensrclist.sh
include Makefile.srcs

# General configuration variables:
DESTDIR ?= /
INCDIR ?= $(DESTDIR)/usr/include
INSTALLDIR ?= $(DESTDIR)/usr/lib

# Converts cr/lf to just lf
DOS2UNIX = dos2unix

LIBRARIES = -lstdc++

MODULES = $(SRCS:.c=.o)
MODULES := $(MODULES:.cpp=.o)
CFLAGS ?= -O3 -fPIC -fexceptions -fvisibility=hidden -s USE_PTHREADS=1
# OpenJPEG
CFLAGS += -DOPJ_STATIC
# LibRaw
CFLAGS += -DNO_LCMS
# LibJXR
CFLAGS += -DDISABLE_PERF_MEASUREMENT -D__ANSI__
CFLAGS += $(INCLUDE)
CXXFLAGS ?= -O3 -fPIC -fexceptions -fvisibility=hidden -Wno-ctor-dtor-privacy -s USE_PTHREADS=1
# LibJXR
CXXFLAGS += -D__ANSI__
CXXFLAGS += $(INCLUDE)

ifeq ($(shell sh -c 'uname -m 2>/dev/null || echo not'),x86_64)
    CFLAGS += -fPIC
    CXXFLAGS += -fPIC
endif

TARGET  = freeimage
STATICLIB = lib$(TARGET).a
SHAREDLIB = lib$(TARGET)-$(VER_MAJOR).$(VER_MINOR).so
LIBNAME = lib$(TARGET).so
VERLIBNAME = $(LIBNAME).$(VER_MAJOR)
HEADER = Source/FreeImage.h

default: all

all: dist

dist: FreeImage
    mkdir -p Dist
    cp *.a Dist/
    cp *.so Dist/
    cp Source/FreeImage.h Dist/

dos2unix:
    @$(DOS2UNIX) $(SRCS) $(INCLS)

FreeImage: $(STATICLIB) $(SHAREDLIB)

.c.o:
    $(CC) $(CFLAGS) -c $< -o $@

.cpp.o:
    $(CXX) $(CXXFLAGS) -c $< -o $@

$(STATICLIB): $(MODULES)
    $(AR) r $@ $(MODULES)

$(SHAREDLIB): $(MODULES)
    $(CC) -s -shared -Wl,-soname,$(VERLIBNAME) $(LDFLAGS) -o $@ $(MODULES) $(LIBRARIES)

install:
    install -d $(INCDIR) $(INSTALLDIR)
    install -m 644 -o root -g root $(HEADER) $(INCDIR)
    install -m 644 -o root -g root $(STATICLIB) $(INSTALLDIR)
    install -m 755 -o root -g root $(SHAREDLIB) $(INSTALLDIR)
    ln -sf $(SHAREDLIB) $(INSTALLDIR)/$(VERLIBNAME)
    ln -sf $(VERLIBNAME) $(INSTALLDIR)/$(LIBNAME)   
#   ldconfig

clean:
    rm -f core Dist/*.* u2dtmp* $(MODULES) $(STATICLIB) $(SHAREDLIB) $(LIBNAME)

And here is an Apothecary branch with the changes for opencv, boost, and assimp: https://github.com/Jonathhhan/apothecary/tree/EmscriptenMultiThread

I am sure there is an easier way for boost, FreeImage and opencv, but at least it works...

Jonathhhan commented 2 years ago

I also got the example from the audioworklet emscripten branch https://github.com/juj/emscripten/tree/audio_worklets from @juj working. But not sure, how to integrate that into OF. It seems (for me) much simpler with the branch I am already using: https://github.com/tklajnscek/emscripten/tree/worklet_support from @tklajnscek

Jonathhhan commented 2 years ago

@roymacdonald one thing redarding the use of std::filesystem. I think its great and all the examples included in OF seem to work, but in ofxPd even pd.openPatch doesnt work anymore, and ofxLua (which I need for ofxOfelia) needs to be recompiled for use with std::filesystem (which shoould both be solvable). Because of that I use still boost::filesystem for now (but it just works with #define OF_USING_STD_FS 0).

And thanks again, we already achieved more than I expected. It still would be really nice, if offscreen canvas rendering and audioworklets could be used at the same time...

And kind of a local file loading mechanism (like with my emscriptenVideoPlayerExample) for audio files that can be read by ofxPd...

And webMIDI of course... (I know, I ask too much) ;)

roymacdonald commented 2 years ago

no problem! glad to be helpful.

So why is the ideo player able to load files while the audio is not? Is the video player not using a file system?

I suspect that pd.openPatch relies in its own file system implementation and that's why it does not work. Maybe there is a way to load the file from the OF side and then pass it to PD? I think that there is an override for openPatch that accepts a Patch object.

What is the difference between audioworklets and the other audio implementation there is"?

Jonathhhan commented 2 years ago

Yes, I guess ofxEmscriptenSoundPlayer can load sounds in a similar way (just that I implemented it for ofxEmscriptenVideoPlayer in a messy way, because I dont know how to do that better). I ask myself, if I can connect the source of an audiofile with ofSoundPlayer to the audioworklet and pass it to the inbuffer there (I will also try the addons you mentioned)? But it would also be great, if I can just connect the input from the soundcard to the audioworklet, like it was possible with the ScriptProcessorNode (my audioWorklet is not configured for input, yet https://github.com/Jonathhhan/openFrameworks/blob/AudioWorklet/addons/ofxEmscripten/libs/html5audio/lib/emscripten/audioworklet_tone_post.js).

    html5audio_sound_load: function(context_id, url){
        var request = new XMLHttpRequest();
        request.open('GET', UTF8ToString(url), true);
        request.responseType = 'arraybuffer';

        var id = AUDIO.lastSoundID++;
        AUDIO.soundGains[id] = AUDIO.contexts[context_id].createGain();
        AUDIO.soundGains[id].connect(audioWorklet);

        // Decode asynchronously
        request.onload = function() {
            AUDIO.contexts[context_id].decodeAudioData(request.response,
                function(buffer) {
                    AUDIO.soundBuffers[id] = buffer;
                },
                function(e){
                    console.log("couldn't decode sound " + id, e);
                }
            );
        };
        request.send();
        return id;
    },

The difference (very roughly) between the two implementations is, that this one works very well with OF but is not official (this is the one I use with OF): https://github.com/tklajnscek/emscripten/tree/worklet_support And this one does not work with OF yet, but it could become the official implementation: https://github.com/juj/emscripten/tree/audio_worklets Here is a discussion about the implementations: https://github.com/emscripten-core/emscripten/pull/12502

I added all the additional changes to my audioWorklet branch: https://github.com/Jonathhhan/openFrameworks/tree/AudioWorklet

Jonathhhan commented 2 years ago

Actually with this it works to play an audio file with drag and drop with Emscripten:

    html5audio_context_create: function(){
        try {
            // Fix up for prefixing
            window.AudioContext = window.AudioContext || window.webkitAudioContext;
            var context = new AudioContext({
                        latencyHint: "interactive",
                        sampleRate: 88200
                });

            // Fix issue with chrome autoplay policy
            document.addEventListener('mousedown', function cb(event) {
                context.resume();
                event.currentTarget.removeEventListener(event.type, cb);
            });

            var id = AUDIO.lastContextID++;
            AUDIO.contexts[id] = context;
            var fft = context.createAnalyser();
            fft.smoothingTimeConstant = 0;
            fft.connect(AUDIO.contexts[id].destination);
            fft.maxDecibels = 0;
            fft.minDecibels = -100;
            AUDIO.ffts[id] = fft;

    window.ondragover = function (e) {
        e.preventDefault()
        e.stopImmediatePropagation();
    }

    window.ondrop = function (e) {
        e.preventDefault();
        e.stopImmediatePropagation();
        console.log("Reading...");
        var length = e.dataTransfer.items.length;
        if (length > 1) {
            console.log("Please only drop 1 file.")
        } else {
            upload(e.dataTransfer.files[0])
        }
    }

    function upload(file) {
        if (file.type.match(/audio\/*/)) {
            var request = new XMLHttpRequest();
        request.open('GET', URL.createObjectURL(file), true);
        request.responseType = 'arraybuffer';

        var id2 = AUDIO.lastSoundID++;
        AUDIO.soundGains[id2] = AUDIO.contexts[id].createGain();
        AUDIO.soundGains[id2].connect(AUDIO.ffts[id]);

        // Decode asynchronously
        request.onload = function() {
        AUDIO.contexts[id].decodeAudioData(request.response,
        function(buffer) {
        AUDIO.soundBuffers[id2] = buffer;

        var source = AUDIO.contexts[id].createBufferSource();
        source.buffer = AUDIO.soundBuffers[id2];
        source.connect(AUDIO.soundGains[id2]);
        source.name = id2;
        source.done = false;
        source.paused = false;
        source.onended = function(event){
        event.target.done = true;
        }
        AUDIO.soundSources[id2] = source;
        source.startTime = AUDIO.contexts[id].currentTime - 0;
        source.start(0);
    },
    function(e){
    console.log("couldn't decode sound " + id, e);
    }
    );
        };
        request.send();
        //html5audio_sound_play(id,id2,0);
            console.log("This file seems to be an audio file.")
        } else {
            console.log("This file does not seem to be an audio file.")
        }
    }       
    return id;
        } catch(e) {
            console.log('Web Audio API is not supported in this browser',e);
            return -1;
        }
    },
roymacdonald commented 2 years ago

I see, thanks for explaining the difference. I am super busy now and cant really take a look at it. The funny thing is that it apparently is a problem on the Javascript side more than on the C++ side.

I will look at it for sure later this week. cheers

Jonathhhan commented 2 years ago

@roymacdonald alright, take your time (I also write a lot here for my own documentation...). Yes, its more a JS problem. One thing: I wonder, why I have to set the samplerate to 88200 in the JS audioContext while in OF it works with 44100... Maybe I should close this Issue (since OF works with Emscripten 3.0.1 and all the issues apart from audioWorklet are solved) and open a new one for audioWorklets and OF?

Jonathhhan commented 2 years ago

Had just one interesting thought. It seems possible to connect the audio of a html5 video with a node of the webAudio API: https://stackoverflow.com/questions/15118524/get-audio-from-an-html5-video With that, maybe its possible to connect the audio node from the video with the audioworklet, so that the audio of the video can be processed by ofxPd...

Jonathhhan commented 2 years ago

Yeah, its actually possible to connect any audio node to the worklet node (thats what i thought could be possible, but was not sure at all)... Here is a strange example: https://emscriptenpdprocessing.handmadeproductions.de/ Drag an .mp3 (it takes a moment for loading, but starts automatically) and manipulate it with reverb, lowpass and volume (the processing happens in ofxOfelia/ ofxPd). The sound sounds kind of distorted, but I am sure, thats because I made some programming mistakes and it should be easy to fix (and for a proof of concept it works). The page also needs to be reloaded after playing the first .mp3 file (it is not possible to load a second file yet), but that should be solvable, too... And it does not seem to work with firefox yet. The code is very messy, but I post it anyway (would be happy, if anybody can improve it...): Thats in library_html5audio.js:

    html5audio_stream_create: function(context_id, bufferSize, inputChannels, outputChannels, inbuffer, outbuffer, callback, userData, pthreadPtr){
        out("Buffer size: " + bufferSize);
                try {
            // Fix up for prefixing
            window.AudioContext = window.AudioContext || window.webkitAudioContext;
            var context = new AudioContext({
                        latencyHint: "interactive",
                        sampleRate: 88200
                });

            // Fix issue with chrome autoplay policy
            document.addEventListener('mousedown', function cb(event) {
                context.resume();
                event.currentTarget.removeEventListener(event.type, cb);
            });

            var id = AUDIO.lastContextID++;
            AUDIO.contexts[id] = context;
            var fft = context.createAnalyser();
            fft.smoothingTimeConstant = 0;
            fft.connect(AUDIO.contexts[id].destination);
            fft.maxDecibels = 0;
            fft.minDecibels = -100;
            AUDIO.ffts[id] = fft;

        // Initialize the pthread shared by all AudioWorkletNodes in this context
        PThread.initAudioWorkletPThread(AUDIO.contexts[context_id], pthreadPtr).then(function() {
        out("Audio worklet PThread context initialized!")
        }, function(err) {
        out("Audio worklet PThread context initialization failed: " + [err, err.stack]);
        });

        // Creates an AudioWorkletNode and connects it to the output once it's created
        PThread.createAudioWorkletNode(
        AUDIO.contexts[context_id],
        'native-passthrough-processor', 
        { numberOfInputs: 1,
        numberOfOutputs : 1,
        inputChannelCount : [1],
        outputChannelCount : [2],
        processorOptions: {
        inputChannels : inputChannels,
        outputChannels : outputChannels,
        inbuffer : inbuffer,
        outbuffer : outbuffer,
        bufferSize : bufferSize,
        callback : callback,
        userData : userData }}
        ).then(function(workletNode) {

        // Connect the worklet to the audio context output
        out("Audio worklet node created! Tap/click on the window if you don't hear audio!");
        workletNode.connect(AUDIO.ffts[context_id]);
                id2=0
                        AUDIO.soundGains[id2] = AUDIO.contexts[id].createGain();
        AUDIO.soundGains[id2].connect(workletNode);
        }, function(err) {
        out("Audio worklet node creation failed: " + [err, err.stack]);
        });

           window.ondragover = function (e) {
        e.preventDefault()
        e.stopImmediatePropagation();
    }

    window.ondrop = function (e) {
        e.preventDefault();
        e.stopImmediatePropagation();
        console.log("Reading...");
        var length = e.dataTransfer.items.length;
        if (length > 1) {
            console.log("Please only drop 1 file.")
        } else {
            upload(e.dataTransfer.files[0])
        }
    }

    function upload(file) {
        if (file.type.match(/audio\/*/)) {
            var request = new XMLHttpRequest();
        request.open('GET', URL.createObjectURL(file), true);
        request.responseType = 'arraybuffer';

        id2 = AUDIO.lastSoundID++;

        // Decode asynchronously
        request.onload = function() {
        AUDIO.contexts[id].decodeAudioData(request.response,
        function(buffer) {
        AUDIO.soundBuffers[id2] = buffer;

        var source = AUDIO.contexts[id].createBufferSource();
        source.buffer = AUDIO.soundBuffers[id2];
        source.connect(AUDIO.soundGains[id2]);
        source.name = id2;
        source.done = false;
        source.paused = false;
        source.onended = function(event){
        event.target.done = true;
        }
        AUDIO.soundSources[id2] = source;
        source.startTime = AUDIO.contexts[id].currentTime - 0;
        source.start(0);
    },

    function(e){
    console.log("couldn't decode sound " + id, e);
    }
    );
        };
        request.send();

        //html5audio_sound_play(id,id2,0);
            console.log("This file seems to be an audio file.")
        } else {
            console.log("This file does not seem to be an audio file.")
        }
    }       

        } catch(e) {
            console.log('Web Audio API is not supported in this browser',e);
            return -1;
        }
        return id;
    },

And thats in the audioWorklet:

/**
 * This is the JS side of the AudioWorklet processing that creates our
 * AudioWorkletProcessor that fetches the audio data from native code and 
 * copies it into the output buffers.
 * 
 * This is intentionally not made part of Emscripten AudioWorklet integration
 * because apps will usually want to a lot of control here (formats, channels, 
 * additional processors etc.)
 */

// Register our audio processors if the code loads in an AudioWorkletGlobalScope
if (typeof AudioWorkletGlobalScope === "function") {
  // This processor node is a simple proxy to the audio generator in native code.
  // It calls the native function then copies the samples into the output buffer
  var counter = 0;
  var inputChannels = 0;
  var outputChannels = 0;
  var inbuffer = 0;
  var outbuffer = 0;
  var bufferSize = 0;
  var callback = 0;
  var userData = 0;
  class NativePassthroughProcessor extends AudioWorkletProcessor {
    constructor (options) {
    super()
    inputChannels = options.processorOptions.inputChannels;
    outputChannels = options.processorOptions.outputChannels;
    inbuffer = options.processorOptions.inbuffer;
    outbuffer = options.processorOptions.outbuffer;
    bufferSize = options.processorOptions.bufferSize;
    callback = options.processorOptions.callback;
    userData = options.processorOptions.userData;
  }
    process(inputs, outputs, parameters) {
        counter = currentFrame % 128;
        if (counter == 0) {
            dynCall("viiii", callback, [bufferSize, inputChannels, outputChannels, userData]);
            const output = outputs[0];
            for (let channel = 0; channel < outputChannels; ++channel) {
                const outputChannel = output[channel];
                Module.HEAPF32.subarray(inbuffer >> 2, (inbuffer >> 2) + bufferSize * inputChannels).set(inputs[0][0]);
                outputChannel.set(Module.HEAPF32.subarray(outbuffer >> 2, (outbuffer >> 2) + bufferSize * outputChannels));
                }
            }
        return true
        }
    }
  // Register the processor as per the audio worklet spec
  registerProcessor('native-passthrough-processor', NativePassthroughProcessor);
}

Actually I think, the audioWorklet for OF should look like something above, only that I do something wrong with the inbuffer, because it sounds distorted (but in theory it works)...

Edit: The distortion is gone, it seems to work surprisingly well...