Latest Emscripten 2.0.33 works well, except file loading. [solved]

[Jonathhhan]

It was possible to compile with the latest Emscripten, for that I had to recompile the OF libs (FreeImage or freetype for example) with the current Emscripten and made some other small changes like using c++17 and std filesystem instead of boost (boost had an undefined symbol). I guess the issue now has to do with the glm qualifier glm::qualifier)0>&, this is the Java Script error message for the imageSubsectionExample (everything compiles fine): imageSubsectionExample.html:1 emscripten_set_main_loop_timing: Cannot set timing mode for main loop since a main loop does not exist! Call emscripten_set_main_loop first to set one up. printErr @ imageSubsectionExample.html:1 imageSubsectionExample.html:1 Aborted(native code called abort()) printErr @ imageSubsectionExample.html:1 imageSubsectionExample.js:1 Uncaught RuntimeError: Aborted(native code called abort()) at abort (imageSubsectionExample.js:1:34496) at _abort (imageSubsectionExample.js:1:119651) at std::__2::__throw_failure(char const*) (imageSubsectionExample.wasm:0x362df2) at std::__2::__fs::filesystem::__canonical(std::__2::__fs::filesystem::path const&, std::__2::error_code*) (imageSubsectionExample.wasm:0x380f6f) at ofToDataPath(std::__2::__fs::filesystem::path const&, bool) (imageSubsectionExample.wasm:0x55a2d) at ofApp::setup() (imageSubsectionExample.wasm:0x22653) at ofNode::onParentOrientationChanged(glm::qua<float, (glm::qualifier)0>&) (imageSubsectionExample.wasm:0x4638a) at std::__2::__function::__func<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&), std::__2::allocator<std::__2::shared_ptr<of::priv::Function<ofKeyEventArgs, std::__2::recursive_mutex> > ofEvent<ofKeyEventArgs, std::__2::recursive_mutex>::make_function<ofMainLoop>(ofMainLoop*, void (ofMainLoop::*)(ofKeyEventArgs&), int)::'lambda'(void const*, ofKeyEventArgs&)>, bool (void const*, ofKeyEventArgs&)>::operator()(void const*&&, ofKeyEventArgs&) (imageSubsectionExample.wasm:0x4c0d6) at ofEvent<ofHttpResponse, std::__2::recursive_mutex>::notify(ofHttpResponse&) (imageSubsectionExample.wasm:0x2a046) at ofCoreEvents::notifySetup() (imageSubsectionExample.wasm:0x4cb73) Is there a way to solve whether the boost, the glm or the std filesystem issue, so that is possible to load data with OF and the latest Emscripten (everything else seems to work fine)?

[Jonathhhan]

It also seems to be possible to record audio with the audioWorklet (could be a nice feature...): https://gist.github.com/flpvsk/047140b31c968001dc563998f7440cc1

[Jonathhhan]

@roymacdonald Just to give a short update (summary): audioWorklet works really well (and better than expected), even processing from video or audio input is possible. latency is better and less artifacts than with the current method. webMidi also works very well, also with ofxPd (but of course, only with the browsers that implemented webMidi). At the moment audioWorklet and midi out do not work together (but that should be solvable). Offscreen Canvas rendering works, but not together with audioWorklet (similar issue as with audioWorklet and webMidi, I guess). Still need to use FS 0 for ofxPd, but thats not really a problem... Will update my code soon...

[roymacdonald]

@Jonathhhan Awesome! thanks for working this out and keeping us posted! I haven't been able to look at any of it but I will as soon as I can. cheers

[Jonathhhan]

@roymacdonald thanks. Btw: It would be totally great, if it is possible to choose the midi device with a dropdown menu inside the (Emscripten) patch. Like I did it on the Java Script side: https://webmidimarkov.handmadeproductions.de/ Already tried it, but without luck (I think theoretically it should be possible)... Sending and receiving (most) values between Java Script and Emscripten is no problem (which is needed for that, I guess).

But anyway, I think updating OF for Emscripten and implementing audioWorklet / webMidi/ maybe offscreen rendering are two seperate steps. Because updating OF for Emscripten and fixing some of the bugs is quite easy and already done (mainly recompiling the libs), but the other stuff needs some thinking. I am happy, if I can help or answer any question.

[ofTheo]

@Jonathhhan not sure if this is the right place for this, but wondering if you have a clean branch with all your emscripten changes / fixes?

Would love to start working on getting clear fixes pulled in.
Happy if you want to try doing some bitesized PRs or if you want to just point us to a branch with all your fixes.

Thanks!! Theo

[Jonathhhan]

@ofTheo this is the branch: https://github.com/Jonathhhan/openFrameworks/tree/AudioWorklet But I will check it the next days, I am not sure if it is up to date (I think I made some additional changes that are not in the branch yet).

[Jonathhhan]

This sounds like a nice API for recording audio or video with OF and Emscripten and saving the result locally: https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API

[ofTheo]

Thanks @Jonathhhan! Here is the diff so far: https://github.com/openframeworks/openFrameworks/compare/master...Jonathhhan:AudioWorklet

[Jonathhhan]

I updated the branch. I removed the whole video / audio drag and drop thing (which would be nice to have, but maybe a bit to complicated for now). And I made an additional method for ofxEmscriptenVideoPlayer and ofxEmscriptenSoundPlayer: loadLocal(). Its for loading files from the local disc. For ofxEmscriptenSoundPlayer::loadLocal() it would be really nice, if it only gets a new sound id, if the file is actually selected in the file browser... At the moment the currently loaded file also stops, if the file browser is cancelled (for video it works fine, because there is no new id passed). It would be possible to pass the id with embind from java script to c++ only if the audio file is selected, but that is maybe not a very elegant way. It also seems that setLoop does not work with ofxEmscriptenSoundPlayer, but it is possible to set it in the html audio settings. And I found a way to optimize Emscripten video a bit (about 50 %): With setUsePixels(false); and setPixelFormat(OF_PIXELS_RGBA); together. Here is it in use: https://audiovideopdprocessing.handmadeproductions.de/ I also optimize some examples (like the one above), and will explain some things later... So far.

[Jonathhhan]

Now I also updated the examples: https://github.com/Jonathhhan/ofEmscriptenExamples They need the ofxPd addon, except the emscriptenGameOfLife example.

emscriptenAudioInput: https://audiovideopdprocessing.handmadeproductions.de/ This example shows, how to load audio from audio files video files and audio input into the input of the audioworklet, so that it can be processed with OF (or in this case ofxPd). It shows also the optimized video playback. Of course the audio should not in any case passed to the input of the audioWorklet and the audio input of OF (for processing with ofxPd for example), so maybe an argument for passing the audio to the audioWorklet input for ofxEmscriptenVideoPlayer and ofxEmscriptenAudioPlayer would be nice?

emscriptenAudioPlayer: https://audioplayer.handmadeproductions.de/ This example shows how to load (large) local audio files into Pure Data arrays. Its also an example for loading large arrays into Emscripten in general.

emscriptenGameOfLife: https://golnative.handmadeproductions.de/ An shader example. This works also with offscreen rendering and proxy to pthreads (like everything without audioworklets). It uses an .xml file for saving patterns, it would be nice to import and export it locally.

emscriptenWebMidiMarkov: https://webmidimarkov.handmadeproductions.de/ This patch shows how to use webMidi with OF and Emscripten. Midi out without user action works currently only without audioworklets (because they need to run in a web environment and not in a worker - or they need to communicate somehow). It would be great to choose Midi in and out inside the OF window. For now there is an alternative webMidi .html template.

[Jonathhhan]

For making audioWorklets work you need to install the Emscripten branch from @tklajnscek: https://github.com/tklajnscek/emscripten/tree/worklet_support

Hey @Jonathhhan, yeah I haven't been merging in the latest changes lately... I've gone ahead and rebased everything on 2.0.34 so this should work for you:

Clone the base SDK folder from https://github.com/emscripten-core/emsdk.git
emsdk install 2.0.34
emsdk activate 2.0.34
remove the content of <sdkdir>/upstream/emscripten
clone the worklet_support branch from my repo https://github.com/tklajnscek/emscripten.git in there
run npm i in there to get acorn etc installed

This should do it. Then go into tests and try running runner.py browser.test_audio_worklet and it should succeed.

The tone generator sample should also build and run - in tests/audioworklet/tone - check the .cpp file for instructions.

Let me know if it works for you :) Also, be aware that it doesn't look likely this will ever make it into Emscripten in it's current form :) https://github.com/emscripten-core/emscripten/pull/12502

I can confirm that it also works with Emscripten 3.0.1 (not sure about later versions...).

And some the OF libs for Emscripten need to be compiled with pthread support (at least boost), because thats needed for sharedArrayBuffer, which is needed for audioWorklets. The libs can also be used without pthreads.

Regarding compiling the libs (from earlier in this thread):

Regarding the OF Libs and pthread compilation: I needed to compile boost, FreeImage and assimp with pthread support. For opencv I just needed to change a path in the shell file (also for single thread compilation). For boost I needed to keep the old files and just replace the new ones. FreeImage was the most complicated one, I had to edit the Makefile.gnu in the FreeImage build folder (which gets deleted during compilation, if the apothecary folder is not renamed). Thats the edited FreeImage Makefile.gnu:

# Linux makefile for FreeImage

# This file can be generated by ./gensrclist.sh
include Makefile.srcs

# General configuration variables:
DESTDIR ?= /
INCDIR ?= $(DESTDIR)/usr/include
INSTALLDIR ?= $(DESTDIR)/usr/lib

# Converts cr/lf to just lf
DOS2UNIX = dos2unix

LIBRARIES = -lstdc++

MODULES = $(SRCS:.c=.o)
MODULES := $(MODULES:.cpp=.o)
CFLAGS ?= -O3 -fPIC -fexceptions -fvisibility=hidden -s USE_PTHREADS=1
# OpenJPEG
CFLAGS += -DOPJ_STATIC
# LibRaw
CFLAGS += -DNO_LCMS
# LibJXR
CFLAGS += -DDISABLE_PERF_MEASUREMENT -D__ANSI__
CFLAGS += $(INCLUDE)
CXXFLAGS ?= -O3 -fPIC -fexceptions -fvisibility=hidden -Wno-ctor-dtor-privacy -s USE_PTHREADS=1
# LibJXR
CXXFLAGS += -D__ANSI__
CXXFLAGS += $(INCLUDE)

ifeq ($(shell sh -c 'uname -m 2>/dev/null || echo not'),x86_64)
    CFLAGS += -fPIC
    CXXFLAGS += -fPIC
endif

TARGET  = freeimage
STATICLIB = lib$(TARGET).a
SHAREDLIB = lib$(TARGET)-$(VER_MAJOR).$(VER_MINOR).so
LIBNAME = lib$(TARGET).so
VERLIBNAME = $(LIBNAME).$(VER_MAJOR)
HEADER = Source/FreeImage.h

default: all

all: dist

dist: FreeImage
    mkdir -p Dist
    cp *.a Dist/
    cp *.so Dist/
    cp Source/FreeImage.h Dist/

dos2unix:
    @$(DOS2UNIX) $(SRCS) $(INCLS)

FreeImage: $(STATICLIB) $(SHAREDLIB)

.c.o:
    $(CC) $(CFLAGS) -c $< -o $@

.cpp.o:
    $(CXX) $(CXXFLAGS) -c $< -o $@

$(STATICLIB): $(MODULES)
    $(AR) r $@ $(MODULES)

$(SHAREDLIB): $(MODULES)
    $(CC) -s -shared -Wl,-soname,$(VERLIBNAME) $(LDFLAGS) -o $@ $(MODULES) $(LIBRARIES)

install:
    install -d $(INCDIR) $(INSTALLDIR)
    install -m 644 -o root -g root $(HEADER) $(INCDIR)
    install -m 644 -o root -g root $(STATICLIB) $(INSTALLDIR)
    install -m 755 -o root -g root $(SHAREDLIB) $(INSTALLDIR)
    ln -sf $(SHAREDLIB) $(INSTALLDIR)/$(VERLIBNAME)
    ln -sf $(VERLIBNAME) $(INSTALLDIR)/$(LIBNAME)   
#   ldconfig

clean:
    rm -f core Dist/*.* u2dtmp* $(MODULES) $(STATICLIB) $(SHAREDLIB) $(LIBNAME)

And here is an Apothecary branch with the changes for opencv, boost, and assimp: https://github.com/Jonathhhan/apothecary/tree/EmscriptenMultiThread

I am sure there is an easier way for boost, FreeImage and opencv, but at least it works...

Its easy to switch between audioWorklet and the old audio processing method, just switch between: PLATFORM_LDFLAGS += --js-library $(OF_ADDONS_PATH)/ofxEmscripten/libs/html5audio/lib/emscripten/library_html5audio.js and PLATFORM_LDFLAGS += --js-library $(OF_ADDONS_PATH)/ofxEmscripten/libs/html5audio/lib/emscripten/library_html5audioWorklet.js in config.emscripten.default.mk.

[Jonathhhan]

I made some changes regarding loading local audio and video files (or urls in general). It needs an ofEvent and embind bindings, but I think its more flexible than my old solution. Its implemented in this patch: https://github.com/Jonathhhan/ofEmscriptenExamples/tree/main/emscriptenAudioInput The easiest solution would be, if various load() methods also accept urls (or blobs like blob:https://audiovideopdprocessing.handmadeproductions.de/6e7eff3f-5efb-4dee-9953-f1bc9578de54) as a string in addition to std::filesystem::path, then nothing more needs to be changed...

Whats also very interesting is the Emscripten filesystem. It seems possible to read and write with it like:

  const data = Uint8Array.from(fs.readFileSync('./flame.avi'));
  Module.FS.writeFile('flame.avi', data);

It seems to read the data without errors into the filesystem, but it wasnt possible for me to read them with OF from the virtual filesystem. I read in this article about it: https://itnext.io/build-ffmpeg-webassembly-version-ffmpeg-js-part-3-ffmpeg-js-v0-1-0-transcoding-avi-to-mp4-f729e503a397

[Jonathhhan]

I have some issues with the ofxEmscriptenSoundPlayer: Besides that setLoop() does not seem to work (but it works in the html5 audio player with source.loop = true), I have to stop the player before I load a new file otherwise it crashes. I if stop the player if it is already stopped, then the player also crashes... Also its not possible to call play directly after loading (maybe thats normal, because it needs some time)...

Two ideas, not sure if they make sense: Would it be possible to stream the audio from the ofxEmscriptenAudioPlayer the same way as the ofxEmscriptenVideoPlayer does it? Because it seems to need much less memory, because audio isnt buffered completely (great for large files)...

And it would be great, if it is possible to load every kind of file from the url (or blob url). loading local video and audio is possible (with some code modification), but loading local images for example did not work.

[ofTheo]

Hey @Jonathhhan

Trying to get C++17 in via this PR: https://github.com/openframeworks/openFrameworks/pull/6844

-std=c++17 is getting past through to the compiler

/emsdk/upstream/emscripten/em++ -g0 -DDEBUG -Wall -std=c++17 

but I am running into this error:

/src/libs/openFrameworks/utils/ofFileUtils.cpp:650:12: error: use of undeclared identifier 'getegid'
229
        }else if (getegid() == info.st_gid){

Did you ever get OF working with C++17 and not using boost for the filesystem stuff?

[Jonathhhan]

Hey @ofTheo, if I remember right, then it worked for the graphicsExample, because there is no data that needs to be loaded. But since most of the patches need data I gave up with that. I changed a lot in the java script file for the ofxEmscriptenAudioPlayer. It loads instantly now, the issues are gone, and it needs much less memory. Not sure if I destroyed some of the functionalty, but it should work if it is done right. Basically it replaces the audioBuffer with MediaElementSource. Here is a (working) draft: https://github.com/Jonathhhan/ofEmscriptenExamples/blob/main/emscriptenAudioInput/library_html5audioWorklet.js

(Its also working without audioWorklets, just need to change 1 or 2 variables...) Only drawback so far, before loading a new sound the old one needs to be unloaded with soundPlayer.unload() for stopping the old sound and revoking the url (maybe its possible to include that in load(), but I dont know how to get the old sound id there...).

Also updated the branch my example to show the changes: https://audiovideopdprocessing.handmadeproductions.de/

I thought about the urls that I create from JS: They maybe only work because they are send back and read from JS. And that works only for ofxEmscriptenVideoPlayer and ofxEmscriptenSoundPlayer (which is very nice, by the way...). Something like ofImage would need a different pointer (I am only guessing). Or I have to check outhow to load local files into the Emscripten filesystem and read from there (some help with that is kindly appreciated)...

[Jonathhhan]

Hey @ofTheo and @roymacdonald, what do you think about my changes of the ofxEmscriptenAudioPlayer? Does it make sense? Pan should also be implementable easily: https://stackoverflow.com/questions/20287890/audiocontext-panning-audio-of-playing-media

var ambientAudio = new Audio('./ambientVideo.mp4');

document.addEventListener('keydown', ambientAudioControl)
function ambientAudioControl(e) {
  if (e.keyCode === 37) panToLeft()
  if (e.keyCode === 39) panToRight()
  if (e.keyCode === 40) panToStereo()
}

const ambientContext = new AudioContext();
const source = ambientContext.createMediaElementSource(ambientAudio);
const ambientPan = ambientContext.createStereoPanner()

function panToLeft(){ ambientPan.pan.value = -1 }
function panToRight(){ ambientPan.pan.value = 1 }
function panToStereo(){ ambientPan.pan.value = 0 }

source.connect(ambientPan)
ambientPan.connect(ambientContext.destination)
ambientAudio.play()

Regarding unvoking the urls: Maybe also create an unload() method for ofxEmscriptenVideoPlayer for that? Or better (but no idea how to do that): Unvoke the old url automatically if a new file is loaded...

And one idea regarding drag and drop: If it would be possible to define a html drop area from OF, then drag and drop would make sense (right now it works, but only for the whole window as the drop area).

[Jonathhhan]

So I added pan for audio and video...

And I ask myself: Is it really necessary to create several audioContexts or would one be enough (I ask because it seems not possible to connect one node of one context to a node of another context, and for what I did one audioContext with id 0 was enough)? And it would be easier then to connect the audio node from the video player to the audioContext. I understand why I need several sound and video id's (one for every player) but not sure about the audioContext. https://stackoverflow.com/questions/60306160/multiple-audio-contexts-whats-the-use-case

And a stream is created as soon as I have an ofApp::audioReceived() and / or an ofApp::audioRequested() method?

And one idea regarding connecting the audio and video player to the audiostream: They could get a method like audioPlayer.connectToStream() or something like that. And for switching between the old audio stream method (scriptProcessor: https://developer.mozilla.org/en-US/docs/Web/API/BaseAudioContext/createScriptProcessor) and audioWorklet there could be a method like ofxEmscriptenSoundStream.connectToAudioWorklet()... (with scriptProcessor as the default)? And one issue with sharedArrayBuffer and soundPlayer.getSystemSpectrum() (which wasnt really needed yet from my side - and is also doable with ofxPd): Uncaught TypeError: Failed to execute 'getFloatFrequencyData' on 'AnalyserNode': The provided Float32Array value must not be shared. (can be avoided with -s PTHREADS=0, but then audioWorklet does not work).

[roymacdonald]

Hey @Jonathhhan Thanks for doing all this. I think taht the players, both video and audio should behave in the same way as the others do, so switching from one to the other makes no difference. So I think that the pan function should be as in ofBaseSoundPlayer where you pass float to set the pan, instead of just having left or right.

If you do really need to use a sound stream to make it work then ofxEmscriptenSoundPlayer and ofxEmscriptenVideoPlayer should both inherit from `ofBaseSoundOutput'

[Jonathhhan]

@roymacdonald you already set a float for pan, but from -1 to 1. I think in OF its 0 - 1? No problem to change that (I hope I understood it right). See it here: https://audiovideopdprocessing.handmadeproductions.de/ What is more difficult (for me) is to set the right initial volume (default is 1), but with pan it gets easily distorted then. Right now I set the initial volume too low, but its not distorted. I have to find the right value, or lower the volume if it gets panned (I guess the volumes of both channels are added at the moment). But it is also great, if it behaves like ofBaseSoundPlayer too.

ofxEmscriptenSoundPlayerand ofxEmscriptenVideoPlayer alone dont need a sound stream. I guess the soundstream is needed for audio input and for connecting the nodes from audio and video player to the input (so that I can process it with ofxPd ;) ), but maybe only if audioworklet is not used. not sure about that.

[roymacdonald]

tha works nicely. I couldnt notice any distortion while moving the pan but I just used my laptop's speakers. No headphones atm. pan is -1 to 1 this is what the documentation says /// \param pan range is -1 to 1 (-1 is full left, 1 is full right).

ofxEmscriptenSoundPlayer and ofxEmscriptenVideoPlayer alone dont need a sound stream. I guess the soundstream is needed for audio input and for connecting the nodes from audio and video player to the input (so that I can process it with ofxPd ;) ), but maybe only if audioworklet is not used. not sure about that.

I see. It depends on what you want to achieve if you need to use a soundstream or not. I am not sure what would be the best option here. Maybe a hybrid, which uses the sound stream if it is connected to it, otherwise it uses what you already implemented through the audio worklet?

[Jonathhhan]

@roymacdonald Yes, I already lowered the initial volume, maybe a bit too much. Another solution would be, to leave the initial value to 1 and lower it with setVolume when needed (like for panning a loud audio file).

Maybe a hybrid, which uses the sound stream if it is connected to it, otherwise it uses what you already implemented through the audio worklet?

That sounds good. The main problem with the audioWorklet is (in my opinion) that it relies on a non official Emscripten branch from @tklajnscek, and I dont know if it will be up to date. But we can ask the author and maybe he will update it for himself. He updated it for 3.0.1, not sure about the current and future versions. With this version it works great. But I also think, if Emscripten audioWorklet will be released officially at some point, the OF implementation does not need to change much.

[Jonathhhan]

I have some questions about the filesystem stuff: It works now to load images, sound, text, etc. into the virtual filesystem, which is quite nice. But I have the feeling, that it still makes sense to have .loadUrl(url);for the audio and video player, because it stays on the java script side (does not make sense to load it into the filesystem for that?). What do think about that? I also had a closer look at ofxEmscriptenURLFileLoader but I dont really know how to use it for my purpose (loading something as url into ofxEmscriptenAudioPlayer and ofxEmscriptenVideoPlayer) or in general.

[roymacdonald]

Hey @Jonathhhan At least from what I can see here it already would work with an URL.

I think there should not be a loadUrlfunction in these clases to be consistent with the rest of OF classes. For instance, take a look here to see how ofImage handles loading from either an URL or from a local file. I think that these classes you are working on should implement a similar thing. It would be also a good idea to allow these to load from an ofBuffer object as well so async loading can be done. Again, look here for how ofImage implements such.

ofxEmscriptenURLFileLoader is used internally by ofURLFileLoader file loader so you shouldn't need to use it directly, rather instead simply using ofURLFileLoader

Check this example https://github.com/openframeworks/openFrameworks/tree/master/examples/input_output/imageLoaderWebExample I think that these classes should behave in the same way as ofImage does in it.

Let me know how it goes. best

[Jonathhhan]

Hey @roymacdonald thanks for the examples. that makes totally sense and would be much better than my solution. I will give it a try, but not sure if I am able implement to implement that into ofxEmscriptenAudioPlayer and ofxEmscriptenVideoPlayer. The only additional methods from ofxEmscriptenAudioPlayer are getDurationSecs() and getDurationMS() (they were already there), but that is quite handy. And .setUsePixels() from ofxEmscriptenVideoPlayer which is really useful. And I added pan to the sound from ofxEmscriptenVideoPlayer not sure, if that makes sense... Regarding images I wonder, if it is better to load them as an url (created from a local file) or to load them into the virtual filesystem (and delete them emmediatly after they are loaded into OF).

[roymacdonald]

The only additional methods from ofxEmscriptenAudioPlayer are getDurationSecs() and getDurationMS() (they were already there), but that is quite handy. And .setUsePixels() from ofxEmscriptenVideoPlayer which is really useful. And I added pan to the sound from ofxEmscriptenVideoPlayer not sure, if that makes sense...

I just realized that ofSoundPlayer does not have a get duration method. which is quite strange that has not been added so far. I think that these should be getDuration() and getDurationMS(), where the former returns to seconds, this way it is consitent with ofVideoPLayer.

Let me know if you need help implementing ofxEmscriptenAudioPlayer or ofxEmscriptenVideoPlayer. Keep on with this great contribution you are doing! Many thanks for it. cheers

[ofTheo]

@roymacdonald would you have some time to help @Jonathhhan wrap up all the fairly clean emscripten fixes into a PR? Happy to leave the more iffy stuff to a separate PR.

I got C++17 working for emscripten in this PR https://github.com/openframeworks/openFrameworks/pull/6844/files

Just needed an extra header include and the C++17 flag in the end.

[roymacdonald]

@ofTheo sure! no prob.

@Jonathhhan can you please point me to the repo in which you are pushing the emscripten upgrades

[Jonathhhan]

Hey @ofTheo and @roymacdonald, thats great news.

I tried the url stuff from ofImage, but without luck. But urls do work if I replace audioPlayer.load(ofToDataPath(fileName) with audioPlayer.load(fileName). With the first one I can load files from the filesystem. Same for the video player. Maybe that helps.

And I removed loadUrl()...

And here is the branch: https://github.com/Jonathhhan/openFrameworks/tree/AudioWorklet

I also made the audio context and the fft global (would also be possible to have a seperate fft for every sound id)... I hope I didnt break too much... ;) But I have all the older steps here if I should revert something... And of course, take what you think makes sense and if you have any questions I am happy to answer. Best.

[Jonathhhan]

Actually the sound player works with url and local file with this methods:

    bool load(const std::filesystem::path& fileName, bool stream = false);
    bool load(const std::string& fileName, bool stream = false);

Somehow it does not work for the video player...

[Jonathhhan]

This works for the video player (but maybe not a nice solution):

bool ofxEmscriptenVideoPlayer::load(const std::string fileName){
    if (ofFile::doesFileExist(ofToDataPath(fileName))){
        html5video_player_load(id, ofToDataPath(fileName).c_str());
        return true;
    } else {
        html5video_player_load(id, fileName.c_str());
        return true;
    }
}

I updated the branch, video and audio player have the method above now. This way I dont have to change a lot. You can remove or change it if it is not save.

[roymacdonald]

Hi @Jonathhhan This is the chunk of code you need in order to tell if it is an URL or not. I think there should be an OF function wrapping this up. It can be quite handy.

#include "uriparser/Uri.h"

    auto uriStr = fileName.string();
    UriUriA uri;
    UriParserStateA state;
    state.uri = &uri;

    if(uriParseUriA(&state, uriStr.c_str())!=URI_SUCCESS){
        const int bytesNeeded = 8 + 3 * strlen(uriStr.c_str()) + 1;
        std::vector<char> absUri(bytesNeeded);
    #ifdef TARGET_WIN32
        uriWindowsFilenameToUriStringA(uriStr.c_str(), absUri.data());
    #else
        uriUnixFilenameToUriStringA(uriStr.c_str(), absUri.data());
    #endif
        if(uriParseUriA(&state, absUri.data())!=URI_SUCCESS){
            ofLogError("ofImage") << "loadImage(): malformed uri when loading image from uri " << _fileName;
            uriFreeUriMembersA(&uri);
            return false;
        }
    }
    std::string scheme(uri.scheme.first, uri.scheme.afterLast);
    uriFreeUriMembersA(&uri);

    if(scheme == "http" || scheme == "https"){
                // LOAD FROM URL. 
        //return ofLoadImage(pix, ofLoadURL(_fileName.string()).data);

    }

    std::string fileName = ofToDataPath(_fileName, true);
//LOAD FROM FILESYSTEM

I will go through all the changes, updates and upgrades you have done and get back to you. cheers

[Jonathhhan]

@roymacdonald thanks a lot. And just to mention: The old html5audio_stream_create method is not implemented in my branch yet, but I guess it can be easily replaced... Without replacing it the stream works only with audioWorklet (but everything else should work independently from that).

[Jonathhhan]

Last update for now. The video player works now with url check (but I had to add "blob" to if(scheme == "http" || scheme == "https" || scheme == "blob" )). And the audio player with those 2 methods (because the url check didnt work there somehow - because the file / url is send as a std::filesystem::path and not as a std::string):

    bool load(const std::filesystem::path& fileName, bool stream = false);
    bool load(const std::string& fileName, bool stream = false);

And here are some of the updated examples: https://github.com/Jonathhhan/ofEmscriptenExamples

@ofTheo

Just needed an extra header include and the C++17 flag in the end.

Thats nice. I can confirm that it works, just needed to add your changes from ofConstants.h, ofFileUtils.cpp and ofUtils.h...

[Jonathhhan]

Here are some good news regarding the future of Emscripten and Audioworklets (at the end of the discussion): https://github.com/emscripten-core/emscripten/pull/12502#issuecomment-1040665141

[Jonathhhan]

Just found another inconsistency between ofVideoPlayer and ofEmscriptenVideoPlayer with getDuration(). With ofVideoPlayer I get values between 0 and 1, while with ofEmscriptenVideoPlayer I get the duration in seconds.

[roymacdonald]

duration between 0 and 1 sounds like a bug? Although what you get is dependent on the platform you are using. On which platform did you see this?

[Jonathhhan]

Its Ubuntu, and I am sorry, I meant getPosition() of course...

[ranjithshegde]

I was able to get it to compile. I mostly needed to recompile the core lib dependencies.

@roymacdonald I have the same errors as you. Could you please inform me what is it that you recompiled ?

[roymacdonald]

I was able to get it to compile. I mostly needed to recompile the core lib dependencies.

@roymacdonald I have the same errors as you. Could you please inform me what is it that you recompiled ?

I really cant recall. But i think that I recompiled emscripten using apothecary, maybe i had to recompile some other things using apothecary as well. That is what would make sense to me

[ranjithshegde]

@roymacdonald Thanks for your response. Sadly I dont know what apothecary is. I checked the git repo and it seems like there are some scripts in there, specific to emscripten was just a method to install it in docker.

Could you please point me towards any resources to understand what that repo is? Or how I would use them?

[roymacdonald]

@ranjithshegde ah sorry. Apothecary is an openframeworks tool for keeping track and updating the libraries it uses. It is in github.com/openframeworks/apothecary Just download that repo into scripts/apothecary and run from there. What you need to do is to find the emscripten "formula", which is also inside the scripts folder and change the version to the latest one and run apothecary so it updates emscripten