search

samdauwe / BabylonCpp

A port of Babylon.js to C++
Apache License 2.0
284 stars 38 forks source link

Compilation to WASM #2

Open gotwig opened 6 years ago

gotwig commented 6 years ago

Is the compilation to WASM in the works?

I realy love your project!

samdauwe commented 6 years ago

Hi gotwig,

Compilation to WASM is definitely planned. I am almost finished updating the codebase with all the changes for the upcoming v3.2 release of Babylon.js. Next is getting the samples back working, after that I was planning to look into supporting WebAssembly. So expect some updates on WASM support in 1-2 weeks.

If you are interested in contributing to this project let me know, there is plenty of work left for getting the same functionality as the Babylon.js framework.

gotwig commented 6 years ago

Awesome, this is nice to hear.

I am working on the BabylonJS Editor, and have a plugin similar to UE4 blueprints in mind.

With WASM I hope that people can visualy work with nodes , and implement the functions they need for their gamecode in their language of choice, instead of just javascript. Appart of that, all underlying code could at some point be realized in C++ as well. I worked in the past a lot with C++ and SFML. A lot of dreaming, but right now im working on a bachelor thesis and dont have that much time left to concentrate on two projects, but I still would love to stay in touch with you.

Can you maybe add me on facebook or twitter?: Twitter: https://twitter.com/masteredu Facebook: https://www.facebook.com/gotwig

Thanks again for your hard work!

samdauwe commented 6 years ago

Thanks for the information! Having an UE4 blueprints like implementation in the BabylonJS Editor would be a nice addition. Do you already have any ideas about the Javascript library you are planning to use for implementing the UE4 blueprints in the BabylonJS Editor? NoFlo ( https://flowhub.io ) seems suitable at first sight but there might be others. In case you want to use C++, you might have a look at Node Editor in ImGui ( https://github.com/thedmd/imgui-node-editor ). Btw, ImGui also runs in the browser: https://pbrfrat.com/post/imgui_in_browser.html

Good luck with your bachelor thesis, is the topic related to 3D (C++/SFML/WebGL)?

Having the BabylonCpp samples working with WebGL 2 and WASM in the browser, would be a nice milestone. But probably this will be not that easy as it sounds. Anyway I will keep you updated on my progress.

Thanks!

gotwig commented 6 years ago

Hey, Do you already have any ideas about the Javascript library you are planning to use for implementing the UE4 blueprints in the BabylonJS Editor? I tested out litegraph.js , and probably I am going to use it. It supports afaik around 1000 nodes on a canvas so it should be efficent enough, and it works via canvas. The underlying code behind that may be implemented in something different.

I did take a look at NoFlo but its more for Flow Based Programming, so encapsulating large code bases into smaller nodes to work with, but the blueprint approach of ue4 is more detailed, handling variables and for loops itself in nodes.

Good luck with your bachelor thesis, is the topic related to 3D (C++/SFML/WebGL)? The topic is to revamp the UI of BabylonJS, improve the user experience and implement this blueprint-like thing I described ^^ so no 3D :x: I think there are a lot of other people who are better in this topic then me, and I think its good that everyone does what hes best in, and I worked a lot with UI and think its nowadays for game engines realy important to make them accessible to a greater audience then just coders (see ue4, they even have very extensive help functions everywhere, describing nearly everything you see on screen)

samdauwe commented 6 years ago

Yes, litegraph.js looks better suited for your use case than NoFlo. I have never used it before, but it looks like you can create graphs similar to UE4 blueprints based on the screenshot on the github page.

The editor of BabylonJS is much more advanced (and more difficult for people that are new to the framework) than the one of ThreeJS. Having the blueprints functionality in the editor would make it much easier for a greater audience to get into using the BabylonJS framework. When you have something ready I can try out, let me know. Especially for creating animations I would use the blueprints functionality.

pthom commented 4 years ago

Compilation to Wasm is under very active work now. The relevant branch is wasm_experiment.

See the first two PR about this subject:

Note to Sam: I think we should continue our discussions about this subject in this issue, in order to avoid having to llook for discussion history in closed PRs.

pthom commented 4 years ago

I did not look in full detail into all the changes that you made, but maybe we could target also OpenGL ES3.0 on desktop for better consistency with the web version, unless you think that is not necessary? I am not very experienced in OpenGL versions actually. However, I think that targeting the same versions on web and desktop would be a good idea, and would simplify the maintenance. Are you willing to investigate on this?

Yes, I also think it will simplify the maintenance and the testing. I am currently working in the branch _v4_renderingissues to test if we can get it working with OpenGL ES 3.0 on desktop.

I am currently working on a different subject (async loading , in order to avoid the monstruous > 360MB >preload).

For this, the 4 things I mentionned before should probably be modified:

There are 4 things to set: they are different between desktop ad emscripten. May be they need to be investigated.

  • OpenGL Version: see Select_Gl_Version() inside runner_glfw.cpp, runner_sdl.cpp and runner_emscripten.cpp
  • Glsl version: see GlSlVersion() inside these same files
  • OpenGl Loader: see InitGlLoader() inside the same files: we use glad on desktop, and nothing on emscripten
  • Glsl shader versions: see src/BabylonCpp/include/babylon/shaders/shadersinclude/glsl_version_3.h. it also varies between desktop and emscripten

Concerning the glad loader, it was generated using the online generator. I do not remember exactly which options I had chosen, but the content of external/glad/include/glad/glad.h seems to suggest that it would support the following apis gl=4.4, gles2=3.2, gles1=1.0.

Thanks for this information, I will try out the online generator.

Some questions concerning my investigations on the async loading.

My intent is to emulate the javascript async mode, even in C++. For this we need to have a "service" that emulates the background browser process. This service will download files in the background and call the success/error callbacks on completion.

I would prefer not to reinvent this wheel, and introduce possible bugs because it is actually complex. I was thinking of adding a new library, such as xhawk18/promise-cpp, which add javascript-like promises. In order to have a "service", we would need to add also boost.asio, so that we could write code like this

int main() {
    boost::asio::io_service io; // this is the background service
    loadfile(io, filename).then([=] {
       processCallbacks();
    });
    io.run();
    return 0;
}

What do you think of adding such dependencies?

Using boost.asio and xhawk18 is fine for me. I did not use this library before but it seems like a standalone module that does not require the full boost library. For now I think we could use any depedency needed and look in the future if we can avoid them.

Related to this, the team working on BabylonJS native implemented their own asynchronous dispatching system (with no external depedencies) that is included in the arcana.cpp repo but it relies on C++ coroutines so we need to update to C++20. An example of using the async dispatching system can be found in this example. But boost asio is more used and therefore better tested.

Other remark: In the future, I would suggest also considering adding another dependency, C++ requests that would make loading from urls simple.

Thanks for the advice.

Cheers, Sam

pthom commented 4 years ago

Some more details concerning your idea to switch to OpenGL ES 3.0.

samdauwe commented 4 years ago

Compilation to Wasm is under very active work now. The relevant branch is wasm_experiment.

See the first two PR about this subject:

* [PR1: initial port efforts](https://github.com/samdauwe/BabylonCpp/pull/63)

* [PR2: first working version of BabylonStudio](https://github.com/samdauwe/BabylonCpp/pull/64), with a [demo video here](http://traineq.org/BabWasm.mp4)

Note to Sam: I think we should continue our discussions about this subject in this issue, in order to avoid having to llook for discussion history in closed PRs.

Ok, I will copy my previous reply here.

samdauwe commented 4 years ago

Some more details concerning your idea to switch to OpenGL ES 3.0.

* Under emscripten, glad is ignored and actually glad.h is replaced by a dummy include [`external/glad_wasm_dummy/include/glad/glad.h`](https://github.com/samdauwe/BabylonCpp/blob/wasm_experiment/external/glad_wasm_dummy/include/glad/glad.h)  that shouid include the relevant OpenGL ES headers. I am _not_ sure I selected the correct headers there. Could you look into it?

* May be it would be a good idea to regenerate a glad loader that support only OpenGL ES3, because as of now we encounter some cases where the emscripten code will not compile correctly.
  If you look at [`src/BabylonImGui/src/GL/gl_rendering_context.cpp`](https://github.com/samdauwe/BabylonCpp/blob/wasm_experiment/src/BabylonImGui/src/GL/gl_rendering_context.cpp) , you will see this code block that shows that some hints about this:
// There is an issue with emscripten builds: the build only works of if glfw is included, 
// although we should not need it! 
// We have errors like "use of undeclared identifier 'GL_MULTISAMPLE', 'GL_BLEND_SRC', etc."
#include <GLFW/glfw3.h>

Hello Pacal,

Below you can find some updates from my side related to the update to OpenGL ES 3.0.

Update Engine code to v4.1.0-alpha.17 The Engine class code that is currently in the master branch is still from v3.3.0 (2018/10/02) which make it difficult to debug while all other code is on v4.1.0-alpha.17. So I updated the Effect and Engine classes to v4.1.0-alpha.17. The result of the update was that nothing was rendering anymore, I spent 2 days checking code and I finally ended up comparing each OpenGL call between the JS and C++ version. The error was in 2 lines that were not properly ported but now it is working.

Update to OpenGL ES 3.0 The GLAD loader was regenerated for OpenGL ES 3.0 - Core profile for both release and debug version. WebGL 2.0 = OpenGL ES 3.0 and GLSL version 300 es, so this GLAD loader that was generated should match what Emscripten is using.

When using the definition "#define GLFW_INCLUDE_NONE" there were some errors like "use of undeclared identifier 'GL_MULTISAMPLE'. These were resolved by this changset. I also had to change the glad loader to use GLES instead of GL (like you can see in this changset)

Finally, I removed the helper header and function that set the GLSL 330 version for consistency with the BabylonJS version.

What works Most examples are working, wireframe rendering works again: afbeelding

What does not work Most of the issues described in this issue report. Furthermore, there seems to be an issue with the textures that are rendered in black color. I did not test yet on Windows.

For the moment all changes are in the v4_rendering_issues branch. For the next couple of days I will continue with debugging the rendering issues.

Cheers, Sam

pthom commented 4 years ago

Hello Sam,

This is very good news!

I started to work on the merge of the two branches , see the wasm_v4 branch; but it is not that easy, because I had to modify some opengl settings also in my branch. The wasm_v4 branch is my ongoing attempt to merge the two branches. I had to cancel some of my changes in it after the merge, so that it is quite broken at the moment.

I had to stop however, because the OpenGL ES mode in your v4_rendering_issues branch does not work on my side, neither on linux, nor on osx. Here are the results of my tests on your branch:

Do you have some ideas about this, expecially on linux?

For mac, there is perhaps a solution with https://github.com/google/angle , which provides OpenGL ES library bindings on Windows, Linux, Mac and Android. Angle is a bit heavy and its compilation is not easy, so if Babylon were to use it, may be it would be easier by using the angle package from vcpkg. This is still very speculative.

On my side, I started to work on having async file loading, even in native mode (it is better to debug this before even trying to get it to work under emscripten).

It is not easy: I tried to not use boost.asio (too heavy). For a while I considered using libuv, and finally I developed an api from scratch.

My first attempt was to have everything threaded (even the callbacks); but then I realised that the callback need to be synchronous otherwise different threads will make conflicting openGL calls at the same time.

So my api provides "async file loading" + "synchronous calbacks" : you can have a look at it in the wasm_v4 branch, in the file babylon/asio/asio.h.

When using this new API for file loadings, I encountered quite a lot of issues with the lambda captures (need to capture the pointer by copy), which required some changes in engine.cpp and scene.cppwhich I will need to backport in your branch.

Cheers!

samdauwe commented 4 years ago

Hello Pascal,

Hello Sam,

This is very good news!

I started to work on the merge of the two branches , see the wasm_v4 branch; but it is not that easy, because I had to modify some opengl settings also in my branch. The wasm_v4 branch is my ongoing attempt to merge the two branches. I had to cancel some of my changes in it after the merge, so that it is quite broken at the moment.

I had to stop however, because the OpenGL ES mode in your v4_rendering_issues branch does not work on my side, neither on linux, nor on osx. Here are the results of my tests on your branch:

* under my linux VM, [eglCreateContext](https://www.khronos.org/registry/EGL/sdk/docs/man/html/eglCreateContext.xhtml) fails inside `external/glfw/src/egl_context.c:582` with an error `EGL_BAD_CONFIG`

* under MacOS, it seems that libEGL is not supported. I am sorry to discover this now, it seems it is [only supported under iOS and tvOS](https://developer.apple.com/documentation/opengles)

* windows fails to build

Do you have some ideas about this, expecially on linux?

Hmmm, so by using OpenGL ES we are getting issues on Linux VM and MacOS that we were not getting before. I pushed some fixes to the shader code processing this afternoon that fixed a couple of examples. Procedural textures are now also working again on my side. But the changes will not fix the rendering issues you describe, I assume.

I will have a look at the Windows build issues. Regarding the Linux rendering issues, I have no idea yet I can try on a different machine or in a VM.

Related to the build issue topic: I noticed that the builds fail when using the latest CMake 3.16 version. We are using functionality that has been deprecated, I can forward you the error if you are interested. For now I downgraded my CMake version to v3.11.

For mac, there is perhaps a solution with https://github.com/google/angle , which provides OpenGL ES library bindings on Windows, Linux, Mac and Android. Angle is a bit heavy and its compilation is not easy, so if Babylon were to use it, may be it would be easier by using the angle package from vcpkg. This is still very speculative.

Yes, Angle is used by the browsers as the graphics drivers abstraction layer on Windows. I you run the BabylonJS inspector on a browser on Windows it should show in the driver info panel that it is using Angle. On Linux it is not used according to me. I looked into Angle before but it was difficult to build, like you mentioned above.

An alternative is using bgfx which supports WebGL 2.0 and emscripten. It has different rendering backends such as Metal and is supported on many platforms. I am willing to switch to bgfx (or any similar library) if that would resolve many of the rendering issues we have on different platforms.

On my side, I started to work on having async file loading, even in native mode (it is better to debug this before even trying to get it to work under emscripten).

That is good news!

It is not easy: I tried to not use boost.asio (too heavy). For a while I considered using libuv, and finally I developed an api from scratch.

My first attempt was to have everything threaded (even the callbacks); but then I realised that the callback need to be synchronous otherwise different threads will make conflicting openGL calls at the same time.

So my api provides "async file loading" + "synchronous calbacks" : you can have a look at it in the wasm_v4 branch, in the file babylon/asio/asio.h.

When using this new API for file loadings, I encountered quite a lot of issues with the lambda captures (need to capture the pointer by copy), which required some changes in engine.cpp and scene.cppwhich I will need to backport in your branch.

Thanks, it sounds very ambitious, implementing your own asyncio library. I will have a look at your asyncio code.

Cheers!

Cheers, Sam

samdauwe commented 4 years ago

Hello again,

related to the asynchronous file loading issue, BabylonJS has an asset manager which I did not port yet.

The idea of the asset manager is that you give it tasks such as "load a mesh file" or "load an image file". These tasks run in the background (asynchronously) and you can register callbacks to start rendering the scene once all files are loaded. For instance you can see here an example of loading a mesh: using the asset manager and without using the asset manager.

In our case, the asset manager could run in a different thread while the main loop checks the status of the asset manager and shows an info screen with the loading status, when the asset manager finishes with all tasks the main loop could start calling the scene render function.

Not sure if this can be of any help or if this asset manager could simplify things on your side.

Cheers, Sam

pthom commented 4 years ago

Hmmm, so by using OpenGL ES we are getting issues on Linux VM and MacOS that we were not getting before. Yes, for the moment, I cannot work with your branch.

Related to the build issue topic: I noticed that the builds fail when using the latest CMake 3.16 version. We are using functionality that has been deprecated, I can forward you the error if you are interested. For now I downgraded my CMake version to v3.11.

I am using cmake 3.15 and I did not encounter an issue. As a matter of fact, the cmake files are quite verbose, with sometimes duplicated code, and often non conformant to modern cmake (where you are supposed to use transitive dependencies).

In the wasm_experiments branch, I pushed some simplification to the cmakelists files such as this : https://github.com/samdauwe/BabylonCpp/commit/051e83f14531f09ec78ad2c28291ea7b0f3dcc89

https://github.com/samdauwe/BabylonCpp/commit/864d80f81ee523f05047f9fe030b30b8a889f03b

https://github.com/samdauwe/BabylonCpp/commit/864d80f81ee523f05047f9fe030b30b8a889f03b

An alternative is using bgfx which supports WebGL 2.0 and emscripten. It has different rendering backends such as Metal and is supported on many platforms. I am willing to switch to bgfx (or any similar library) if that would resolve many of the rendering issues we have on different platforms.

Why not. The important thing is that we can merge the two branches soon enough, and for this I will need to somehow be able to run the new OpenGl ES mode (or as last resort go back to the old mode)

Thanks, it sounds very ambitious, implementing your own asyncio library. I will have a look at your asyncio code.

I made the API generic enough to be able to revert it to libuv, boot.asio or whatever in the future. I tested it during execution, added unit tests, and it seems to works as of now, but I make no promises that it is bullet proof :-)

Some update concerning the progress in the wasm_experiment branch.

Most of the files and images are loaded asynchornously, and it seems to work, with a few regressions although. It was quite an effort, as I had to change a lot of things to make this work.

See the PR here: https://github.com/samdauwe/BabylonCpp/pull/65/files

Here is a summary:

Status for gltfLoader.

gltfLoader is quite hard to adapt for the new async API.

I prepared the transition to async, by changing al the functions that looked like this 

ArrayBufferView& loadBufferViewAsync

to this

std::shared_ptr<ArrayBufferView>& loadBufferViewAsync

the main issue is that GLTFLoader::loadUriAsync should return a promise not a value, as it is done in the js code.

But there are some other issues which I saw: I highlighted them in this commit: https://github.com/samdauwe/BabylonCpp/pull/65/commits/7bf29eb1accbd4c4a200626a5e69af79f7da2821

pthom commented 4 years ago

related to the asynchronous file loading issue, BabylonJS has an asset manager which I did not port yet.

I do not think this would help in my current issues. However, the asio api is quite relevant in order to port the assets manager, as it can offer the callbacks and ensure that they are called later, but synchronously.

pthom commented 4 years ago

A small update concerning macos + angle. Angle helps a little under macOS. Without Angle, dlopen(libEGL) failed directly. With Angle it works, but there is a subsquent failure in a call to EGL_CreateWindowSurface.

I suspect the issue is the same as this one: https://github.com/glfw/glfw/issues/1169 (and the fix mentioned there is not enough..)

I installed and linked with Angle this way:

  1. Install

    # install vcpkg , then
vcpkg install angle

  2. Hack extenal/glfw/CMakeLists.txt by adding:

    find_package(unofficial-angle CONFIG REQUIRED)
target_link_libraries(glfw PUBLIC unofficial::angle::libEGL unofficial::angle::libGLESv2)

  3. Add vcpkg cmake toolchain; Run cmake with -DCMAKE_TOOLCHAIN_FILE=..../vcpkg/scripts/buildsystems/vcpkg.cmake

pthom commented 4 years ago

Hello,

I reverted all my changes to gtlf, and added this hack which forces gltf to load synchronously.

Gltf now works again in desktop mode; but I doubt an easy solution for async loading & gltf can be found. Anyhow, this port to wasm proves to be hard, may be too hard.

Let's not lose hope,

I wish you a merry christmas!

samdauwe commented 4 years ago

Hello,

I reverted all my changes to gtlf, and added this hack which forces gltf to load synchronously.

Gltf now works again in desktop mode; but I doubt an easy solution for async loading & gltf can be found. Anyhow, this port to wasm proves to be hard, may be too hard.

Let's not lose hope,

I wish you a merry christmas!

Hello Pascal,

Thanks for the fixes! I will reply later on the changes you made, when I have more time

Merry Christmas to you too :)

pthom commented 4 years ago

I pushed a PR which is the first wasm version that can be used on a remote server.

The initial downloads sizes are as follows:

image

And here is a first christmas gift 🎁 (even if there are lots of broken samples)

https://traineq.org/BabylonStudio/BabylonStudio.html

image

Cheers!

samdauwe commented 4 years ago

I pushed a PR which is the first wasm version that can be used on a remote server.

The initial downloads sizes are as follows:

image 
* `emscripten_static_assets` contains the fonts + the samples source code. It is embedded inside `BabylonStudio.data`, which is downloaded at the app start

* `emscripten_http_assets`contains all the other assets (scenes, textures, screenshots, etc): they are downloaded asynchronously on demand when using the app

And here is a first christmas gift gift (even if there are lots of broken samples)

https://traineq.org/BabylonStudio/BabylonStudio.html

image

Cheers!

Wow great work, this looks really nice! Thanks for all the effort you have put into getting the wasm version working!!

Based on your previous replies I understood the following remaining issues:

I will try the OpenGL ES 3.0 version on Windows to check the status on that platform.

Cheers, Sam

pthom commented 4 years ago

Hi Sam,

I hope everything is ok for you. You do not need to answer quickly to this long message, in this christmas period :-) I am going to hit the road tomorrow, and I will be off for a while for christmas and new year's eve.

Based on your previous replies I understood the following remaining issues:

  • OpenGL ES 3.0 gives issues on Windows and OSX (and Linux). Angle helps a little under macOS. Without Angle, dlopen(libEGL) failes directly. With Angle it works, but there is a subsquent failure in a call to EGL_CreateWindowSurface

That is right. Some more details:

  • Not all examples are working yet in wasm.

There are three main kinds of failures:

  1. 3D shows for one fraction of a second, then disappears: may be these will be solved by merging with your branch.
  2. Segfault / exception / process hung
  3. Incomplete 3D scene
  • gltfLoader is quite hard to adapt for the new async API

That is right.

I would like to discuss with you the idea of switching to promises in the c++ code: I made an incomplete test, in order to see whether the code would compile using promise-cpp.

You can see here a commit were I made some modifications in order to experiment with this promise api. I reverted this commit soon after, since this was just an experiment, but it seemed to be a an interesting approach.

Some hightlights of this approach:

Step 1: define a wrapper around promise-cpp in order to be even closer to the js code:

For example, see below (which was a quick and dirty test):

#include "promise.hpp"

// promise-cpp does not enable to denote the output type, but we can simulate it 
// by using a wrapper like "MyPromise" below 
template<typename T> using MyPromise = promise::Defer;
using MyPromise_Any = promise::Defer;

// The js code often returns a bare value in order to fulfill a promise
// In C++, instead of 
//     "return value;" 
//  we would write 
//     "return  MyPromise_AlreadyFullfilled(value)"
template<typename T> auto MyPromise_AlreadyFullfilled(T t) {
  auto r = promise::newPromise();
  r->resolve(t);
  return r;
}

Step 2: review the structures inside gltf and add optional / promise whenever needed, in order to match the js code:

For example:

struct IBufferView : public IGLTF2::IBufferView, IArrayItem {
  ArrayBufferView _data;
  BufferPtr _babylonBuffer = nullptr;
}

would become this, which is closer to the js version:

struct IBufferView : public IGLTF2::IBufferView, IArrayItem {
  std::optional<MyPromise<ArrayBufferView>> _data;
  std::optional<MyPromise<BufferPtr>> _babylonBuffer;
}

Step 3: use promises in the code:

For example:

ArrayBufferView& GLTFLoader::loadBufferViewAsync(const std::string& context,
                                                 IBufferView& bufferView)
{
  if (bufferView._data) {
    return bufferView._data;
  }

  auto& buffer    = ArrayItem::Get(String::printf("%s/buffer", context.c_str()), _gltf->buffers,
                                bufferView.buffer);
  const auto data = _loadBufferAsync(String::printf("/buffers/%ld", buffer.index), buffer);

  // ASYNC_FIXME: We cannot treat the data right now, it will be otained later!
  try {
    bufferView._data = stl_util::to_array<uint8_t>(
      data.uint8Array, data.byteOffset + (bufferView.byteOffset.value_or(0)),
      bufferView.byteLength);
  }
  catch (const std::exception& e) {
    throw std::runtime_error(String::printf("%s: %s", context.c_str(), e.what()));
  }

  return bufferView._data;
}

would become:

MyPromise<ArrayBufferView> GLTFLoader::loadBufferViewAsync(const std::string& context,
                                                 IBufferView& bufferView)
{
  if (bufferView._data) {  // bufferView._data is of type optional<MyPromise<ArrayBufferView>>
    return bufferView._data.value();
  }

  auto& buffer    = ArrayItem::Get(String::printf("%s/buffer", context.c_str()), _gltf->buffers,
                                bufferView.buffer);
  bufferView._data = _loadBufferAsync(String::printf("/buffers/%ld", buffer.index), buffer);

  return bufferView._data.value(); // 
}

What do you think?

A connex issue is that ArrayBufferView, which should be a kind of union<UInt8Array, Float32Array, Uint16Array, ...> is instead implemented as 7 copies of the same data.

A month ago I made a test in order to improve this, you can see the idea in this commit It was working correctly without regression thoughout the code, except inside the gltf code. This commit is located on a separate branch ( wasm_optim ).

samdauwe commented 4 years ago

Hello Pascal,

I hope everything is ok for you. You do not need to answer quickly to this long message, in this christmas period :-) I am going to hit the road tomorrow, and I will be off for a while for christmas and new year's eve.

Enjoy the holiday period! I already wish you a happy new year's eve :)

Based on your previous replies I understood the following remaining issues:

  • OpenGL ES 3.0 gives issues on Windows and OSX (and Linux). Angle helps a little under macOS. Without Angle, dlopen(libEGL) failes directly. With Angle it works, but there is a subsquent failure in a call to EGL_CreateWindowSurface

That is right. Some more details:

* Under macos, the failing call to EGL_CreateWindowSurface is inside glfw. We could hope that the trio "Angle + OpenGL ES 3.0 + SDL" might work, once the branch wasm_experiment was merged (since it brings SDL support).

* Under linux:  on my linux VM [the webGL2 test page](https://get.webgl.org/webgl2/) says it does not support it. Howver it was running at 600fps with desktop openGL. May be angle could help here too, but it's a wild guess

* Under windows: not tested

I merged the wam_experiment branch into the master branch and deleted the wam_experiment branch. I made a couple of changes while merging:

I am planning to drop GLFW support so we don't have to maintain both SDL and GLFW.

  • Not all examples are working yet in wasm.

There are three main kinds of failures:

1. 3D shows for one fraction of a second, then disappears: may be these will be solved by merging with your branch.

2. Segfault / exception / process hung

3. Incomplete 3D scene

On Windows I noticed there was occasionally a crash when loading assets from file. They are not always reproducible, so I assume it is a concurrency issue. I will do some further testing.

  • gltfLoader is quite hard to adapt for the new async API

That is right.

I would like to discuss with you the idea of switching to promises in the c++ code: I made an incomplete test, in order to see whether the code would compile using promise-cpp.

You can see here a commit were I made some modifications in order to experiment with this promise api. I reverted this commit soon after, since this was just an experiment, but it seemed to be a an interesting approach.

Some hightlights of this approach:

Step 1: define a wrapper around promise-cpp in order to be even closer to the js code:

For example, see below (which was a quick and dirty test):

#include "promise.hpp"

// promise-cpp does not enable to denote the output type, but we can simulate it 
// by using a wrapper like "MyPromise" below 
template<typename T> using MyPromise = promise::Defer;
using MyPromise_Any = promise::Defer;

// The js code often returns a bare value in order to fulfill a promise
// In C++, instead of 
//     "return value;" 
//  we would write 
//     "return  MyPromise_AlreadyFullfilled(value)"
template<typename T> auto MyPromise_AlreadyFullfilled(T t) {
  auto r = promise::newPromise();
  r->resolve(t);
  return r;
}

Step 2: review the structures inside gltf and add optional / promise whenever needed, in order to match the js code:

For example:

struct IBufferView : public IGLTF2::IBufferView, IArrayItem {
  ArrayBufferView _data;
  BufferPtr _babylonBuffer = nullptr;
}

would become this, which is closer to the js version:

struct IBufferView : public IGLTF2::IBufferView, IArrayItem {
  std::optional<MyPromise<ArrayBufferView>> _data;
  std::optional<MyPromise<BufferPtr>> _babylonBuffer;
}

Step 3: use promises in the code:

For example:

ArrayBufferView& GLTFLoader::loadBufferViewAsync(const std::string& context,
                                                 IBufferView& bufferView)
{
  if (bufferView._data) {
    return bufferView._data;
  }

  auto& buffer    = ArrayItem::Get(String::printf("%s/buffer", context.c_str()), _gltf->buffers,
                                bufferView.buffer);
  const auto data = _loadBufferAsync(String::printf("/buffers/%ld", buffer.index), buffer);

  // ASYNC_FIXME: We cannot treat the data right now, it will be otained later!
  try {
    bufferView._data = stl_util::to_array<uint8_t>(
      data.uint8Array, data.byteOffset + (bufferView.byteOffset.value_or(0)),
      bufferView.byteLength);
  }
  catch (const std::exception& e) {
    throw std::runtime_error(String::printf("%s: %s", context.c_str(), e.what()));
  }

  return bufferView._data;
}

would become:

MyPromise<ArrayBufferView> GLTFLoader::loadBufferViewAsync(const std::string& context,
                                                 IBufferView& bufferView)
{
  if (bufferView._data) {  // bufferView._data is of type optional<MyPromise<ArrayBufferView>>
    return bufferView._data.value();
  }

  auto& buffer    = ArrayItem::Get(String::printf("%s/buffer", context.c_str()), _gltf->buffers,
                                bufferView.buffer);
  bufferView._data = _loadBufferAsync(String::printf("/buffers/%ld", buffer.index), buffer);

  return bufferView._data.value(); // 
}

What do you think?

Wow, yes that is fine for me. The current functions have an async postfix but are all synchronous because that was easiest for implementing them. Your plan to make them asynchronous sounds fine to me!

A connex issue is that ArrayBufferView, which should be a kind of union<UInt8Array, Float32Array, Uint16Array, ...> is instead implemented as 7 copies of the same data.

Yes, the ArrayBufferview class is an union. Having 7 copies was fine for testing but will give issues in the wasm version.

A month ago I made a test in order to improve this, you can see the idea in this commit It was working correctly without regression thoughout the code, except inside the gltf code. This commit is located on a separate branch ( wasm_optim ).

Nice solution! Do you know why it was failing in the gltf code? If needed I can write unit tests that can help to debug the issue.

Cheers, Sam

pthom commented 4 years ago

I merged the wam_experiment branch into the master branch and deleted the wam_experiment branch. I made a couple of changes while merging:

Nice! You may encounter (perhaps) some new issues because the loading is now asynchronous. If you encounter an issue in a callback, most of the time you can solve it by looking at the call stack, and look at lambda capture by references which might need to be replaced by captures by copy.

I removed code for which the unit tests were failing and the functionality was not used in the code: SIMD support, core/future_then, compile time calculation unit test file. There was a lab folder that I also removed, you have it still in on your branch for testing so you can put it back if needed.

That is fine;

I am planning to drop GLFW support so we don't have to maintain both SDL and GLFW.

I am not sure it is required, as the code for glfw / sdl is now neatly contained inside runner_glfwand runner_sdl and nowhere else. This could lead to sample projects for users who may want to use BabylonCpp in their own project. What do you think?

On Windows I noticed there was occasionally a crash when loading assets from file. They are not always reproducible, so I assume it is a concurrency issue. I will do some further testing.

There is a way to hunt possible memory fault, unknown behaviors, etc. which I would like to suggest: compile the project with clang's memory sanitizer and ub sanitizer, like this: 

cmake .. -DCMAKE_SHARED_LINKER_FLAGS=-fsanitize=address -DCMAKE_CXX_FLAGS=-fsanitize=address

may be we could group them but I did not try

cmake .. -DCMAKE_SHARED_LINKER_FLAGS=-fsanitize=undefined -DCMAKE_CXX_FLAGS=-fsanitize=undefined

A month ago I made a test in order to improve this, you can see the idea in this commit It was working correctly without regression thoughout the code, except inside the gltf code. This commit is located on a separate branch ( wasm_optim ). Nice solution! Do you know why it was failing in the gltf code? If needed I can write unit tests that can help to debug the issue.

It was not failing, I simply gave up adapting the code inside gltf, because the gltf code was to different from the js version, and I felt it was more urgent to first re-adapt the gltf code. If you are interested, you could perhaps try to merge the wasm-optim branch in a separate checkout on your side. There are 3 commits:

samdauwe commented 4 years ago

Hmm, it looks that I did not reply yet on your last comments...

I merged the wam_experiment branch into the master branch and deleted the wam_experiment branch. I made a couple of changes while merging:

Nice! You may encounter (perhaps) some new issues because the loading is now asynchronous. If you encounter an issue in a callback, most of the time you can solve it by looking at the call stack, and look at lambda capture by references which might need to be replaced by captures by copy.

I will have to do some more testing to find out the status about the asynchronous loading. I will get back to you in this one.

I removed code for which the unit tests were failing and the functionality was not used in the code: SIMD support, core/future_then, compile time calculation unit test file. There was a lab folder that I also removed, you have it still in on your branch for testing so you can put it back if needed.

That is fine;

I am planning to drop GLFW support so we don't have to maintain both SDL and GLFW.

I am not sure it is required, as the code for glfw / sdl is now neatly contained inside runner_glfwand runner_sdl and nowhere else. This could lead to sample projects for users who may want to use BabylonCpp in their own project. What do you think?

Okay, sounds fine to me, we can keep them both :)

On Windows I noticed there was occasionally a crash when loading assets from file. They are not always reproducible, so I assume it is a concurrency issue. I will do some further testing.

There is a way to hunt possible memory fault, unknown behaviors, etc. which I would like to suggest: compile the project with clang's memory sanitizer and ub sanitizer, like this:

cmake .. -DCMAKE_SHARED_LINKER_FLAGS=-fsanitize=address -DCMAKE_CXX_FLAGS=-fsanitize=address

may be we could group them but I did not try

cmake .. -DCMAKE_SHARED_LINKER_FLAGS=-fsanitize=undefined -DCMAKE_CXX_FLAGS=-fsanitize=undefined

Thanks for this information, I will try out those flags when debugging the examples.

A month ago I made a test in order to improve this, you can see the idea in this commit It was working correctly without regression thoughout the code, except inside the gltf code. This commit is located on a separate branch ( wasm_optim ). Nice solution! Do you know why it was failing in the gltf code? If needed I can write unit tests that can help to debug the issue.

It was not failing, I simply gave up adapting the code inside gltf, because the gltf code was to different from the js version, and I felt it was more urgent to first re-adapt the gltf code. If you are interested, you could perhaps try to merge the wasm-optim branch in a separate checkout on your side. There are 3 commits:

* The fist add "spans", which can avoid memory copies

* the seconds add a new ArrayBufferView,

* the third disables gltf, by replacing it with a dummy "throw only" version. You may want to discarfd this commit.

Ah, making the gltf code asynchronous like the JS version seems to require some effort. It is synchronous at this moment because it is easier to debug. Addressing the glTF loading issues we can do if we fixed the more basic examples first I assume.

I will have a look at those ArrayBufferView improvements you implemented and merge them.

Thanks! Sam