Multiple OBJ models rendering

[EtagiBI]

Hello,

First of all, I would like to thank you for all your efforts! Your project seems to be the only one alive project dedicated to Three.js photorealistc rendering.

As for my question, is it possible to render multiple OBJ models with your PTR? My friend and I are working on a Three.js based room planner as our university project. We have a bunch of textured furniture models with different material determined via corresponding MTL files. Is this feature supprted by PTR? As I can see from your demos, at the moment PTR works with simple shapes only.

[erichlof]

Hello @EtagiBI

Thank you for posting your question here - I'm sure there are others out there who are wondering the same about the ability to load multiple OBJ files, and possibly MTL files as well.

The short answer is unfortunately, not at the moment. I currently have a demo that loads a single OBJ file. So my path tracer has the raw ability to load a list of vertices/faces in OBJ file format, and path trace the mesh relatively quickly. However that Crane model in the demo only has 36 triangles. Even that amount lowers the frame rate on my laptop from 60 to 40 fps. Things get worse if you, for instance, go into the HTML source code and comment out the Crane model and choose another listed model with a higher poly count. So if you and your friend are trying to load rooms full of thousands or even hundreds of triangles, the frame rate will probably slow to a crawl, like 1 or 2 fps, or even lose the WebGL context. Users would not be able to smoothly look around your room with their mouse, and would probably get frustrated.

Now for the long answer, if you're interested ;-)

When I started this project I was naive in thinking that if I could make geometric mathematical shapes render at 60 fps, then I'll just eventually throw a list of triangles at it and it would be the same. But sadly, checking rays against every triangle in a detailed mesh really adds up. And that's just for the initial raycast - you have to multiply that cost by 4 if you want to bounce those same rays around the room to get global illumination. So I looked into acceleration structures like grids, KD trees, and the BVH - the latter seemed to be the preferred choice for ray tracing and path tracing, so I ported a BVH builder in C++ over to Javascript. This seems to help a lot, BVH attempt , but the problem is that WebGL 1.0 does not support dynamic indexing of arrays during a loop, such as correctBranch[x]. In WebGL 1.0, that 'x' must be a constant like '2' or a predefined value. That means that I can't prune large parts of the tree, which is the whole point of having an acceleration structure in the first place. When WebGL 2.0 is supported inside three.js (hopefully soon), then I can revisit the BVH with dynamic indexing allowed, and even look into the LBVH which uses Morton codes and bit manipulations (allowed only by WebGL 2.0), which can rebuild the entire structure on the GPU for a scene of moving, dynamic triangles - every frame.

I would love to be able to list a bunch of meshes like you and your friend want to do in the room scenes, and have it all just work. But getting it to run smoothly is the most difficult aspect of ray/path tracing I'm finding out. Traversing the BVH is not too hard to understand, and it can be done in a tight loop inside a GPU shader in 30 lines or so. What is difficult to grasp and implement on the GPU, is the BVH builder, which must quickly load thousands of triangles into texture memory, create bounding boxes and indices for all of those triangles, and then order them in some efficient fashion to be examined and pruned by the tracer. Unfortunately, GPU BVH builders and source code (even in CUDA, let alone WebGL) are poorly documented and explained. It's no coincidence that a lot of cutting-edge research has been done in this area, and if good results are found, they are assimilated into existing renderers that might be behind patents and not available for public viewing of the GPU source code. However, maybe this will change with more people out there like me trying this stuff out on their own, and posting it online for all to learn from.

It's frustrating because I know interactively rendering large amounts of triangles is possible, even inside seemingly low WebGL. For instance I found this cool piece of software, which is similar to my project, but has much more capabilities in terms of rendering large meshes: Antimatter . Scroll down and hit the 'Launch Prototype' button. Then you can choose from a list of large meshes. If they can do it in the browser with a BVH, it has to be possible. I would have said you guys should go with that for your project, but it looks like it is going to be for purchase only, and not open source - plus I don't know how deep into needing three.js in your project you already are.

I wish I could be of more help to you both, I hope you can find a different approach or a workaround in the meantime. If you would like to ask anything else or need clarification or advice, please don't hesitate.

-Erich

[EtagiBI]

Wow, thanks for a thorough reply!

I forgot to mention that we want to render our scenes server side, not in real-time. The whole plan looked like this: 1) We create a scene in our "home-made" three.js 3D planner 2) We press some "Render" button that throws our ticket in a rendering queue on server 3) We patiently wait... 4) We receive a rendered photorealistic image!

Is it possible? I clearly understand that real-time rendering of multiple textured models is really heavy for resources.

[erichlof]

Hello @EtagiBI Ah, so offline rendering while you wait is ok. Mmm, I hadn't imagined this kind of use-case yet for a browser path tracer. I actually need to get some sleep, it's late here where I am. But let me think about it for a little bit and if I can't do a demo myself with the tools I have in this repo, I may be able to point you in the right direction using a javascript path tracer that can handle any amount of mesh data, but maybe without fancy or crafty GPU shaders to make it go real time, since you don't mind it being an offline rendering. I'll get back to you soon, thanks!

[erichlof]

Hi again @EtagiBI

After thinking about it for a while, unfortunately I don't think the tools that I have developed here can help you at the moment. What you are needing is a robust loader, simple editor, and an offline beauty renderer. What I have here currently is a non-robust, simple shape renderer that goes as fast as possible so you can move and look around in real time, but still have correct global illumination effects. I started the project with the latter in mind and geared every line of code towards that purpose. I would have to essentially start over with what you have in mind to really do it right, instead of trying to hack something robust (loading .obj files, textures, etc.) into the blazing fast GPU path tracer that was made for different purposes.

But I really like the idea that you have, which is, if I understand correctly, to use three.js and the browser as a visualizing tool to load .obj files into a simple editor where you can click and move furniture around a virtual room using the three.js default WebGL renderer at 60fps. Then when you are happy with how the room looks, you hit 'Render' and it dumps all of that data (a big list of triangles, vertex normals, texture uv's) into the path tracer, which ray casts against those triangles, finds the closest intersection, looks up the normal and uv data for that correct triangle, and colors that pixel. When all pixels have been calculated, then it saves the final render as a .png or something. That part would take several minutes without acceleration structures, but you mentioned that you don't mind waiting offline for it to finish the calculations. I might go off and attempt a side project of my own that loads a whole scene of .obj files and then renders offline because this sounds like a neat project idea, but mine would be minus the editor part, that is a whole other tool set that would take weeks to develop - editor .

It just occurred to me that someone has already done what you are wanting: Ben Houston and his Clara.io project. Here's the link: Clara.io Ben is a nice fellow who has contributed a lot to the three.js code base. He wants his browser based editor/renderer/modelling software to be able to compete with 3dsMax, Maya, Blender, etc. I hope he and his team succeed because having the ability to be online and collaborating real time while using a sophisticated 3D modeling package is a great idea. In a nutshell, his software loads meshes of any file type (.obj, .fbx, etc), uses three.js renderer in real time to drag, reshape, resize the scene and meshes, then you hit fast preview and the sophisticated V-Ray rendering farm jumps into action. If you like the preview, you hit full render, wait, then hit save image and you're done. It's free to try out, you might give it a go - if only to get some ideas for your project.

Sorry I couldn't be of more help with my tools in this repo, but hopefully you have an idea of what needs to happen in your software, and hopefully I at least pointed you in the right direction. If you like, you can post any future findings or breakthroughs, or links to your project here so we can all benefit.

-Erich

Edit: just found a perfect resource that does what you are wanting and is open source: XRay Hope this helps!

[EtagiBI]

Hello @erichlof!

I'm sincerely impressed by amount of useful information given by you. Thank you very much!

I'll try to keep this topic up-to date.

[erichlof]

Hi @EtagiBI ,

Glad to be of help, sorry I couldn't have my existing project just work out of the box for what you wanted. However, you have inspired me to start planning ahead and working towards being able to call the THREE.OBJLoader on various meshes, and then they get their data get saved to a texture to be consumed by the GPU and path traced. Earlier today I just figured out how to let the THREE.OBJLoader do all the .obj file parsing work (I task I kind of understand but not completely), and then 'hijack' the newly created THREE.Mesh and open it up and read and save its geometry data to a texture. Next is to try to do the same with materials data from a .mtl file. I'll post a short demo here soon!

Thanks :)

[erichlof]

Hi @EtagiBI Just wanted to give you an update of my progress with this issue. I didn't want you to think I had forgotten about you and this feature request! :) I have figured out how to use three.js's THREE.OBJLoader (I was using a less-robust custom one that I had ported from C++ before) and I am now able to hook into the resulting mesh and save its entire triangle data to a THREE.DataTexture, which can be saved and quickly loaded onto the GPU for use with the path tracing engine. Demo and its Source Code

I understand the .obj and .mtl file formats now (they were well designed which is why they have lasted so long I guess! ), but I'm trying to decide how to retrieve the material data and then make physical ray tracing materials out of them. You can see from the demo, I just assigned a matte bright purple color to all the triangles of the crane origami .obj model. But suppose there was an accompanying .mtl file that said they wanted the crane's neck to be transparent blue, the wings to be shiny silver, and the body to be matte gray or something? For the path tracing engine to use that data, it needs to have its physical reflectance properties as well as a color and shininess or color and transparency. So the matte would be the easiest, I could just assign that to be Lambertian diffuse in my engine, the shiny one would have to have a metalness flag saying if it is a metallic specular object, or just a shiny piece of plastic. The transparent one would need an Index of Refraction (IoR) saying how much the rays should bend when they enter a transparent surface like glass, clear coat plastic, or water.

So I am kind of going back and forth as to how I should implement the materials loading part of all this. I'll give you more updates as I work towards a solution. If I can eventually get the BVH working with WebGL 2.0 supported in three.js, then all objects will load and render in real time!

[EtagiBI]

Excellent news! As far as I understand, all materials have these specific characteristics (light reflection, light absorbtion, etc.) within MTL file. We've managed to add dynamic lighting in our editor, so models with different material characteristics look different now.

Since three.js has obtained shader support, we're trying to implement native shadows for models.

[FishOrBear]

@EtagiBI Hi, How is your project going?

I also want to do similar projects. Any suggestions?

I understand a lot of open source renderers, and I'm ready to try Blender Cycles now.

[EtagiBI]

@FishOrBear Yes, we decided on Blender Cycles as well. Render quality is great, but it took a while to adjust parameters.

[FishOrBear]

@EtagiBI Did you start using it?

I understand that some of the Cycles documentation is missing.

I found that the mitsuba document is sound. I am now considering what rendering engine to use.

[MEBoo]

Hi @erichlof any news about?

I'm very interested in your project... but to do online client background rendering (no realtime) of Three.js scene created online loading OBJs and applying custom materials dinamically (no MTLs involved).

It would be nice having more OBJs and three.js support, before Febrary 2019...

Thankyou

[erichlof]

Hi @MEBlabs I'm sorry but I'm kind of at a stand-still with OBJ rendering until the folks at three.js implement WebGL 2.0 support. In order to render larger scenes or multiple OBJ files, I need to physically load the triangle data onto the GPU, and the only way to do that currently is through a data texture. I have successfully done it with one small OBJ file, see the simple OBJ demo - but as soon as I try loading multiple files or 1 large OBJ file (the Utah teapot with 1000 triangles, for instance) the renderer slows to a crawl and crashes or just fails to compile.

Therefore I needed some kind of data acceleration structure for randomly sorting through all those triangles, and the only way to do that is to have random access into large arrays, which is not supported by WebGL 1.0 (which is still only supported by three.js). WebGL 2.0 brings with it the possibility to look up data randomly inside an array through the GPU fragment shader, which I absolutely must have in order to continue with that part of the project.

If wait-time for rendering is not a concern, you could go with a CPU renderer, which is not accelerated of course, but definitely has the capability to sort through huge amounts of OBJ triangle data pretty quickly. It would just render a static image though, which is not the focus and direction of my project.

The only other alternative I can think of is that you create something with three.js, use their converter to convert the entire scene to a readable file by a production renderer, like Octane for Cinema4D, Cycles for Blender, V-Ray for Clara.io, etc. and hit the render button inside their software. It should be the best of both worlds, able to be accelerated somewhat with their proprietary acceleration structure, and able to handle huge amount of scene data through streaming.

Sorry I can't be of more help. Best of luck to you with your project. Please let us know if you find a temporary solution with other software, as I have had some similar questions and requests for larger amounts of data rendering.

[mrboggieman]

Check it out: http://www.zentient.com/

Loading obj in shader via texture (bvh)

On 4 Sep 2018, at 17:05, Erich Loftis notifications@github.com wrote:

Hi @MEBlabs I'm sorry but I'm kind of at a stand-still with OBJ rendering until the folks at three.js implement WebGL 2.0 support. In order to render larger scenes or multiple OBJ files, I need to physically load the triangle data onto the GPU, and the only way to do that currently is through a data texture. I have successfully done it with one small OBJ file, see the simple OBJ demo - but as soon as I try loading multiple files or 1 large OBJ file (the Utah teapot with 1000 triangles, for instance) the renderer slows to a crawl and crashes or just fails to compile.

Therefore I needed some kind of data acceleration structure for randomly sorting through all those triangles, and the only way to do that is to have random access into large arrays, which is not supported by WebGL 1.0 (which is still only supported by three.js). WebGL 2.0 brings with it the possibility to look up data randomly inside an array through the GPU fragment shader, which I absolutely must have in order to continue with that part of the project.

If wait-time for rendering is not a concern, you could go with a CPU renderer, which is not accelerated of course, but definitely has the capability to sort through huge amounts of OBJ triangle data pretty quickly. It would just render a static image though, which is not the focus and direction of my project.

The only other alternative I can think of is that you create something with three.js, use their converter to convert the entire scene to a readable file by a production renderer, like Octane for Cinema4D, Cycles for Blender, V-Ray for Maya, etc. and hit the render button inside their software. It should be the best of both worlds, able to be accelerated somewhat with their proprietary acceleration structure, and able to handle huge amount of scene data through streaming.

Sorry I can't be of more help. Best of luck to you with your project. Please let us know if you find a temporary solution with other software, as I have had some similar questions and requests for larger amounts of data rendering.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[MEBoo]

Thanks @erichlof ... so let's wait for a three.js upgrade to WebGL 2.0

Exporting data to another offline renderer is not an option, since I need to render client side and online.

But thankyou for your explanation.

When you say that the scope of your project is not a static rendering, please consider that your work is the best I've seen here, speaking about pathtracing etc., the tech you built can be used in a real context, so I would consider to simply generate a client-side hi-res and quality (but slow) render, other then the super fast real-time rendering that is awesome but more for science.

So I hope you'll consider my suggestions when the three.js upgrade become ready...

See you next months ;)

[MEBoo]

@mrboggieman thankyou! But I've already seen that project... the problem is that it'isnt opensource... I can't find a repository.

And the work done by @erichlof seems better, regarding the final render quality after many samples... and AntiMatter is not Three.js based :(

[erichlof]

@MEBlabs Thank you for the kind words! I will definitely keep the quality static render scenario in mind once I get the BVH going with WebGL 2.0 support from three.js, etc. Speaking of AntiMatter, yes unfortunately that renderer is not three.js based, nor is it open source from what I can tell. I have looked extensively through the author's GitHub repo's and his website but there is nowhere to view his source code for that particular project. However, one glimmer of hope: I was able to right-click and 'view page source' while running the AntiMatter browser app, and I was able to piece together what looked to be a working BVH using WebGL 1.0. Mind you it is in minified/compact mode, so all the functions are like one letter or 2 letters long, lol. So it can be done, I just have to try to piece it together. I've already made some headway with his GPU fragment shader source. It looks like he implemented some crafty workarounds to sidestep the random array indexing problem. It's ugly and wasteful but it works! ;-)

I'll let everyone know if I can gain any ground with WebGL 1.0 while I'm waiting for WebGL 2.0 support.
Thanks again, -Erich

[MEBoo]

@erichlof wow that's great! I've also checked his code and saw that it is minified... but I don't have many skills about :(

if I were you, I'd wait for webgl 2.0, but with WebGL 2.0 it would be awesome to also have a working PBR shader, simply configurable (like the one included in Three.js), and that supports at least a simple texturing method... Let me know if it is possible or a dream!

I can wait some months :)

Thankyou again!

[erichlof]

@MEBlabs Hi, yes it is possible to incorporate PBR materials - they would need to be set in the triangle data list and loaded onto the GPU as a data texture. Currently I only have positional data for each triangle vertex packed in a texture, so something like dataTexture[0,0].r = 270.01; dataTexture[0,0].g = -65.86; dataTexture[0,0].b = 483.2; And that defines where the triangle vertex is located in model space, XYZ = .r, .g, .b

To get PBR data on top of that, we would need to pack the data into a larger structure inside the texture, such as: dataTexture[0,0].r = 270.1; dataTexture[0,0].g = -65.86; dataTexture[0,0].b = 483.2; dataTexture[0,0].a =43.9; dataTexture[1,0].r = 124.7; dataTexture[1,0].g = 54.8; dataTexture[1,0].b = -843.1; dataTexture[1,0].a =675.0; .rgb is first vertex XYZ, .a .rg is second vertex XYZ, .b and .a is third vertex XY dataTexture[2,0].r = 453.8; dataTexture[2,0].g = 1.0; dataTexture[2,0].b = 0.0; dataTexture[2,0].a = 0.0; (the .r channel here is continuation of the previous vertex list vertex Z, because 9 / 8 has a remainder of 1 (annoying, right?), .g .b. a. defines the usual rgb diffuse color, so a red triangle.

dataTexture[3,0].r = 1.0; dataTexture[3,0].g = 1.0; dataTexture[3,0].b = 1.0; dataTexture[3,0].a = 0.5; still on the same triangle, now it's the PBR stuff like specularColor.r, .g, .b, and a 'roughness' value

dataTexture[4,0].r = 0.0; dataTexture[4,0].g = 1.0; dataTexture[4,0].b = 0.5; dataTexture[4,0].a = 0.25; still the same triangle, but finally we finish up with PBR stuff like 'metalness', Index of Refraction, clearCoat IoR, clearCoat roughness.

So it can be done, but as you can see there needs to be a lot more texture data slots (around 20) for each triangle. The final texture size would need to be the number of triangles in the model, times 5 rgba slots per triangle. So roughly calculating, if we had a 10,000 triangle model, that's 50,000 .rgba texture slots (200,000 total floating-point data entries packed in). Which seems like a lot, but most cell phones even can handle a 4096x4096 .rgba texture, so that's 16,777,216 available texture elements, each with their own .rgba channels (4 values for each texture element), so 67,108,864 possible triangle data entries packed into 1 texture.

edit : argh, I forgot uv texture coordinates need to be specified for each of the 3 vertices, so that's 6 more data numbers to pack in, maybe pad it to 8 numbers (2 .rgba texture elements) to be memory look-up effecient, and use the 2 remaining slots for Texture ID (which texture to use), extra texture info, etc.

So I kind of know how I would approach it, but getting materials to change on the fly inside some kind of material editor would take a lot more work and know-how. That's why they have teams of 10, 20 or more working on OTOY Octane, Blender Cycles, etc. I don't know if I could do all that alone, it isn't really part of the scope of this project. But it definitely is possible and has already been implemented with more sophisticated renderers. Hope that helped! :)

[MEBoo]

@erichlof wow seems a little complicated... I've not studied this yet... so the problem is how to send data to the GPU to elaborate it with a Fragment Shader? With WebGL2 there is no other new fancy way?

But when your base tech will be ready, I can work on like a monkey and you could tell me what to write :D (and I may pay other monkeys to help me)

[erichlof]

@MEBlabs Yes it is pretty complex unfortunately. The easiest part was getting the basic path tracer to work. There are many tutorials out there such as the ones on ScratchAPixel and ShaderToy, and of course Kevin Beason's smallpt in C++, which is where I started. In a couple hundred lines of code you can have a decent looking path tracer. When you are rendering simple shapes like spheres, it's great, but as soon as you start trying to render large triangular models, everything slows to a halt or crashes. Then you realize an acceleration structure is needed to speed up rendering and load all those triangles. I have yet to find a full tutorial for streaming acceleration structure data (like a BVH - bounding volume hierarchy) to the GPU. There aren't even decent tutorials on how to create the data structure in the first place, let alone sending it to a fragment shader as a texture. I was lucky enough to find a C++ CPU BVH example here on GitHub and I ported it successfully (this is what is used on my current model rendering demos). But there is almost 0 information on how to do that on the GPU, and in WebGL (1.0 or 2.0). Then there's the problem we were discussing about how to store all that material data for each triangle made in a 3d modelling program. Again, no tutorials on that as a GPU texture out there on the internet.

I'm still looking at the AntiMatter source - I might post some of it here - I have gone in and renamed the minified variables and functions so instead of reading 'float m=m0(g, h);' it reads 'float result = calculateFresnel(vec3 intersectionPoint, int material_ID);' I'm still looking into it. :)

[MEBoo]

@erichlof thanks and good work... let me know if I have to do something in future ;)

[erichlof]

Hi @MEBlabs and all, As promised here is the antimatter glsl shader source in its original minified form: antimatter.min.glsl It was all smashed together on one long line of code, so I did a quick document format, adding spaces and line returns for some readability.

And after going through his code line by line, here is my decompressed version: antimatter.glsl I reverse-engineered and added clear, meaningful variable names and function names, for maximum readability.

There are a lot of similarities between his path tracing shader and mine here on my repo. I added a comment to the function StackElement_data getStackElement_byIndex(const in float ix) indicating this is how he did the workaround for the WebGL 1.0 limitation of no dynamic array indexing.

I am in the process of decompressing the index antimatter.js file that does all the WebGL initialization stuff and creates the BVH to data texture for GPU consumption. This part is even more complicated than the shader intersectBVH function. Creating the acceleration structure is always more difficult because you have to order all of the thousands (or millions) of triangles, vertex data, and material data, sort them, and compress them onto a texture in a meaningful way. But hopefully by studying his source I can make some headway. ;-)

[MEBoo]

@erichlof ahaha great work! only 500 lines of code to render the reality, awesome! Do you think that the workaround has the same performance as the webgl 2.0 will have?

Regarding BVH, you should already have your implementation done, right? Or yours is not for traingles, but for primitives only?

[erichlof]

@MEBlabs Yes it's amazing what a couple hundred lines of code can produce. This style of rendering was proposed all the way back in the late 1970's and early 1980's (ray tracing and then path tracing, respectively) but computers were way too slow to even think about trying it. Hence the branch in a different direction of rasterization of triangles to the screen, which was doable, but many complicated hacks had to be introduced to get close to what you could do with a small ray tracing program (even to this day, i.e. reflection probes, cascading shadow mapping, etc..). Then along came 3d acceleration cards which helped with throwing triangles at the screen fast, and the rest is history. However, with Nvidia's recent introduction of RTX ray tracing card technology (now that computers and GPU's are fast enough in 2018), they are trying to bring back ray tracing first (you can view a couple of their presentation demos on YouTube), and then eventually real time path tracing once they get their deep-learning A.I. assisted denoisers fast enough - I have no idea how all that A.I. denoising stuff works. But yes, at its core, ray tracing is an elegant, short, simple way of rendering photo-realistic images - a window into reality!

Regarding the BVH, I have a basic triangle BVH builder that works. Typically you can survive without a BVH if you are just intersecting mathematical shapes likes spheres, cylinders, cones, boxes, etc., that's why you see everyone's beginning ray tracers rendering only those type of shapes. When you get into loading models though, it requires many triangles to be searched and intersected (because in the graphics world, most 3d models are represented as triangles), so that's when you need to introduce a BVH. I ported a C++ BVH that I found on GitHub (I gave credit to the author in the comments) to JavaScript. So it produces a list of bounding boxes and a list of raw triangles (vertex position data) in 2 different textures to be loaded on the GPU. The roadblock I was hitting was not being able to maintain a working bounding box stack on the GPU (a WebGL 1.0 limitation) so that I could search through the stack with dynamic array indexing and intersect it.

I don't know if there will be any performance difference between employing the Antimatter webgl 1.0 workaround, or just using array indexing in webgl 2.0. That remains to be seen. I'll keep you updated - I'm about to try his hack and stick it into my code and just see if it works at all. :)

[MEBoo]

@erichlof yes I know the history of 3d graphics, I found you just after the nVidia keynote... searching "webgl raytracing"... I thought that if nVidia starts to deliver RT dedicated hardware, some mad boy could have a solution to implement it in Three.js :D ... and let's see when webgl will support the new hardware ;) But looking to your code, is still awesome that 4/5 hundreds of code can display the reality. wow!

Ok I'll wait for good updates.. thank you!

[MEBoo]

Hey you did it man!! 🎉🎉🎉 Awesome!

[erichlof]

Hi @MEBlabs and all ,

Yay, initial success! (well, for the most part anyway). Wouldn't you know it, the very day I get the WebGL 1.0 BVH shader code working is the day that three.js releases version r97, which has initial support for WebGL 2.0! Haha

So now I'm at a fork in the road. The WebGL 1.0 version works great for lower than 800-triangle models, but as soon as I tried heftier models of just a little over that, say 1000 triangles, it either will not compile, or I can see the rendered model for a brief moment (in all its glory) before the frame rate drops to 0, the browser tab crashes, and the WebGL context is lost, so you have to close and reopen the tab which is annoying.

I have a couple of ideas of why this is happening - one is all those 'if' statements inside the workaround functions for WebGL 1.0's limitation of no dynamic array indexing. GPUs do not like too many branching 'if' statements - if you throw too many at it, the shader just crashes, or won't even compile. The only glimmer of hope is that maybe I overlooked something because antimatter.js was able to load thousands of triangles with WebGL 1.0, and I am using the same workaround functions that I manually unpacked from the minified file. The other culprit might be quality of the BVH build. As you can see from my total overhaul of the BVH builder .js file, it is pretty complex with multiple levels of recursion (definitely the most complex 300 lines of code I have ever written/ported). I printed out the tree for low poly models and it seemed to be doing a good job of recursively splitting the triangles at each juncture, all the way down until it reaches 1 triangle, which it designates as a 'leaf'. But as it scales up to thousands of triangles, who knows if it still is doing its job right. It's hard to print out 20,000 lines of data and make sense of it.

The other fork I could take is just to abandon the WebGL 1.0 workaround and go with WebGL 2.0, getting rid of all those 'if' statements in the workarounds. This sounds simple, just call WebGL 2.0 renderer from three.js setup code - but it is more involved. Since WebGL 2.0 uses OpenGL ES 3.0 (I know confusing right?) all of my established path tracing shader code on this repo will not compile right away. I have to manually go in and change stuff like 'attribute' to 'in' and 'varying' to 'in' and 'out', among other things. Here's a nice list of TODO's to get WebGL 2.0 working: WebGL 2.0 from 1.0

So once I can get WebGL 2.0 working, I can see if dynamic array indexing under WebGL 2.0 helps my crashing issue. As always I'll keep you all updated with my progress. Sorry it has been quiet around this repo the last couple of months, I have been wrestling with BVHs and WebGL 1.0 limitations and the like. :)

[MEBoo]

ahaha... I read about the crashes... may be too many calls/recursions with 1000+ triangles? I would consider to directly go for the WebGL 2.0 way... to simplify/optimize your BVH code.

Great work!

[erichlof]

@MEBlabs I'm happy to report I successfully integrated WebGL 2.0 into my older test path tracing demo! So that means I can definitely go the WebGL 2.0 way now. I'll be making some updates to all the demos on this repo very soon. I need to manually go in and reflect the changes on each shader and the THREE.WebGLRenderer() setup code inside the init() functions for each demo. But it should work.

Yeah I don't know yet why the WebGL 1.0 version crashes after 1000 triangles. You mentioned recursion call amount, but actually no GPU shaders allow recursion of any kind, hence the stackLevel[x] approach. So you push and pop all the various branches as you descend the BVH tree which is why I needed dynamic array indexing. Now on the CPU side, it loves recursion. So that's why I went the recursion way with the Builder.js file. That uses JavaScript and is strictly CPU-side. I wish though that GPUs allowed recursion, that would make things much simpler! ;) I'll keep investigating but in the meantime I'll try the WebGL 2.0 way, which has more promise.

[MEBoo]

Wow great work!! Yes sorry I didn't studied yet :(

Leave the old webgl 1.0 demo alone, and go for the new promise ;)

Thankyou! PS: can we communicate in other way other than using the issues in reddit? I should ask you only something about materials and shading... can we use in Skype/Discord?

[erichlof]

Hi @MEBlabs Sorry for the late reply, I have been busy migrating the whole repo to WebGL 2.0. Also I just changed the GPU random number generator from a common GPU hack - fract(sin(largeNumberSeed)) found all over the internet, to a more traditional integer hash using bit shifting, provided by iq on ShaderToy. WebGL 2.0 allows bit manipulations inside shaders. The result is faster, cleaner convergence on all demos.

This bit manipulation capability also points towards Morton Codes and Z-order curves for real-time building of dynamic BVH's that are created every animation frame (2 or 3 milliseconds for each build of the entire scene!) which would allow animation of triangle geometry while being rendered.

If you have a question regarding materials and shading, maybe just open a new issue so we can focus on that with the discussion. That way all viewers can benefit from your questions and my responses (hopefully, ha).

[erichlof]

Hi @EtagiBI , @MEBlabs , and everyone,

Just wanted to let you know that I recently implemented a much better .obj model BVH builder. This Iterative version replaces the Recursive version that was prone to crashing due to infinite loops. You can check out the demo: BVH WebGL 2.0.

Next on the TODO list is what this issue started out as: handling multiple OBJ files in the same scene. I am actively working on this, now that the BVH builder for single .obj model files is much more robust. I'll keep you all posted with any progress I can make! -Erich

[MEBoo]

Hi @MEBlabs Sorry for the late reply, I have been busy migrating the whole repo to WebGL 2.0. Also I just changed the GPU random number generator from a common GPU hack - fract(sin(largeNumberSeed)) found all over the internet, to a more traditional integer hash using bit shifting, provided by iq on ShaderToy. WebGL 2.0 allows bit manipulations inside shaders. The result is faster, cleaner convergence on all demos.

This bit manipulation capability also points towards Morton Codes and Z-order curves for real-time building of dynamic BVH's that are created every animation frame (2 or 3 milliseconds for each build of the entire scene!) which would allow animation of triangle geometry while being rendered.

If you have a question regarding materials and shading, maybe just open a new issue so we can focus on that with the discussion. That way all viewers can benefit from your questions and my responses (hopefully, ha).

Hi Erich. I'm sorry but this month I had too much work ... but now I'm ready to test your progress! My question maybe obvious or stupid, that's the why I didn't open a new issue! But after more code-testing maybe I'll be ready to ask that question opening new issue

[MEBoo]

Hi @EtagiBI , @MEBlabs , and everyone,

Just wanted to let you know that I recently implemented a much better .obj model BVH builder. This Iterative version replaces the Recursive version that was prone to crashing due to infinite loops. You can check out the demo: BVH WebGL 2.0.

Next on the TODO list is what this issue started out as: handling multiple OBJ files in the same scene. I am actively working on this, now that the BVH builder for single .obj model files is much more robust. I'll keep you all posted with any progress I can make! -Erich

Holy shit, fast real time OBJ rendering! You did it! Do you think that handling many OBJ will be a problem regarding performance? It seems everything so smooth now!

[erichlof]

@MEBlabs Yay! About multiple OBJ files, I'm trying to decide how to tackle this. I may be able to use the THREE.OBJLoader multiple times to load each model, and just assign each object to its own GPU texture with its own BVH. The alternative is to merge all the scene models' triangles into one huge BVH texture, but that seems on the outset a little too rigid to me (what if we want to individually move the models around the scene?, etc...)

First I'm currently taking baby steps - tonight I just figured out how to merge multiple parts (children / subMeshes) specified in the same single .obj file. If you look at some of the .obj files like male02.obj, there are separate 'objects' designated with the letter 'o' (or 'g' in this case) inside the file that define the different subMeshes/children of the model. For instance, the male02.obj has the body suit frame as the main parent object, then the head, hair, feet, and hands as children objects. Now this is the type of file that 'does' need to be merged into one bigger BVH because it makes sense to create the BVH from all the triangles of the various parts; they all belong to the same model and are typically located right next to each other.

I'll be posting a demo of that soon! :-)

[MEBoo]

@MEBlabs Yay! About multiple OBJ files, I'm trying to decide how to tackle this. I may be able to use the THREE.OBJLoader multiple times to load each model, and just assign each object to its own GPU texture with its own BVH. The alternative is to merge all the scene models' triangles into one huge BVH texture, but that seems on the outset a little too rigid to me (what if we want to individually move the models around the scene?, etc...)

First I'm currently taking baby steps - tonight I just figured out how to merge multiple parts (children / subMeshes) specified in the same single .obj file. If you look at some of the .obj files like male02.obj, there are separate 'objects' designated with the letter 'o' (or 'g' in this case) inside the file that define the different subMeshes/children of the model. For instance, the male02.obj has the body suit frame as the main parent object, then the head, hair, feet, and hands as children objects. Now this is the type of file that 'does' need to be merged into one bigger BVH because it makes sense to create the BVH from all the triangles of the various parts; they all belong to the same model and are typically located right next to each other.

I'll be posting a demo of that soon! :-)

Yes I think it's right... 'g' means 'group' and an OBJ could have many groups inside... everyone should be merged into one BVH. Stupid question: can every sub mesh have its material assigned?

For multiple OBJ I think that the way is to have multiple BVH, so it's possibile to delete/add a new OBJ runtime without refreshing the others

[erichlof]

@MEBlabs Yes every sub-mesh can have its material assigned (Well, it currently can when doing normal three.js WebGL rendering). I don't have them on this repo at the moment, but most of the OBJ files that I use to load the model geometry come with accompanying MTL files. These files are human-readable and define the material properties for the main parent mesh and each of its children (parts). If the model needs textures or maps of any kind, they are specified in the MTL file as well.

It is pretty straight forward to handle these files and assign material properties to the geometry when doing 'normal' or 'typical' 3d rendering - we've been doing it for decades. What is not so clear is how to intercept all this data, and when to intercept it (at loading time? or later when the three.js mesh has already been created? etc..) and how to stick it all inside a packed 2048x2048 triangle data-texture so that the GPU can consume it efficiently while ray tracing. A ray can strike any triangle of a 30,000 triangle mesh and it needs to immediately know what that triangle's properties are. I am currently trying to tackle this problem and I don't have a simple solution yet.

And regarding different OBJ models, yes each needs to have its own BVH, because like you mentioned, you might want to add or delete meshes at any time. I've heard that some renderers like Octane have a BVH for the BVHs! Meaning, all the scene's mesh general bounding boxes are put in a big over-arching hierarchical structure that covers the entire world. Then, when a ray hits one of those general big bounding boxes for a mesh, the individual BVH for that mesh is fetched from the GPU, and then the personal tree for that mesh is descended, however deep it might go.

p.s. I've also got a working GLTF model viewer inside the path tracer - whoo hoo! Well, I have the geometry at least. Materials problem is same as stated above. But it will be nice to support OBJ/MTL as well as GLTF (the latter is much more web-friendly and recommended by the authors of three.js)

[MEBoo]

@erichlof yes I know about OBJ + MTL, that's the why I asked if your pathtracer can render every type of OBJ or not ... About intercepting data (object material properties): I suggest to "read" the mesh after it has been loaded... since so it could be possible to assign/change a material property runtime... If you manage to pass any "geometry with triangles" with its own material data to the GPU, instead of an "OBJ optimized data", we can render every type of scene.... a format indipendent pathtracer... Is this so hard now that BVH is complete?

BVH for BVHs ... I vote for this, that's the way :)

[erichlof]

@MEBlabs Sorry you already knew about some of the topics I was writing about - I want others to be able to publicly read our discussion here and know what we're talking about. About when to intercept the data stream, yes that is a good suggestion to capture it after the three.js mesh has been created, thus providing a model "format-agnostic" path tracer.

Luckily the smart folks behind three.js have provided various loaders for the dozens of possible model formats in the wild, and then they convert them into a three.js 'Mesh' with geometry and materials. I can directly access the geometry, that's how I got things to work in the first place. But with materials, it's a little less clear to me at the moment: I need to study how three.js encodes those on a per-triangle basis, if at all, or the next best thing, on a model child-object basis (so all the triangles for that child part of the model have the same material properties) - I could also make that work with that info.

[MEBoo]

Yes, in THREE js there is the GROUP (or Object3d) that contains MESHes ... and the Material is applied to the mesh, not to the group... so when you speak about "children", you are meaning "meshes" inside a group. A Mesh cannot contain another mesh... so I think that your material problem is "only" about the mesh's material itself ... then it's like the "BVH for BVHs" problem -> a group can contain more meshes each one with its material but the difficulty is only on the mesh's material.

I don't know exactly how the path tracer works, but I think that you should traverse the threejs scene and pass everything ... building BVH for BVHs etc

PS: i changed my nick due to a reorganization ;)

[EtagiBI]

Wow, I'm impressed by your progress. Congratulations! Although we use non-three.js solution for rendering now, it would be very interesting to follow your project.

[erichlof]

@MEBoo I went back and studied how three.js turns the model format data ( .obj, .mtl, .gtlf, .fbx, etc. ) into a THREE.Mesh with geometry and materials. So after three.js loads and converts the data into the THREE.Mesh, I intercept the mesh/scene and traverse its children (which are meshes themselves, as you mentioned). I successfully got the 'male02' OBJ+MTL model working, which was tricky because if you inspect the male02.obj file, you'll see that it has 14 meshes, all with their own geometry and materials, the male02.mtl file shows 5 materials some of which use different or the same textures, but there are only 3 textures in the materials folder! Maybe I should have chosen a more simple model example (ha! too late now), but it's proof that the path tracer can load in a multiple child-mesh .obj that re-uses an arbitrary number of diffuse-albedo color texture maps.

GLTF formats are a little more straightforward, there's just one .gltf file that contains the info for the meshes and texture paths. The triangle vertex data is neatly packed in a .bin file that accompanies the .gltf file. I haven't tried FBX models yet, but I will soon - they should work too (in theory).

The good news is that the path tracer is essentially format-agnostic: it doesn't care how you get the models imported or what format they are. It just intercepts the scene/meshes once three.js creates them. Hats off to the authors of all those different types of loaders - it makes my life much easier!

[erichlof]

@EtagiBI and @MEBoo and all,

Thanks @EtagiBI ! Yes, luckily I have been making some good progress in this area of the path tracer. I must give credit to the authors of the various loaders, and of the three.js Mesh creation algos from those loaders' data. This all wouldn't be possible without their great initial work.

I'm happy to report another progress milestone: the ability to load models of various formats and now use their diffuse texture maps for materials. Here is a OBJ+MTL model loading demo. p.s. you might have to refresh/re-load the page a couple of times, for some reason it gave me a image.src 'could not find' error on the first try - will investigate to see if I can prevent that from happening.

And to prove that the path tracer is format-agnostic, here is a GLTF model loading demo

Note that the only difference between the source code of both demos is that I call the constructor of the desired loader, then call the '.load' method for that loader, with the file path to the desired model of that type.

Although everything appears to be working, there remains much work to do: If you look carefully at the models, they all use the 'clearcoat' material, this is because MTL files were invented long ago before PBR materials were a thing, and there's no way of me knowing what the user wants for each material of each part of a model - is it diffuse, is it metal, is it emissive, is it glass, what's the IoR, what's the roughness?, etc. Ben Houston, creator of Clara.io, and frequent contributor to three.js, has proposed an industry-wide addendum to the MTL format to help with this very problem: proposal . Although this would be perfect, I haven't seen wide adoption of this extension by major modelling software vendors.

With the GLTF format, it is more up-to-date, with parameters for most, if not all, of the PBR material options. Now that I have the diffuse maps working, it would be nice to also use and render the normal, emissive, metalness, roughness maps, etc.

After I have a solution to the above materials problems, I will begin work on creating a BVH for the BVHs in order to start loading multiple models, each with their own meshes, geometries, and materials. As always, I will keep everyone updated! :-)

[MEBoo]

@erichlof great job in a short time!! I just read your code and I suggest you what I said before. You are now reading the output of a Loader (OBJ or GLTF) that can or can't support some material property. But if I use THREE JS, I use it to build a scene, loading models in various format, positioning them, adding lights, tuning its materials, changing them. So for the best THREE JS integration, you should parse the entire scene with its THREE.js meshes and materials. You could start supporting MeshPhysicalMaterial (that adds "coating" to the MeshStandardMaterial) with all/some its properties, so no problem if OBJ/GLTF doesn't support "metalness" etc. I know, I'm talking more about code architecture when you are concentrated on working on algorithms... but I think that a developer using your renderer shoud do: 1) creating a THREE scene 2) erichPathTracer(scene,renderer)

So, as you said, before developing BVH for BVHs, you could find solution to the materials problems, but supporting/intepreting actual three materials properties (color, map, metalness, roughness, metalnessMap, roughnessMap, reflectivity, clearCoat, clearCoatRoughness, etc) and reading it from meshes (1 for the moment) instanced on a scene.

The properies supported here https://threejs.org/docs/#api/en/materials/MeshPhysicalMaterial

Hey I hope you understand what I only suggest for your already huge and impressive work.

[MEBoo]

@MEBoo I went back and studied how three.js turns the model format data ( .obj, .mtl, .gtlf, .fbx, etc. ) into a THREE.Mesh with geometry and materials. So after three.js loads and converts the data into the THREE.Mesh, I intercept the mesh/scene and traverse its children (which are meshes themselves, as you mentioned). I successfully got the 'male02' OBJ+MTL model working, which was tricky because if you inspect the male02.obj file, you'll see that it has 14 meshes, all with their own geometry and materials, the male02.mtl file shows 5 materials some of which use different or the same textures, but there are only 3 textures in the materials folder! Maybe I should have chosen a more simple model example (ha! too late now), but it's proof that the path tracer can load in a multiple child-mesh .obj that re-uses an arbitrary number of diffuse-albedo color texture maps.

GLTF formats are a little more straightforward, there's just one .gltf file that contains the info for the meshes and texture paths. The triangle vertex data is neatly packed in a .bin file that accompanies the .gltf file. I haven't tried FBX models yet, but I will soon - they should work too (in theory).

The good news is that the path tracer is essentially format-agnostic: it doesn't care how you get the models imported or what format they are. It just intercepts the scene/meshes once three.js creates them. Hats off to the authors of all those different types of loaders - it makes my life much easier!

hey sorry I just read this first reply after replying to the other! Now I understand better your second reply :D

[erichlof]

Hi @MEBoo Thanks for your suggestion about the three.PhysicalMaterial - yes that one would be the best fit, although the three.StandardMaterial could work as well, because it has enough PBR parameters to make do. The issue stems from the fact that (as you know now), three.js converts all OBJ+MTL models into meshes with three.PhongMaterial as the default. And PhongMaterial is not enough info for my path tracer to do the model justice. It is not three.js's fault, because historically, the OBJ+MTL model system was defined with the BlinnPhong lighting model in mind, that's why three.js chose that material. I do hope Ben Houston's suggestion of extending the MTL model to include PBR will gain wide support someday. But until that day, In your opinion, should I suggest the use of GLTF models for the default pathtracing model format, rather than the older OBJ+MTL? For those that only have OBJ+MTL, there are plenty of free converters online, like the aforementioned Clara.io

[MEBoo]

@erichlof in my opinion that is only an OBJ+MTL problem that should not be a your problem, you should support the THREE new standard PBR materials (standard or physical or both) and not more ... If you could parse the three scene (mesh geometries, materials, lights), then anyone would be free to use the loader which want to, and could convert the meshes materials to a PBR one, since you are developing a PBR renderer then no body expects to render a non PBR material ;)

For the Stanrdard vs Physical, the last give some better tuning for non metal materials, since clear-coat and reflectivity applies more to non-metal surfaces like plastics or glossy woods... you could check the physical shader implementation and find how these props are applied

thanks for your attention :)

[erichlof]

@MEBoo and all,

Ok thanks for responding. So it should be up to the user to make sure that the materials are in PBR form before they are loaded into the path tracer. With Blender, Clara.io, etc. there are plenty of free software solutions to help with that. In my opinion, the format which offers the least resistance is GLTF. It is an open format, and all modern game engines and modelling software have importers, converters, and exporters for GLTF. I believe that three.js will turn the GLTF vertex encoding into a three.Mesh with enough PBR info to use in my path tracer. Do you agree?

I'm just trying to get an idea of what the user's workflow would be: I'm thinking they first acquire a model with PBR textures, or PBR parameters set, then use a free converter to convert it to GLTF form. Then they use three.GLTFLoader (inside my html file) to load the entire scene with its models and lights. This then is converted by three.js to a three.Scene with various three.Mesh's. Finally, I extract the PBR info, textures, vertex data, lighting info, etc. and then place everything in the GPU BVH. Does this sound about right? Am I missing any steps?

Thanks for your input. To others reading this, feel free to chime in. I'm just trying to get a feel for how everything should flow in and out of the path tracer.

[MEBoo]

@erichlof it's very simple to give the users a Material converter like this: scene.traverse( if (mesh) { copy color then map specular to roughness / metalness } ) so I think that is not a problem now. For the GLTF i think that you can suggest the users to adopt it because it uses PBR materials, but if you can parse the scene then your path tracer is totally format indipendent.

For the last question, that sounds very right! Only a thing more: if you can do what you said, then consider the case where the scene may change runtime. For example I may change colors or a material property, or generating a geometry (a wall or other) and assign a material -> that's the why I would consider both the THREEJS PBR materials (even if GLTF doen's export the full three.physical material). Consider a web car configurator where a user can change materials, these are defined by the developer and not exported from a software. Regarding this, I think that it would be only an addon to your trace equation, so only a little work about undestand what the other properties stand for.

That's all, this will become the best practical and quality pathtracer available