Reference: Add texture support

[Zinggi]

Hi there :wave:

This adds very experimental texture support:

This pull request is not meant to be merged directly, but as a hopefully useful reference for the future.

I thought quite a bit about how I could best contribute to this amazing project.

At first I tried to integrate both textures and the single value API, but I quickly realized, that that would mean a lot of code duplication, especially for glsl code. Also, basically copy pasting your shader code with modifications produces a very ugly diff.

So instead, I decided to make changes to the existing API, without supporting the old API, this produces a much nicer diff.

Before this can really be implemented, I think the problem of the growing glsl complexity has to be addressed first. Since the new webgl library no longer supports unsafeShader, I think the only maintainable way for the future is gonna be code generation (kinda like dillonkearns/elm-graphql). (Or figuring out how to bring back unsafeShader with elm 0.19...)

This way, all sorts of combinations between textures, single values and generated textures becomes possible. E.g. ideally, I'd like the material type to be something like this:

type alias Material = 
    { baseColor: ColorInput
    , metallic: NumberInput
    , roughness: NumberInput
    }

type ColorInput = SingleColor Vec3 | ColorTexture Texture | GeneratedTexture GenFunction
type NumberInput = Number Float | SingleChannelTexture Texture

type GenFunction = Perlin | ...

Another thing missing from this PR is generating UV coordinates for the generated meshes. I currently don't know how to do that, but if I find some more time I'll add it to this PR as well at some later point.

Another thing I'd like to look into is normal and displacement maps, but that requires tangents, which I also don't know yet how to compute...

Cheers, I hope this form of contribution is helpful

[ianmackenzie]

Thanks for doing this @Zinggi! I confess I was putting off looking through the changes since I thought there was going to be a ton, but it's not as much as I thought =)

I was wondering how you handled texture coordinates - the sphere things is clever.

I also don't see a way to support both single values and textures cleanly without some sort of code generation. Having an uber shader of some sort would be nice, but I think that's a non-starter since I don't think there's any way in Elm to have a dummy/unset texture. (If a shader has a sampler2D, then that means a Texture is going to have to be passed in as a uniform, and the only way to get a Texture is via a Task...). Code generation would even help with the existing shaders - a bunch of stuff like gamma correction and computing incident light values is currently duplicated between different shaders. I hate having complex build systems, but it seems unavoidable here! I might do something like also commit the generated files, so at least people working on unrelated parts of elm-3d-scene don't have to worry about any kind of build scripts.

I also largely agree with what you've sketched out for Material, although I might try to tweak things a little bit so that the simple case is still fairly simple - I was thinking of something like

Drawable.realisticMaterial : { baseColor : Color, roughness : Float, metallic : Float } -> Mesh ... -> Drawable ...
Drawable.materialTextures { baseColor : Texture, roughness : Texture, metallic : Texture } -> Mesh ... -> Drawable ...

where Texture is an elm-3d-scene type that allows for textures loaded from file, procedural textures, or simple constant value textures. I'd like to play around with a few different versions and see what they feel like.

[Zinggi]

I thought a bit more about code generation and over the weekend I played around with an idea for generating GLSL code.

I started looking at an old, incomplete project of mine, but that approach has lots of issues and is too high level for this anyway. So I tried a different approach, which is almost just string interpolation, just with a bit of extra magic.

The new project is here.
As a test, I used it to generate the physicalFragment shader, with and without textures. The code for this is here.

I think it turned out quite nice, but I'm not sure if this will still be viable if more variations are needed.

It's currently missing vertex shaders and some docs, but I'm hoping to finish it up over the next weekend for publishing.

Maybe this will be useful for your project, I'm curious to hear what you think!

[ianmackenzie]

Hey @Zinggi, just getting texture support working natively in elm-3d-scene now (plain color textures are already working) and was about to try to adapt this example - was there a particular reason for the UV coordinate swizzling in

vec3 diffColor = texture2D(color, uv.yx).rgb;

? Was that just a bit of a hack since the texture image orientations don't match each other or something? Would the 'right' solution here be to fix the original images or is UV tweaking like this common, in your experience?

Fundamentally I'm wondering if I'll need to add something to the elm-3d-scene API to (for example) specify whether UV (0, 0) is at the top left, bottom left or middle, or whether it's reasonable to just assume that all supplied texture images have (0, 0) at the bottom left.

[ianmackenzie]

BTW if you were curious, I ended up writing my own shader code generation script (based on another project of mine, elm-script) - the relevant files are GenerateShaders.elm and Glsl.elm.

I was pretty happy with how it worked out, you basically define a bunch of attributes/uniforms/constants/functions/shaders separately, declare the dependencies between them, and the generator figures out the complete set of stuff that needs to be included in the full generated shader. So if a particular shader uses functions foo and bar, but foo depends on baz, then all three functions will end up getting included in the generated code.

[Zinggi]

Awesome to see you're making such progress!

was there a particular reason for the UV coordinate swizzling in (...) uv.yx

I think this was a typo. Looking at the screenshot, the elm logo is flipped :laughing: Sorry if that caused confusion.

I'm wondering if I'll need to add something to the elm-3d-scene API to [specify UV coordinates]

I don't know if this would be out of scope for this package, but a common way to create animated textures like water, fire, smoke, etc. is to animate the UV offset. E.g. like here.

If this is not something you want to support, then I don't think flipping UV coordinates is needed, as this can always be done in a program like Blender.

---

BTW if you were curious, I ended up writing my own shader code generation script

Looks like you found a good trade-off between reuseability and copy-paste-ability. And elm-script looks pretty cool, I want to try that out some time. I'd love to create a macro system to generate elm code at compile time, but that's a pretty big project that's very hard to get right...

Keep up the good work :+1:

[ianmackenzie]

Thanks @Zinggi! I think I'll probably keep UV handling simple/non-customizable for now, then, and once elm-3d-scene 1.0 is out the door we can think about how best to approach fancier cases like the animation stuff you describe. Explicitly supporting custom UV mappings could be one approach, but I'd also like to make it possible to define a Material by just supplying your own fragment shader (which could then do whatever tricky texture lookups you want).

[ianmackenzie]

Hey @Zinggi PBR textures are now (mostly) working in master! There's a live example at https://ianmackenzie.github.io/elm-3d-scene/examples/metal.html (I switched back to the versions without the Elm logo since I didn't want to have to deal with the stetching). That corresponds to the TexturedSphere example in the repo.

Doesn't seem quite right yet though - for one thing spinning the sphere with the mouse works but there's an annoying pause the first time you try it (which is weird, because the sphere is rendered so clearly all the meshes/shaders/textures have been loaded already). More importantly, it just doesn't look very metallic so there may still be some fundamental rendering problems. It certainly doesn't look nearly as metallic as the preview at https://cc0textures.com/view?tex=Metal03; it's possible that that's just because that preview has some environmental lighting and my example doesn't, but I'm not convinced.

If you have any ideas what might be going on I'd love to hear them - in the shader I'm converting from sRGB to linear colour when reading the colour texture and using the raw linear value when reading roughness/metallic, which I think is correct. I should probably try to find some simple test/calibration scenes somewhere that I can use to validate my PBR shaders...

[Zinggi]

I already replied on slack, but if others are reading this:

I think that the image based reflection and lightning technique that cc0textures uses makes all the difference in selling the metal look. Your render looks correct to me (but I'm no expert).
Also, when you look at e.g. this page, they seem to also use a cube map of the sky when roughness decreases and metal-ness increases.

By the way, I don't notice any pause on first mouse move on my system, your demo works perfectly for me.

[ianmackenzie]

I think now that this has been implemented in elm-3d-scene, this PR can be closed. Was a very useful reference - thank you @Zinggi!

[Zinggi]

Sure, this was never meant to be merged anyway :smile: Congratulations on following through, I think it turned out very nice! I'm looking forward to the first official release of elm-3d-scene!