Regression in rendering textured meshes (imported via Wavefront OBJ files)

[trevorld]

I'm observing a regression in rendering textured meshes (originally imported into {rayrender} from Wavefront OBJ files). It seems that part of the textures are now rendered as transparent (even though they shouldn't be since there is no transparent alpha value set in the textures' color values). Here is a reproducible example rendered with the development version of {rayrender}:

library("piecepackr")
library("ppgames") # remotes::install_github("piecepackr/ppgames")
library("magrittr")
library("rayrender")

df <- ppgames::df_xiangqi()
envir <- game_systems("dejavu3d", round=TRUE, pawn="peg-doll")
l <- pmap_piece(df, piece, trans=op_transform, envir = envir, scale = 0.98, res = 150, as_top="pawn_face")
table <- sphere(z=-1e3, radius=1e3, material=diffuse(color="green")) %>%
         add_object(sphere(x=5,y=-4, z=30, material=light(intensity=420)))
scene <- Reduce(rayrender::add_object, l, init=table)
rayrender::render_scene(scene, lookat = c(5, 5, 0), lookfrom = c(5, -7, 25),
                        width = 500, height = 500, samples=200, clamp_value=8)

In contrast when using {rayrender} versions (strictly) before 88b28a80780fabb0c5379302db5b3 (May 25) that scene with the same code instead rendered like this without such partial-transparent-textures issues:

OS info (uname -a): Linux 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[trevorld]

I've re-written the above reproducible example to get rid of the piecepackr::piece() abstraction over rayrender::obj_model(). One half of code generates the Wavefront OBJ files (with {piecepackr} and {ppgames}) and the next half of code imports/renders these Wavefront OBJ files using only {rayrender}:

# generate Wavefront OBJ files with {piecepackr} and {ppgames}
library("piecepackr")
library("ppgames") # remotes::install_github("piecepackr/ppgames")
df <- ppgames::df_xiangqi()
envir <- game_systems("dejavu3d", round=TRUE, pawn="peg-doll")
l <- pmap_piece(df, save_piece_obj, trans=op_transform, envir = envir, scale = 0.98, res = 150, as_top="pawn_face")
obj_files <- sapply(l, \(x) x$obj)

# import/render Wavefront OBJ files in {rayrender}
library("rayrender")
l <- lapply(obj_files, \(x) obj_model(x, texture = TRUE))
table <- sphere(z=-1e3, radius=1e3, material=diffuse(color="green")) |>
         add_object(sphere(x=5,y=-4, z=30, material=light(intensity=420)))
scene <- Reduce(add_object, l, init=table)
render_scene(scene, lookat = c(5, 5, 0), lookfrom = c(5, -7, 25),
             width = 500, height = 500, samples=200, clamp_value=8)

[tylermorganwall]

Some (not all) of the images you are generating are in fact 4-layer PNGs with some alpha values set to zero (see attached image).

The alpha detection code checks the alpha layer for any value not equal to one and if it finds any, initializes the alpha texture. I would examine the output of your image generation code to make sure you are indeed generating completely opaque images (load the image and check the 4th layer in the array if it exists).

[trevorld]

Thanks for the follow-up!

The alpha detection code checks the alpha layer for any value not equal to one and if it finds any, initializes the alpha texture.

I use gridGeometry::polyclipGrob() to fill in any "empty" space around game pieces in png textures. It isn't perfect and sometimes it leaves random pixels #0000 even though the actual object overall is actually quite opaque. Since users can customize their game pieces with arbitrary grid grobs it is hard for me to know ahead of time if a game piece is not transparent and initially fill the texture background with an opaque color (because that would mess things up when we do need transparent textures).

Would it be possible (like {rgl}) to add an option to rayrender::obj_model() that allows the user to manually turn on/off the alpha texture (defaulting to alpha detection)? Or alternatively tweak the alpha texture detection code so it only loads if the overall texture opaqueness is less than something like 0.999? I use something like the following in {piecepackr} to detect whether a given texture should go to "alpha transparent" mode when loading Wavefront OBJ files into {rgl} and I could also use it with {rayrender} if it had that option:

    if (is.na(textype)) {
        r <- png::readPNG(obj$png)
        textype <- ifelse(dim(r)[3] > 3 && mean(r[,,4]) < 0.999, "rgba", "rgb")
    }

[tylermorganwall]

You might want to look and make sure that you are indeed only setting a few pixels here and there in the alpha channel—I just pulled up those pixels as an example that the "no transparent alpha value set" wasn't exactly correct since I found them right away (and I wasn't getting the transparency behavior you're showing), but it appears the transparency you're generating in the letters is text image is being copied into the alpha channel as well.

[trevorld]

(and I wasn't getting the transparency behavior you're showing)

So my example doesn't reproduce on your machine? Would it be helpful at all to try to generate a Docker image (with an Ubuntu base) that reproduces the result?

You might want to look and make sure that you are indeed only setting a few pixels here and there in the alpha channel

If I render the Wavefront OBJ files in {rgl} with alpha transparent textures activated I don't observe any of the transparency effect that I do with {rayrender}:

# generate Wavefront OBJ files
library("piecepackr")
library("ppgames") # remotes::install_github("piecepackr/ppgames")
df <- ppgames::df_xiangqi()
envir <- game_systems("dejavu3d", round=TRUE, pawn="peg-doll")
l_obj <- pmap_piece(df, save_piece_obj, trans=op_transform, envir = envir, scale = 0.98, res = 150, as_top="pawn_face")
obj_files <- sapply(l_obj, \(x) x$obj)

# render Wavefront OBJ files in {rgl}
library("readobj")
library("rgl")
open3d()
clear3d()
bg3d(color = "green")
l_rgl <- lapply(l_obj, \(x) {
    material <- list(color = "white", texture = x$png, textype = "rgba")
    mesh <- readobj::read.obj(x$obj, convert.rgl = TRUE)
    rgl::shade3d(mesh, material = material)
})
view3d(phi=-45, zoom = 0.9)
if (Sys.which("wmctrl") != "") system("wmctrl -r RGL -e 0,-1,-1,600,600")
Sys.sleep(3)
rgl.snapshot("rgl_snapshot.png", top = FALSE)

If I zoom in on one of the textures of one of the Wavefront OBJ and plot it onto a red color background the texture seems to be opaque except for a few dots around the edges of the circle "coin faces" (seems {gridGeometry} isn't quite able to fully compute a perfect inverted circle to perfectly match the circular coin grobs). In particular the green crown and black edge parts of the texture are opaque but appear transparent in my {rayrender} rendering:

# render Wavefront OBJ files in {rayrender} with arrow pointing at 29th OBJ
library("rayrender")
l_ray <- lapply(obj_files, \(x) obj_model(x, texture = TRUE))
table <- sphere(z=-1e3, radius=1e3, material=diffuse(color="green")) |>
         add_object(sphere(x=5,y=-4, z=30, material=light(intensity=420)))
scene <- Reduce(add_object, l_ray, init=table)
scene <- add_object(scene, rayrender::arrow(start = c(5, 0, 1), end = c(5, 3.5, 1), material = glossy(color = "purple")))
render_scene(scene, lookat = c(5, 5, 0), lookfrom = c(5, -7, 25),
             width = 500, height = 500, samples=200, clamp_value=8, filename = "scene_with_arrow.png")

# focus in on texture of 29th OBJ
library("grid")
library("png")
texture <- png::readPNG(l_obj[[29]]$png)
png("texture_with_red_bg.png", bg = "red", width = 281, height = 112)
grid.raster(texture)
dev.off()

[tylermorganwall]

I'm not getting the behavior on my local system, but I've also overhauled the OBJ meshing internals so it may be fixed. Check out the memory_improvements branch and see if you still encounter the alpha texturing issue.

[trevorld]

Check out the memory_improvements branch and see if you still encounter the alpha texturing issue.

I checked out and installed the memory_improvements branch but I can't figure out how to now load in the textures. Reading the documentation from ?obj_model it seems I should replace obj_model(x, texture=TRUE) with obj_model(x, load_material = TRUE, load_textures = TRUE) but this just leads to white material being rendered without textures.

# generate Wavefront OBJ files
library("piecepackr")
library("ppgames") # remotes::install_github("piecepackr/ppgames")
df <- ppgames::df_xiangqi()
envir <- game_systems("dejavu3d", round=TRUE, pawn="peg-doll")
l_obj <- pmap_piece(df, save_piece_obj, trans=op_transform, envir = envir, scale = 0.98, res = 150, as_top="pawn_face")
obj_files <- sapply(l_obj, \(x) x$obj)

# render Wavefront OBJ files in {rayrender}
library("rayrender")
l_ray <- lapply(obj_files, \(x) obj_model(x, load_material = TRUE, load_textures = TRUE))
table <- sphere(z=-1e3, radius=1e3, material=diffuse(color="green")) |>
         add_object(sphere(x=5,y=-4, z=30, material=light(intensity=420)))
scene <- Reduce(add_object, l_ray, init=table)
render_scene(scene, lookat = c(5, 5, 0), lookfrom = c(5, -7, 25),
             width = 500, height = 500, samples=200, clamp_value=8)

[tylermorganwall]

Update to the latest version of the branch (477d921d9e6acb09c66ae8af15f559efac20e34d) and try again.

[trevorld]

Update to the latest version of the branch (https://github.com/tylermorganwall/rayrender/commit/477d921d9e6acb09c66ae8af15f559efac20e34d) and try again.

Output looks good to me after updating to the latest version of the branch: