"entity_id" mode always exports pure white?

[kevinfoley]

When testing the "entity_id" mode of nvisii.render_data(), I am finding that the rendered image is always pure white.

In path_tracer.cu, it appears that this block of code is responsible for determining the color:

else if (LP.renderDataMode == RenderDataFlags::ENTITY_ID) {
        renderData = make_float3(float(entity_id));
}

The make_float3() function in float3.h:

__device__ float3 make_float3(float c) {
    return make_float3(c, c, c);
}

The entity_id is a unique integer. Assuming it starts at 0, the colors corresponding to each entity would look like this, in order:

(0,0,0) (1,1,1) (2,2,2) (3,3,3)

etc. If NVISII works like most engines, (0,0,0) would be pure black, (1,1,1) would be pure white, and any greater value such as (2,2,2) would still appear as pure white. In other words, the very first entity would be pure black and every other entity would be pure white. I'm not sure why the background is also pure white.

[kevinfoley]

I see the same issue with the 09.meta_data_exporting.py example script; every pixel in the exported entity_id.exr image is pure white.

[natevm]

Hi Kevin,

I’m fairly confident in the entity ID code, as we use that in several research projects successfully.

i believe your testing methodology is wrong, the background ID might also be non-zero (might be infinity IIRC? Something non-negative but also definitely not a normal entity ID…). The exr isn’t really meant to be reinterpreted as color values here, so “white” occurs from you reinterpreting an entity ID as a color value.

[natevm]

Also, cameras and lights are entities, and have IDs, yet won’t be visible on screen. So you might not see a black object with an ID of 0

[kevinfoley]

@natevm So you're saying that the values in the EXR will match the entity IDs, but aren't valid colors? We'll check that.

However, it would be very nice if the entity_id mode produced valid colors (or you added an alternate mode with valid colors) so that segmentation can clearly be seen in an image editor such as Photoshop. For example,

else if (LP.renderDataMode == RenderDataFlags::ENTITY_ID) {
        float brightness = float(entity_id) / float(INT_MAX);
        renderData = make_float3(brightness);
}

[kevinfoley]

Yes, it looks like the image data contains valid entity IDs that just aren't valid colors and get interpreted as pure white. Here's a workaround for converting the original image data into an EXR with valid grayscale colors:

entity_count = entity.get_count()

segmentation_array = nvisii.render_data(
    width=camera_width, 
    height=camera_height, start_frame=0,
    frame_count=1, bounce=0,
    options="entity_id")

# convert to valid shades of gray
segmentation_array2 = np.array([x / entity_count for x in segmentation_array])
# reshape and export as exr
segmentation_array3 = segmentation_array2.reshape(camera_height,camera_width,4).astype("float32")
cv2.imwrite("segmentation.exr", segmentation_array3)

@natevm It would be helpful if the documentation noted this quirk, that the entity_id image will always appear to be pure white in an image editor but does contain valid segmentation data. Otherwise, users may reasonably assume that something is wrong when they open up the image file for quick visual verification and it appears to be blank.

[natevm]

However, it would be very nice if the entity_id mode produced valid colors (or you added an alternate mode with valid colors) so that segmentation can clearly be seen in an image editor such as Photoshop.

In photoshop you can simply shift the exposure of the HDR image... This seems like feature creep, and sets a bad precedent that all rendered data must have a "color transformed" version. This transformation is also completely subjective, and it's not necessarily the case that your proposed transformation is representative to what all users might want.

It would be helpful if the documentation noted this quirk, that the entity_id image will always appear to be pure white in an image editor but does contain valid segmentation data. Otherwise, users may reasonably assume that something is wrong when they open up the image file for quick visual verification and it appears to be blank.

Rendered data must be transformed in order to be visualized. And again, you can change the exposure in photoshop to see the segmentation. Many data values include negative ranges, which cant be easily visualized in photoshop, but we never intended for users to be able to do this anyway. Feel free to modify the documentation here in a pull request if you feel it is insufficient, and I would be more than happy to review and merge it in. Documentation is defined in the oxygen comments above function declarations.

https://github.com/owl-project/NVISII/blob/master/include/nvisii/nvisii.h#L320 https://github.com/owl-project/NVISII/blob/24e6cf8b6c9a72331a1124322ea2a2d3f57730b8/include/nvisii/nvisii.h#L350

[kevinfoley]

Most of the render_data_to_file() modes do generate images that can be viewed without any color transformation. I'm new to the world of CV, so maybe this is just a convention I'm unfamiliar with, but generally when an application outputs an image file, I'd expect the image to be visible as-is in an image viewer or editor. We did some work with another framework that output full-color segmentation images, and the banner image when you first visit this repo appears to have a grayscale segmentation image (third from the right in the bottom row, though perhaps I'm just misinterpreting another type of data), so I was thrown off when the entity_id output appeared to be blank.

[TontonTremblay]

Hey Kevin, welcome to the nice world of CV and graphics. Segmentation is normally never expressed as a color when stored. Having the entity id being stored is convenient for training and it is more flexible, e.g., you can go from entity to object class, but you cannot go from object class to entity. So that level is very flexible.

you can use the following snippet of code to get colors.

                from skimage.color import label2rgb 
                seg = pyexr.open(self.imgs[index][1]).get()
                # set background to 0
                seg[seg>3.4028235e+37] = 0 
                seg[seg<-3.4028235e+37] =0 
                seg = np.uint8(seg[:,:,0])
                image = np.uint8(image)

                print(seg.shape,seg.min(),seg.max())
                print(image.shape,image.min(),image.max())
                arr = label2rgb(
                    image = image,
                    label=seg.astype(int),
                    # alpha = 0.5,
                    bg_label=0
                )
                print(arr.shape,arr.min(),arr.max())
                image = PIL.Image.fromarray(np.uint8(arr*255))

This code loads an image and the segmentation, the background is removed and set to 0. Then you get the image mixed with the labels. I use this with my datasets I generate. Check the documentation for this label2rgb function online.

[kevinfoley]

Thanks! I'll mark this is closed. Hopefully if someone else gets confused by the segmentation image format in the future, they'll find this issue and it will clear up their questions.