Color errors in semantic map (UE5)

[cgsaxner]

Describe the bug I only tested this in UE5. I'm not sure whether this is a bug or I'm doing something wrong, but for some colors, the Plugin will produce erroneous colors in the semantic map and consequently, pixels belonging to the same class will have color values different by one digit.

For example, I have a class to which I assign the RGB color (1.0, 0.502886, 0.0) using the Color Picker in UE. According to the SemanticClasses.csv, this class gets mapped to (255, 188, 0). However, in the final image, some pixels within the class are incorrectly assigned the value (255, 189, 0). This only happens for some color values.

To Reproduce Steps to reproduce the behavior:

1. Build UE5 and EasySynth ue5 branch from source
2. Create level & open EasySynth Plugin
3. Manage Semantic Classes, add a class and select a slightly odd color, e.g. RGB = (1.0, 0.502886, 0.0)
4. Assign the semantic class to some objects
5. Render (tested with jpg and png)

Expected behavior All pixels belonging to one semantic class have the exact same

Screenshots Color in Unreal: Color after opening & displaying image using Python:

Configuration (if applicable):

• OS: Windows 10 Pro
• GPU: NVIDIA GeForce GTX 1080 Ti

[NikolaJov96]

Hi @cgsaxner,

Semantic colors are stored as integers throughout the plugin and saved as such in SemanticClasses.csv. It seems that the issue here is with the numerical accuracy of the post-processing material. Unfortunately, we cannot additionally process images after the rendering, as they are directly saved to the disc. It would require opening, altering, and resaving generated images.

I'll try to reproduce and see what I can do.

The quickest solution is probably to write a simple Python script that would round image pixels to the closest color found in SemanticClasses.csv.

[cgsaxner]

Thanks for the reply & insights. I also can only imagine that during post-processing, there is some small value added somewhere, which causes different rounding when going from float set with the color wheel to integer.

As you suggested, I created a Python script to map the pixel values. Maybe it is useful to anyone else:

from PIL import Image
import numpy as np        

# open SemanticClasses.csv and semantic image
classes = pd.read_csv(path.join(root_path, 'SemanticClasses.csv'))
seg_array = np.array(Image.open(seg_path))[:, :, :3]

# get colors from the .csv
colors = []
for cla in classes.iterrows():
    colors.append(tuple((cla[1].loc['255'], cla[1].loc['255.1'], cla[1].loc['255.2'])))

# compute closest color value
distances = np.zeros((seg_array.shape[0], seg_array.shape[1], len(colors)))
for i in range(len(colors)):
    distances[..., i] = np.mean(np.abs(seg_array - colors[i]), axis=2)

mask = np.argmin(distances, axis=2).astype(np.uint8)

# replace the pixels
for color_in_mask in np.unique(mask):
    seg_array[mask == color_in_mask] = colors[color_in_mask]