segmentation mask rendering problem?

[HaozhiQi]

Hi,

I was trying to generate a bunch of images and masks for the place_cups task. I generated the dataset using the following command: python tools/dataset_generator.py --save_path ./debug --tasks place_cups --image_size '256,256' --episodes_per_task 1 --variations 1

I assume each number in the segmentation map corresponds to certain types of objects. And indeed, some of the numbers are correctly associated with the object. For example, the image looks like:

The number 36 corresponds to:

which seems to be correct.

However, I found that some number corresponds to the wrong region. Specifically, number 35 corresponds to:

---

I guess this is not an expected phenomenon? Is this a bug of mask rendering? Or how should I get the correct mask for objects?

Thanks a lot!

[stepjam]

Hi @HaozhiQi.

This is not expected behaviour, so theres probably a bug in the mask decoding. I'll take a look. Thanks for pointing this out. Would it be possible for you send me a reproducible script that shows the bug?

[HaozhiQi]

Hi @stepjam ,

Sure! I use RLBench with version 1.0.9 (git checkout 1.0.9) and installed with python setup.py install --user. My pyrep version is 1.2.

As I mentioned above, I generated the observations using the official script:

python tools/dataset_generator.py --save_path ./debug --tasks place_cups --image_size '256,256' --episodes_per_task 1 --variations 1

Then I run the following script:

import os
import cv2
import shutil
import numpy as np

from tqdm import tqdm
from glob import glob

def main():
    src_dir = './debug/place_cups/'
    tar_dir = './debug/parsed_place_cups/'

    variations = sorted(glob(f'{src_dir}/variation*/'))
    for v_i, variation in enumerate(tqdm(variations)):
        episodes = sorted(glob(f'{variation}/episodes/episode*/'))
        for e_i, episode in enumerate(episodes):
            image_path = f'{episode}/front_rgb/'
            images = glob(f'{image_path}/*.png')
            os.makedirs(f'{tar_dir}/image/v{v_i:03d}_e{e_i:03d}/', exist_ok=True)
            for image in images:
                im_id = int(image.split('/')[-1].split('.')[0])
                tar_path = f'{tar_dir}/image/v{v_i:03d}_e{e_i:03d}/{im_id:04d}.png'
                shutil.copy2(image, tar_path)
                mask = cv2.imread(image.replace('rgb', 'mask'), 0)
                obj_ids = list(np.unique(mask))
                for o_id, obj_id in enumerate(obj_ids):
                    im = cv2.imread(image)
                    masked_image = (mask == obj_id)[..., None] * im
                    os.makedirs(f'{tar_dir}/{v_i}_{e_i}/', exist_ok=True)
                    cv2.imwrite(f'{tar_dir}/{v_i}_{e_i}/{obj_id}_{im_id}.png', masked_image)

if __name__ == '__main__':
    main()

[stepjam]

The issue was actually caused by a boundary being rendered. This is resolved in 1.0.9.

FYI, you might find it more convenient to use the task.get_demos() functionality, rather than parsing the demo folders yourself (see here). If you pass in the live=False flag, then it gives you a list of demos loaded from file. You can also set image_paths=True if you want the paths rather than the images themselves.

[HaozhiQi]

Hi @stepjam,

I observed something weird for the place cups task. For id 33, it gives: There is a small green segment on top of the cup, which should not be the case.

Moreover, when I'm playing with other tasks, a similar issue happened. The problematic task is block_pyramid. Using a similar script (see the bottom), it produces the following results:

The id 53 corresponds to: which should be a green region indicating the target.

Moreover, I though each id will correspond to a single object, but id 54 and 55 give:

This seems to be a systematic error rather than a bug for a single task. I'm thinking there may exist some bug in the .ttm file generation? Could you take a look at it?

The script I used to parse block_pyramid is:

python tools/dataset_generator.py --save_path ./debug --tasks block_pyramid --image_size '256,256' --episodes_per_task 1 --variations 1

import os
import cv2
import shutil
import numpy as np

from tqdm import tqdm
from glob import glob

def main():
    src_dir = './debug/block_pyramid/'
    tar_dir = './debug/parsed_block_pyramid/'

    variations = sorted(glob(f'{src_dir}/variation*/'))
    for v_i, variation in enumerate(tqdm(variations)):
        episodes = sorted(glob(f'{variation}/episodes/episode*/'))
        for e_i, episode in enumerate(episodes):
            image_path = f'{episode}/front_rgb/'
            images = glob(f'{image_path}/*.png')
            os.makedirs(f'{tar_dir}/image/v{v_i:03d}_e{e_i:03d}/', exist_ok=True)
            for image in images:
                im_id = int(image.split('/')[-1].split('.')[0])
                tar_path = f'{tar_dir}/image/v{v_i:03d}_e{e_i:03d}/{im_id:04d}.png'
                shutil.copy2(image, tar_path)
                mask = cv2.imread(image.replace('rgb', 'mask'), 0)
                obj_ids = list(np.unique(mask))
                for o_id, obj_id in enumerate(obj_ids):
                    im = cv2.imread(image)
                    masked_image = (mask == obj_id)[..., None] * im
                    os.makedirs(f'{tar_dir}/{v_i}_{e_i}/', exist_ok=True)
                    cv2.imwrite(f'{tar_dir}/{v_i}_{e_i}/{obj_id}_{im_id}.png', masked_image)

if __name__ == '__main__':
    main()

[stepjam]

Hi, There may be a bug. Could you first swap to loading demos through the Environment interface? In your script that you linked, you are nor decoding the masks correctly.

[HaozhiQi]

Hi,

Sorry, I didn't realize the mask is converted to 3 channel RGB image in the tools/dataset_generator.py so my parsing mask code is wrong. After changing it to single channel mask image, it works as expected.

Thanks a lot for your time!