Segmentation Fault for obtaining instance segmentation with particles (water)

[HSL31]

Hi, all.

I've been working with omnigibson and am trying to obtain instance segmentation with the particles within the environment. From the examples given, particle_applier_remover_demo.py I added camera modalities(seg_semantic, seg_instance) and tried to print each about the frame and the following was the error I encountered.

I am wondering if this is just an issue with lack of memory within my device or if I were to change something to validate the input of some kind. Thank you in advance.

[cgokmen]

@hang-yin can investigate this early next week. Thanks for the report!

[hang-yin]

Hi @HSL31, thanks for reporting this! I've tried to replicate your setup by incorporating the additional camera modalities (seg_semantic, seg_instance) into the viewer camera within the particle_applier_remover_demo.py and attempted to extract observations from it. However, I haven't been able to reproduce the seg fault. To better understand the problem, could you provide us with a bit more detail regarding the exact modifications you made to the demo? This includes where and how you added the camera modalities, and how you are retrieving the observations. Thanks again!

[HSL31]

Thank you for the reply.

With the given example, we chose the initial settings as followed.

modifier_type = "particleApplier"
particle_type = "water"
method_type = "Projection"

When initializing the viewer_camera, we named it cam

cam = og.sim.viewer_camera
cam.set_position_orientation(
        position=np.array([-1.61340969, -1.79803028,  2.53167412]),
        orientation=np.array([ 0.46291845, -0.12381886, -0.22679218,  0.84790371]),
    )

For faster debugging, we placed the modifier still in one position for a short period of time

deltas = [
        [5, np.array([0, 0, 0])],
    ]
 for t, delta in deltas:
     for i in range(t):
         modifier.set_position(modifier.get_position() + delta)
         env.step(np.array([]))

Lastly, we removed the modifier as our interest were to obtain just the segmentation information of the fluid and the table and added the modalities to the camera.

og.sim.remove_object(modifier)
cam.add_modality('seg_semantic')
cam.add_modality('seg_instance')

For visualization,

while True:
    env.step(np.array([]))
    cv2.imshow('seg', np.array(cam.get_obs()[0]["seg_instance"], dtype=np.uint8))
    cv2.waitKey(1)

other than the changes I've made above, everything else is the same as the given code. I will add the full code below.

import numpy as np
import cv2

import os
import sys
sys.path.append(r'/git/OmniGibson')
os.environ['KMP_DUPLICATE_LIB_OK']='True'

import omnigibson as og
from omnigibson.object_states import Covered
from omnigibson.objects import DatasetObject
from omnigibson.macros import gm, macros
from omnigibson.systems import get_system
from omnigibson.utils.usd_utils import create_joint
from omnigibson.utils.ui_utils import choose_from_options
from omnigibson.utils.constants import ParticleModifyMethod

# Set macros for this example
macros.object_states.particle_modifier.VISUAL_PARTICLES_REMOVAL_LIMIT = 1000
macros.object_states.particle_modifier.PHYSICAL_PARTICLES_REMOVAL_LIMIT = 8000
macros.object_states.particle_modifier.MAX_VISUAL_PARTICLES_APPLIED_PER_STEP = 4
macros.object_states.particle_modifier.MAX_PHYSICAL_PARTICLES_APPLIED_PER_STEP = 40
macros.object_states.covered.MAX_VISUAL_PARTICLES = 300

# Make sure object states and GPU dynamics are enabled (GPU dynamics needed for fluids)
gm.ENABLE_OBJECT_STATES = True
gm.USE_GPU_DYNAMICS = True
gm.ENABLE_HQ_RENDERING = True

def main(random_selection=False, headless=False, short_exec=False):

    modifier_type = "particleApplier"
    modification_metalink = {
        "particleApplier": "particleapplier_link",
        "particleRemover": "particleremover_link",
    }
    particle_type = "water" #antifreeze

    modification_method = {
        "Adjacency": ParticleModifyMethod.ADJACENCY,
        "Projection": ParticleModifyMethod.PROJECTION,
    }

    projection_mesh_params = {
        "Adjacency": None,
        "Projection": {
            "type": "Cone",
            "extents": np.array([0.1875, 0.1875, 0.375]),
        },
    }

    method_type = "Projection"

    abilities = {
        modifier_type: {
            "method": modification_method[method_type],
            "conditions": {
                particle_type: [],
            },
            "projection_mesh_params": projection_mesh_params[method_type],
        }
    }

    table_cfg = dict(
        type="DatasetObject",
        name="table",
        category="breakfast_table",
        model="kwmfdg",
        bounding_box=[3.402, 1.745, 1.175],
        position=[0, 0, 0.98],
    )

    cfg = {
        "scene": {
            "type": "Scene",
        },
        "objects": [table_cfg],
    }

    env = og.Environment(configs=cfg)
    og.sim.stop()

    cam = og.sim.viewer_camera
    cam.set_position_orientation(
        position=np.array([-1.61340969, -1.79803028,  2.53167412]),
        orientation=np.array([ 0.46291845, -0.12381886, -0.22679218,  0.84790371]),
    )

    # If we're using a projection volume, we manually add in the required metalink required in order to use the volume
    modifier = DatasetObject(
        name="modifier",
        category="dishtowel",
        model="dtfspn",
        bounding_box=[0.34245, 0.46798, 0.07],
        visual_only=method_type == "Projection",  # Non-fluid adjacency requires the object to have collision geoms active
        abilities=abilities,
    )

    modifier_root_link_path = f"{modifier.prim_path}/base_link"
    modifier._prim = modifier._load()
    if method_type == "Projection":
        metalink_path = f"{modifier.prim_path}/{modification_metalink[modifier_type]}"
        og.sim.stage.DefinePrim(metalink_path, "Xform")
        create_joint(
            prim_path=f"{modifier_root_link_path}/{modification_metalink[modifier_type]}_joint",
            body0=modifier_root_link_path,
            body1=metalink_path,
            joint_type="FixedJoint",
            enabled=True,
        )

    modifier._post_load()
    modifier._loaded = True
    og.sim.import_object(modifier)
    modifier.set_position(np.array([0, 0, 5.0]))

    og.sim.play()
    for _ in range(25):
        env.step(np.array([]))

    og.sim.enable_viewer_camera_teleoperation()

    #method_type == "Projection":
    z = 1.85

    modifier.keep_still()
    modifier.set_position_orientation(
        position=np.array([0, 0.3, z]),
        orientation=np.array([0, 0, 0, 1.0]),
    )

    deltas = [
        [5, np.array([0, 0, 0])],
    ]
    for t, delta in deltas:
        for i in range(t):
            modifier.set_position(modifier.get_position() + delta)
            env.step(np.array([]))

    og.sim.remove_object(modifier)
    cam.add_modality('seg_semantic')
    cam.add_modality('seg_instance')

    while True:
        env.step(np.array([]))
        cv2.imshow('seg', np.array(cam.get_obs()[0]["seg_instance"], dtype=np.uint8))
        cv2.waitKey(1)

    env.close()

if __name__ == "__main__":
    main()

Thank you once again.

[hang-yin]

Hi @HSL31 , thanks a lot for your repro script! This is a great catch and it might need a systematic fix on our end. I'll be working on fixing this tomorrow morning but to not keep you waiting, here's a hot fix for your problem:

1. move the add_modality calls to be right after env = og.Environment(configs=cfg)
2. after og.sim.remove_object(modifier), call og.sim.render() several times

These should help you get rid of the seg fault. One thing to note is that there might need to be more water particles for them to be visible in the segmentation maps. Please let me know if you run into any other problems!

[HSL31]

With the adjustments made to the code, I was able to avoid Segmentation Fault issues. However the particles or the fluid as a whole seems not to be appearing on the segmentation map no matter how much I spawn the particles.

I've only added the codes you suggested and changed the delta so the duration of projecting particles get longer.

Thank you.

[hang-yin]

This is really interesting. I'm working on a fix right now, but here's a summary of my current findings:

1. Particle systems segmentation doesn't work properly from the replicator's side when gm.ENABLE_HQ_RENDERING is set to True
2. Somehow, removing the particle applier makes recently spawned particles unlabelled in segmentation maps. Here are some screenshots to explain this:

If we remove the projection particle applier, only some particles show up as water (the yellow ones) in semantic segmentation, while the purple particles are marked as unlabelled:

Somehow, in this case (remove particle applier), instance segmentation shows up correctly before our internal remapping (where we remap micro physical particles according to semantic segmentation):

If we keep the particle applier, most particles show up as water (the green-ish ones):

While we look into this, @HSL31 you can checkout this branch for some intermediate progress on this: https://github.com/StanfordVL/OmniGibson/tree/fix/particles-seg-fault. You will need to set gm.ENABLE_HQ_RENDERING to False. If you do need to see the water particles in the segmentation maps, you can avoid removing the particle applier by moving it to somewhere else with modifier.set_position([x,y,z]). Alternatively, you could instead

1. start with the particle remover example without spawning the remover
2. use adjacency mode instead of projection mode for the applier

I've verified that both of these work.

[HSL31]

I think this will do for just about what we needed. Thank you!

[hang-yin]

@HSL31 Glad it worked! Please feel free to let me know if there's anything else we can help you with. I'll update you once I fix everything.