Semantic Segmentation Lane Markings not Showing Up

[prestonbarnett]

Issue Description

I recently started working with Beamng.tech and BeamNGpy for an academic demonstration with behavior cloning and autonomous vehicles. I started to implement the Camera sensor and was wondering if it's possible to extract lane markings from the semantic segmentation output. I noticed that lane annotation colors are present in the JSON file (e.g., SOLID_LINE, etc), but I can't seem to get them to appear on the output of the Camera sensor.

It appears, however, that lane markings are visible in the demonstration in the README. I'm not sure if this is due to a setting or parameter I'm missing. I'd appreciate any help in getting the lane markings to appear as it'll help in training my model on a simplified view of the world for testing purposes.

Versions

Beamng.tech: 0.31 BeamNGpy: 1.28 OS: Windows 10 Enterprise

[alessiogambi]

@prestonbarnett I am also interested in this issue.

[alessiogambi]

@aivora-beamng I also have the problem that procedurally generated roads show up just black. Lane marking also shows up just black (or they do not show up at all). See picture:

Could it be that the material I use is not valid? FYI: I create roads using the track_editor_C_center material and the lane markings using: line_yellow, line_dashed_short_yellow, line_white, and line_dashed_short

See picture:

[alessiogambi]

@aivora-beamng Is it possible that Roads objects do not have a way to set the ObjectAnnotation field that one can see from the World Editor? Sadly, I cannot find any existing level where this attribute is set.

[alessiogambi]

Quick update: I patched prefab.json and hardcoded the STREET annotation to all the (Decal) Roads. I can see in the World editor that the label/annotation is the correct one (the color is RED); unfortunately, the annotation and instance cameras always produce the wrong image (see post above).

[alessiogambi]

If I put a vehicle in front of the camera I got the following result:

The road is still Black, but the vehicle is visible.

[alessiogambi]

Since we at it, running the code https://github.com/BeamNG/BeamNGpy/blob/master/examples/annotation_bounding_boxes.ipynb produces the expected result (cannot say anything about the lane markings, tho)

[alessiogambi]

I executed the same code on the Italy map using scenario.add_vehicle(ego, pos=(91, 53, 39), rot_quat=(0.0173, -0.0019, -0.6354, 0.7720)) to place the vehicle. The grey area that corresponds to the asphalt is there, but no lane marking. See picture.

One confusing thing is that the World editor says that none of the corresponding roads have any annotation attached to it...

[alessiogambi]

I ended up recreating the level and importing all the materials from west_coast_usa. I checked that all the materials have proper annotation. For example, road_asphalt_light declares "annotation": "STREET". However, now the situation seems worse than before. Not sure what I can do more than this... Colored camera:

Semantic Camera:

[alessiogambi]

@aivora-beamng A quick update. I tried the code using the template_tech level (with and without road_insible and after removing the annoying grass field). I changed the zorder and also manually set the annotations in the world editor, but the result is now that instead of an all green image (like the one above) I got an all black image.

[aivora-beamng]

Hi @alessiogambi, sorry for the confusion, the template_tech level is not the .tech level which is designed to work with the .tech tools. It is called Tech Ground (tech_ground), and it will be included as part of the 0.32 BeamNG.tech release. I tested it with the new Road Architect tool designed to create roads, and the annotations are behaving as expected, including the lane markings (we fixed this one for the upcoming release).

Sorry for the troubles caused, I think it will be best to wait for the release happening soon.

[aivora-beamng]

Hi @prestonbarnett, the lane markings are now available in BeamNG.tech 0.32 and BeamNGpy v1.29.

@alessiogambi Try to use the new Tech Ground level released with BeamNG.tech 0.32 today, the annotations should be working much better than on smallgrid.

[alessiogambi]

@aivora-beamng As far as I can see, the annotations are working now. (see pictures below).

Note: I updated the tech_ground level because the material I needed (from the west_coast) where missing.

[AbdelrahmanElsaidElsawy]

If is issue is resolved, we can close this ticket. Thank you for the feedback @alessiogambi @prestonbarnett