[Bug] Raycaster gives different results in settings that only differ by height

[benaslater]

I was working with two configs which start the agent in settings that are identical (I checked the diff between the two arena definitions and it does look like it's only a change in height), but in one is raised higher above the ground than the other. There is a moveable block built into the wall that the raycaster can detect when it is in the lower height area but cannot when it is in the upper height arena.

I've done my best to confirm this isn't caused by a mistake on calculating the heights on my part.

Impact: Low (just creating this for tracking purposes)

Workaround: Increasing the height of the block to 0.75 allows the raycaster to pick up the location of the block

Observed Output: The initial raycast observation (using the raycastparser) Lower arena:

ARENA : [0. 0. 0. 0. 0.]
IMMOVABLE : [0.01876533 0.00577801 0.01009248 0.         0.        ]
MOVABLE : [0.         0.         0.         0.00577098 0.        ]
GOODGOAL : [0.         0.         0.         0.         0.01308224]
GOODGOALMULTI : [0. 0. 0. 0. 0.]
BADGOAL : [0. 0. 0. 0. 0.]

Higher Arena:

ARENA : [0. 0. 0. 0. 0.]
IMMOVABLE : [0.01876549 0.00577878 0.01009248 0.00577438 0.0187655 ]
MOVABLE : [0. 0. 0. 0. 0.]
GOODGOAL : [0. 0. 0. 0. 0.]
GOODGOALMULTI : [0. 0. 0. 0. 0.]
BADGOAL : [0. 0. 0. 0. 0.]

Expected Output: The value for MOVABLE is the same, given the movable block is in the same position relative to the agent in both scenarios

Steps to reproduce: Run the provided config with raycasts. (I'm not sure how other people tend to test configs for raycast but I've hacked together a solution that lets you play while seeing the output - I could tidy it up an share it if it would be useful)

Config: (sorry for some reason github doesn't support uploading YAML)

!ArenaConfig
arenas:
  # Lower height arena: Raycaster can detect the cardbox
  0: !Arena
    pass_mark: 0
    t: 250
    items:
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 20, y: 0, z: 21.5}
      sizes:
      - !Vector3 {x: 0.5, y: 2, z: 37}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 20, y: 0, z: 2.5}
      sizes:
      - !Vector3 {x: 1, y: 0.5, z: 1}
      rotations: [0]
      colors:
      - {r: 0, g: 0, b: 255}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 20, y: 0, z: 1}
      sizes:
      - !Vector3 {x: 3, y: 0.5, z: 2}
      rotations: [0]
      colors:
      - {r: 0, g: 0, b: 255}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 18, y: 0, z: 1}
      sizes:
      - !Vector3 {x: 1, y: 1, z: 2}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 18.625, y: 0, z: 4.5}
      sizes:
      - !Vector3 {x: 2.25, y: 2, z: 0.5}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 21.375, y: 0, z: 4.5}
      sizes:
      - !Vector3 {x: 2.25, y: 2, z: 0.5}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 17.25, y: 0, z: 3.125}
      sizes:
      - !Vector3 {x: 0.5, y: 2, z: 2.25}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22.75, y: 0, z: 3.125}
      sizes:
      - !Vector3 {x: 0.5, y: 2, z: 2.25}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
# ------------------------Raycaster adjustments---------------------------------------
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22, y: 0, z: 0.5}
      sizes:
      - !Vector3 {x: 1, y: 1, z: 1}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22, y: 0, z: 1.5}
      sizes:
      - !Vector3 {x: 1, y: 0.5, z: 1}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 23, y: 0, z: 1.5}
      sizes:
      - !Vector3 {x: 1, y: 1, z: 1}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Cardbox2
      positions:
      - !Vector3 {x: 22, y: 0.5, z: 1.5}
      sizes:
      - !Vector3 {x: 1, y: 0.5, z: 1}
      rotations: [0]
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22, y: 1.001, z: 1}
      sizes:
      - !Vector3 {x: 1, y: 0.5, z: 2}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22, y: 1.501, z: 0.5}
      sizes:
      - !Vector3 {x: 1, y: 0.5, z: 1}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
# ----------------------------------------------------------------------
    - !Item
      name: Agent
      positions:
      - !Vector3 {x: 20, y: 0.5, z: 0.5}
      rotations: [0]
    - !Item
      name: GoodGoal
      positions:
      - !Vector3 {x: 21.5, y: 0, z: 3.6}
      sizes:
      - !Vector3 {x: 1, y: 1, z: 1}
  # Higher height arena: Raycaster can detect the cardbox
  1: !Arena
    pass_mark: 0
    t: 250
    items:
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 20, y: 0, z: 21.5}
      sizes:
      - !Vector3 {x: 0.5, y: 5.5, z: 37}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 20, y: 0, z: 2.5}
      sizes:
      - !Vector3 {x: 1, y: 4, z: 1}
      rotations: [0]
      colors:
      - {r: 0, g: 0, b: 255}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 20, y: 0, z: 1}
      sizes:
      - !Vector3 {x: 3, y: 4, z: 2}
      rotations: [0]
      colors:
      - {r: 0, g: 0, b: 255}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 18, y: 0, z: 1}
      sizes:
      - !Vector3 {x: 1, y: 4.5, z: 2}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 18.625, y: 0, z: 4.5}
      sizes:
      - !Vector3 {x: 2.25, y: 5.5, z: 0.5}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 21.375, y: 0, z: 4.5}
      sizes:
      - !Vector3 {x: 2.25, y: 5.5, z: 0.5}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 17.25, y: 0, z: 3.125}
      sizes:
      - !Vector3 {x: 0.5, y: 5.5, z: 2.25}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22.75, y: 0, z: 3.125}
      sizes:
      - !Vector3 {x: 0.5, y: 5.5, z: 2.25}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
# ------------------------Raycaster adjustments---------------------------------------
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22, y: 0, z: 0.5}
      sizes:
      - !Vector3 {x: 1, y: 4.5, z: 1}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22, y: 0, z: 1.5}
      sizes:
      - !Vector3 {x: 1, y: 4, z: 1}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 23, y: 0, z: 1.5}
      sizes:
      - !Vector3 {x: 1, y: 4.5, z: 1}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Cardbox2
      positions:
      - !Vector3 {x: 22, y: 4, z: 1.5}
      sizes:
      - !Vector3 {x: 1, y: 0.5, z: 1}
      rotations: [0]
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22, y: 4.501, z: 1}
      sizes:
      - !Vector3 {x: 1, y: 0.5, z: 2}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
    - !Item
      name: Wall
      positions:
      - !Vector3 {x: 22, y: 5.001, z: 0.5}
      sizes:
      - !Vector3 {x: 1, y: 0.5, z: 1}
      rotations: [0]
      colors:
      - {r: 153, g: 153, b: 153}
# ----------------------------------------------------------------------
    - !Item
      name: Agent
      positions:
      - !Vector3 {x: 20, y: 4, z: 0.5}
      rotations: [0]
    - !Item
      name: GoodGoal
      positions:
      - !Vector3 {x: 21.5, y: 0, z: 3.6}
      sizes:
      - !Vector3 {x: 1, y: 1, z: 1}

[alhasacademy96]

Dear @benaslater, I can confirm that your issue has been noted and will be looked at sometime this week in more detail.

P.S. You can use this block to format YAML code:

!ArenaConfig
randomizeArenas: false
showNotification: false 
canResetEpisode: true 
canChangePerspective: true 
arenas:
  0: !Arena
    pass_mark: 0
    t: 250
    items:
    - !Item
      name: Agent
      positions:
      - !Vector3 {x: 20, y: 0, z: 20}
      rotations: [0]

[github-actions[bot]]

This issue is stale because it has been open for 45 days with no activity.

[alhasacademy96]

The issue with raycasting (in Unity,) where the movable block is detected in a lower height area but not in an upper height arena, can likely be attributed to the configuration of the raycasting settings and the spatial relation of the objects within your scene. Here's a detailed, yet simplified explanation of the situation based on the provided information and the script overvieq:

Raycasting Configuration: The script defines a ray perception sensor framework used for detecting objects in a 3D environment. This sensor can perform either 2D or 3D raycasting, based on the RayPerceptionCastType setting. It uses parameters like ray length, angles, offsets, and the radius for sphere casting (if used) to determine how and where rays are cast into the environment.

Ray Perception Input and Detection: For each ray, the script calculates start and end positions based on the provided angles and other parameters. It then performs raycasting from these start points towards the end points, checking for collisions with objects that have specified tags. The detection of objects is also influenced by the height offsets and the length of the rays, which are adjusted based on the agent's scale.

Issue with Height: Given that in one configuration the movable block is detectable and in another (with only a height difference) it is not, suggests that the issue might be related to how the ray start and end positions are being calculated relative to the agent's height. Since the rays' start and end positions depend on the agent's position and orientation (including height), a change in the agent's height without a corresponding adjustment in ray configuration might result in rays being cast above or below the intended targets.

Observations and Workaround: Your observation and workaround further suggest that the raycasting setup is sensitive to the relative heights of objects within the scene. By increasing the height of the block, you essentially brought it back into the path of the rays being cast, which implies that the rays' vertical range was insufficient to detect the block at its original height in the higher arena setting.

I've checked the docs on ml-agents and it's possible that the ray height offsets could be the issue here as they are currently defined as 0 respectively in the unity configuration (RayPerceptionSensor, which is a 3D ray configuration). It could be simple as increasing these offsets by +1. Another solution I gathered is that an adjustment to the ray length and angles might help.

REF: docs

[github-actions[bot]]

This issue is stale because it has been open for 45 days with no activity.

[alhasacademy96]

Closing as stale. @benaslater You should reopen if error/bug is persistent or needs to be looked into more detail.