search

CMU-TBD / SocNavBench

A Grounded Simulation Testing Framework for Evaluating Social Navigation: https://arxiv.org/abs/2103.00047
MIT License
34 stars 6 forks source link

ValueError: the array phi contains no zero contour (no zero level set) #3

Closed AshwiniUthir closed 3 years ago

AshwiniUthir commented 3 years ago

I followed the Installation and set up SocNavBench. I am trying to test the socnav simulator and I am running into below error:

$PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/test_socnav.py Waiting for Joystick connection... Robot <-- Joystick (receiver) connection established Robot --> Joystick (sender) connection established

Starting episode "test_socnav_univ" in building "Univ"

Simulator running in synchronous mode, dt=0.050s Render mode: Schematic view (TOPVIEW only) Generated Auto Humans: 0 Generating pedestrians from "univ" in range [0, 100]: 47 Traceback (most recent call last): File "tests/test_socnav.py", line 150, in test_socnav() File "tests/test_socnav.py", line 138, in test_socnav simulator.add_agent(robot_agent) File "/home/ashwini/Documents/SocNavBench/simulators/simulator_helper.py", line 61, in add_agent keep_episode_running=self.params.keep_episode_running) File "/home/ashwini/Documents/SocNavBench/agents/robot_agent.py", line 37, in simulation_init keep_episode_running=keep_episode_running) File "/home/ashwini/Documents/SocNavBench/agents/agent.py", line 43, in simulation_init Agent._init_fmm_map(self) File "/home/ashwini/Documents/SocNavBench/agents/agent.py", line 256, in _init_fmm_map mask_grid_mn=obstacle_occupancy_grid) File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 122, in create_fmm_map_based_on_goal_position mask_grid_mn=mask_grid_mn) File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 42, in init self.compute_fmm_distance_and_angle() File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 58, in compute_fmm_distance_and_angle phi, dx=self.fmm_distance_map.map_scale * np.ones(2)) File "/home/ashwini/anaconda3/envs/socnavbench/lib/python3.6/site-packages/skfmm/pfmm.py", line 111, in distance int(self_test), DISTANCE, order, narrow, periodic) ValueError: the array phi contains no zero contour (no zero level set)

Ubuntu: 16.04.7 LTS; Python version: 3.6.9 Can someone help me resolve this?

ajdroid commented 3 years ago

Hi, can you try PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/test_episodes.py instead of PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/test_socnav.py?

AshwiniUthir commented 3 years ago

Hi, Thank you. I tried PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/test_episodes.py and got the same error. I am also running into the error --> /SocNavBench/metrics/metrics_sim_utils.py:247: RuntimeWarning: divide by zero encountered in true_divide ttc_all = botped_distances / botped_component

$ PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/test_episodes.py Waiting for Joystick connection... Robot <-- Joystick (receiver) connection established Robot --> Joystick (sender) connection established

Starting episode "t_univ1" in building "Univ"

Simulator running in synchronous mode, dt=0.050s Render mode: Schematic view (TOPVIEW only) Generating Auto Humans: 0 Generating pedestrians from "univ" in range [0, -1]: 39 Running simulation on 39 agents Sending episode data to joystick... Robot powering on A: 39 Success: 28 Collide: 0 Time: 11 Frames: 500 T = 25.000 Simulation completed in 19.54390001296997 real world seconds

Robot powering off, received 500 commands Robot termination cause: Pedestrian Collision Successfully wrote episode log to episode_log.txt The metric personal_space_cost is not implemented yet /home/ashwini/Documents/SocNavBench/metrics/metrics_sim_utils.py:247: RuntimeWarning: divide by zero encountered in true_divide ttc_all = botped_distances / botped_component Successfully wrote episode metrics to /home/ashwini/Documents/SocNavBench/tests/socnav/test_SamplingPlanner/t_univ1/episode_score_t_univ1.pkl Rendering movie with fps=10 Started processes: 251 out of 251, 100.000% Finished processes: 251 out of 251, 100.000% Movie progress: 251 out of 251, 100.000% Rendered gif at /home/ashwini/Documents/SocNavBench/tests/socnav/test_SamplingPlanner/t_univ1/movie_t_univ1.gif Cleaned directory

Starting episode "t_univ2" in building "Univ"

Simulator running in synchronous mode, dt=0.050s Render mode: Schematic view (TOPVIEW only) Generating Auto Humans: 0 Traceback (most recent call last): File "tests/test_episodes.py", line 135, in test_episodes() File "tests/test_episodes.py", line 111, in test_episodes generate_robot(episode.robot_start_goal, simulator) File "tests/test_episodes.py", line 55, in generate_robot simulator.add_agent(robot_agent) File "/home/ashwini/Documents/SocNavBench/simulators/simulator_helper.py", line 61, in add_agent keep_episode_running=self.params.keep_episode_running) File "/home/ashwini/Documents/SocNavBench/agents/robot_agent.py", line 37, in simulation_init keep_episode_running=keep_episode_running) File "/home/ashwini/Documents/SocNavBench/agents/agent.py", line 43, in simulation_init Agent._init_fmm_map(self) File "/home/ashwini/Documents/SocNavBench/agents/agent.py", line 256, in _init_fmm_map mask_grid_mn=obstacle_occupancy_grid) File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 122, in create_fmm_map_based_on_goal_position mask_grid_mn=mask_grid_mn) File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 42, in init self.compute_fmm_distance_and_angle() File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 58, in compute_fmm_distance_and_angle phi, dx=self.fmm_distance_map.map_scale * np.ones(2)) File "/home/ashwini/anaconda3/envs/socnavbench/lib/python3.6/site-packages/skfmm/pfmm.py", line 111, in distance int(self_test), DISTANCE, order, narrow, periodic) ValueError: the array phi contains no zero contour (no zero level set)

AshwiniUthir commented 3 years ago

@ajdroid Is there any other tests I could try?

ajdroid commented 3 years ago

Hi, we're trying to reproduce the first bug that you posted. Can you try downloading a fresh set of meshes from: https://drive.google.com/drive/u/1/folders/1sNFi0fiqSRy26U_JT4sfM2Co2hDbedvQ and then re running tests/test_episodes.py. You can place these within /home/username/SocNavBench/sd3dis/stanford_building_parser_dataset/mesh. We suspect that there may have been a problem with the map definitions themselves.

I found the bug for /SocNavBench/metrics/metrics_sim_utils.py:247: RuntimeWarning: divide by zero encountered in true_divide that you posted and will be sending a patch soon.

AshwiniUthir commented 3 years ago

Hi, I downloaded the meshes from the drive and replaced them, but I am still running into the same error. Am I missing any steps?

$ PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/test_episodes.py Waiting for Joystick connection... Robot <-- Joystick (receiver) connection established Robot --> Joystick (sender) connection established

Starting episode "t_univ1" in building "Univ"

Simulator running in synchronous mode, dt=0.050s Render mode: Schematic view (TOPVIEW only) Generating Auto Humans: 0 Generating pedestrians from "univ" in range [0, -1]: 39 Running simulation on 39 agents Sending episode data to joystick... Robot powering on A: 39 Success: 28 Collide: 0 Time: 11 Frames: 500 T = 25.000 Simulation completed in 19.6157968044281 real world seconds

Robot powering off, received 500 commands Robot termination cause: Pedestrian Collision Successfully wrote episode log to episode_log.txt The metric personal_space_cost is not implemented yet /home/ashwini/Documents/SocNavBench/metrics/metrics_sim_utils.py:247: RuntimeWarning: divide by zero encountered in true_divide ttc_all = botped_distances / botped_component Successfully wrote episode metrics to /home/ashwini/Documents/SocNavBench/tests/socnav/test_SamplingPlanner/t_univ1/episode_score_t_univ1.pkl Rendering movie with fps=10 Started processes: 251 out of 251, 100.000% Finished processes: 251 out of 251, 100.000% Movie progress: 251 out of 251, 100.000% Rendered gif at /home/ashwini/Documents/SocNavBench/tests/socnav/test_SamplingPlanner/t_univ1/movie_t_univ1.gif Cleaned directory

Starting episode "t_univ2" in building "Univ"

Simulator running in synchronous mode, dt=0.050s Render mode: Schematic view (TOPVIEW only) Generating Auto Humans: 0 Traceback (most recent call last): File "tests/test_episodes.py", line 135, in test_episodes() File "tests/test_episodes.py", line 111, in test_episodes generate_robot(episode.robot_start_goal, simulator) File "tests/test_episodes.py", line 55, in generate_robot simulator.add_agent(robot_agent) File "/home/ashwini/Documents/SocNavBench/simulators/simulator_helper.py", line 61, in add_agent keep_episode_running=self.params.keep_episode_running) File "/home/ashwini/Documents/SocNavBench/agents/robot_agent.py", line 37, in simulation_init keep_episode_running=keep_episode_running) File "/home/ashwini/Documents/SocNavBench/agents/agent.py", line 43, in simulation_init Agent._init_fmm_map(self) File "/home/ashwini/Documents/SocNavBench/agents/agent.py", line 256, in _init_fmm_map mask_grid_mn=obstacle_occupancy_grid) File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 122, in create_fmm_map_based_on_goal_position mask_grid_mn=mask_grid_mn) File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 42, in init self.compute_fmm_distance_and_angle() File "/home/ashwini/Documents/SocNavBench/utils/fmm_map.py", line 58, in compute_fmm_distance_and_angle phi, dx=self.fmm_distance_map.map_scale * np.ones(2)) File "/home/ashwini/anaconda3/envs/socnavbench/lib/python3.6/site-packages/skfmm/pfmm.py", line 111, in distance int(self_test), DISTANCE, order, narrow, periodic) ValueError: the array phi contains no zero contour (no zero level set)

ajdroid commented 3 years ago

Can you tell us if the unit tests succeeded? 

# Ensure the unit tests succeed with 
python3 tests/all_unit_tests.py
AshwiniUthir commented 3 years ago

@ajdroid I am extremely sorry! I overlooked the unit tests and I missed executing them. Below are the test results for the unit tests. 10 test cases passed, 4 test cases failed.

The following are the test cases that passed: test_coordinate_transform() test_cost_function() test_cost() test_dynamics() test_fmm_map() test_goal_psc() test_image_space_grid() test_lqr() test_spline() test_voxel_interpolation()

The following are the test cases that failed: test_goal_angle() test_goal_distance() test_obstacle_map() test_obstacle_objective()

The following is the error message I get for the test case: test_goal_angle()

$ PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/all_unit_tests.py
Coordinate transform tests passed!
Cost function tests passed!
Quad costs tests passed!
Dynamics tests passed!
Fmm_map tests passed!
Traceback (most recent call last):
  File "tests/all_unit_tests.py", line 23, in <module>
    test_goal_angle()
  File "/home/ashwini/Documents/SocNavBench/tests/unit_tests/test_goal_angle_objective.py", line 75, in main_test
    test_goal_angle_distance()
  File "/home/ashwini/Documents/SocNavBench/tests/unit_tests/test_goal_angle_objective.py", line 71, in test_goal_angle_distance
    p.test_goal_ang_obj_ans, atol=1e-2)
AssertionError

The following is the error message I get for the test case: test_goal_distance()

Traceback (most recent call last):
  File "tests/all_unit_tests.py", line 24, in <module>
    test_goal_distance()
  File "/home/ashwini/Documents/SocNavBench/tests/unit_tests/test_goal_distance_objective.py", line 81, in main_test
    test_goal_distance()
  File "/home/ashwini/Documents/SocNavBench/tests/unit_tests/test_goal_distance_objective.py", line 77, in test_goal_distance
    assert numerical_error <= .01
AssertionError

The following is the error message I get for the test case: test_obstacle_objective()

Traceback (most recent call last):
  File "tests/all_unit_tests.py", line 29, in <module>
    test_obstacle_objective()
  File "/home/ashwini/Documents/SocNavBench/tests/unit_tests/test_obstacle_objective.py", line 106, in main_test
    test_avoid_obstacle(False)
  File "/home/ashwini/Documents/SocNavBench/tests/unit_tests/test_obstacle_objective.py", line 73, in test_avoid_obstacle
    assert np.allclose(objective_values_13[0], p.test_obs_obj_ans, atol=1e-4)
AssertionError
ajdroid commented 3 years ago

Hi, did you change the map in params/user_params.ini from openspace to something else to run these tests? I think we neglected to include that map in our uploads. These tests are failing since we have hardcoded the correct answers for those tests into their respective files but since changing the map changes what the correct answers would be, we get these assertion errors. We will send a patch to fix this as well.

However, we are unable to reproduce the bug you have with test_episodes.py. Can you provide more details on the version of python you are using (e.g. is it with conda or no)? Did the patches to pyassimp and OpenGL mentioned in the Installation procedure work for you?

AshwiniUthir commented 3 years ago

@ajdroid I did not make any changes in params/user_params.ini

Regarding the bug with test_episodes.py, my version of python is 3.6.9 and my Ubuntu version: 16.04.7 LTS.

Please let me know what is the preferred python and Ubuntu version so that I can replicate your exact setup.

Regarding the patches to pyassimp and OpenGL, it worked fine. I did not get any errors on that. However, I now see that you have pushed some changes, I will retry the updated code base.

ajdroid commented 3 years ago

Hi, can you try a fresh install with the updated base and then tell us how the unit tests go?

AshwiniUthir commented 3 years ago

@ajdroid As you suggested, I tried a fresh install with the updated code and below are the results of the unit tests. 

$ PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/all_unit_tests.py
Coordinate transform tests passed!
Cost function tests passed!
Quad costs tests passed!
Dynamics tests passed!
Fmm_map tests passed!
Goal-angle objective tests passed!
Goal-distance tests passed!
Generating control pipeline, this may take some time...
Goal-psc tests passed!
Image space grid tests passed!
Lqr tests passed!
Obstacle map tests passed!
Obstacle objective tests passed!
Spline tests passed!
Voxel interpolation tests passed!

All tests passed!

Below are the results from running test_socnav.py

$ PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 tests/test_socnav.py
Waiting for Joystick connection...
Robot <-- Joystick (receiver) connection established
Robot --> Joystick (sender) connection established

Starting episode "test_socnav_univ" in building "Univ"

Simulator running in synchronous mode, dt=0.050s
Render mode: Schematic view (TOPVIEW only)
Generating auto humans: 0
Generating recorded humans from "univ" in range [0, 100]: 47
Running simulation on 47 agents
Sending episode data to joystick...
Robot powering on
A: 47 Success: 16 Collide: 3 Time: 31 Frames: 329 T = 16.450 
Robot powering off, received 330 commands
A: 47 Success: 16 Collide: 3 Time: 31 Frames: 330 T = 16.500 
Simulation completed in 13.3063 real world seconds
Robot termination cause: Success
Successfully wrote episode log to episode_log.txt
Successfully wrote episode metrics to /home/ashwini/Documents/SocNavBench/tests/socnav/test_SamplingPlanner/test_socnav_univ/episode_score_test_socnav_univ.pkl
Rendering movie with fps=20
Rendered frames: 331 out of 331, 100.000%
Finished rendering all frames
rendering took 21.51880s
Movie progress: 331 out of 331, 100.000% 
Rendered gif at /home/ashwini/Documents/SocNavBench/tests/socnav/test_SamplingPlanner/test_socnav_univ/movie_test_socnav_univ.gif 
Cleaned directory 
$ PYOPENGL_PLATFORM=egl PYTHONPATH='.' python3 joystick/joystick_client.py
Joystick running SamplingPlanner algorithm
Joystick using positional dynamics
Initiated joystick locally (AF_UNIX) at "/tmp/socnavbench_joystick_recv" & "/tmp/socnavbench_joystick_send"
Robot <-- Joystick (sender) connection established
Robot --> Joystick (receiver) connection established
Received episodes: ['test_socnav_univ']
Waiting for episode: test_socnav_univ
Running test for test_socnav_univ
Updated state of the world for time = 16.000 out of 30.000
powering off joystick, robot terminated with: Success
Finished episode: test_socnav_univ 
Finished all episodes

I ran test_episodes.py as well and I don't see any errors. Thank you so much for the bug fixes. I don't however see any simulator window/ visualization window pop up at this time. Is that an expected behavior?

Am I all set to move on to USAGE.md? Please let me know.

ajdroid commented 3 years ago

Yes, you should be good to move on. Are you trying to use SocNavBench to evaluate your own social navigation algorithm?

Also, you won't see a real time simulator window but gifs of each episode should be rendered once they finish, such as:

Rendered gif at /home/ashwini/Documents/SocNavBench/tests/socnav/test_SamplingPlanner/test_socnav_univ/movie_test_socnav_univ.gif

AshwiniUthir commented 3 years ago

@ajdroid Yes, I am trying to use SocNavBench to evaluate my social navigation algorithm.

ajdroid commented 3 years ago

Great, can you tell us a little bit about your algorithm? What kind of inputs does it take (RGBD images/pedestrian positions/costmaps)? Does it have a local and global planner built-in etc

AshwiniUthir commented 3 years ago

@ajdroid Target location and LiDAR scans are given as input to the neutral network. Speed and direction are the outputs. There are no built-in planners. The neural network is trained using pedestrians trajectory from the ETH dataset.