search

jkulhanek / robot-visual-navigation

Visual Navigation in Real-World Indoor Environments Using End-to-End Deep Reinforcement Learning Official Implementation
MIT License
58 stars 11 forks source link

[Problem] An error caused by inputting camera and target image to pytorch model in real world mode in ros folder #20

Closed jianffu closed 2 years ago

jianffu commented 2 years ago

We try to run the real world part of the open source code. While, an error occured when running the code described in the part of the [Using the trained model for the final navigation] in README file(we follow the instruction in your README.md in ~/Documents/AGV_visualnav_dev/robot-visual-navigation/ros/)

The launch file to run:

image

As we write in the launch, in order to run the goal visual navigation in the real world, we run 3 parts of the codes in the ros folder, which are as follows:

  1. main.py in ros/src/ros_agent_service folder
  2. main.py in ros/src/robot_controller folder
  3. set_goal.py in ros/src/goal_picker folder, whose argument is the goal image

The error information in the terminal output:

图片 12 
(visualnav) agv01@agv01:~/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/goalvisualnav_ros$ bash goalviusalnav.sh 
... logging to /home/agv01/.ros/log/a5a32776-291a-11ed-8ba0-02421e8cbe4b/roslaunch-agv01-32765.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://agv01:45733/

SUMMARY
========

PARAMETERS
 * /part2/image: /home/agv01/Docu...
 * /rosdistro: melodic
 * /rosversion: 1.14.13

NODES
  /part1/
    agent_service (ros_agent_service/main.py)
    controller (robot_controller/main.py)
  /part2/
    goal_pick (goal_picker/setgoal.py)

ROS_MASTER_URI=http://localhost:11311

process[part1/agent_service-1]: started with pid [321]
process[part1/controller-2]: started with pid [322]
process[part2/goal_pick-3]: started with pid [323]
[INFO] [1661943087.874406]: starting is sdu
Controller init success
Puller init success

…………………………..
…………………………..
…………………………..
[INFO] [1661943191.246993]: running
[INFO] [1661943191.253674]: waiting for target
/home/agv01/.local/lib/python3.6/site-packages/gym/core.py:26: UserWarning: WARN: Gym minimally supports python 3.6 as the python foundation not longer supports the version, please update your version to 3.7+
  "Gym minimally supports python 3.6 as the python foundation not longer supports the version, please update your version to 3.7+"
ros_agent_service: starting
model: turtlebot
/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/model.py:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Registering trainer turtlebot
Registering trainer turtlebot-noprior
Registering trainer turtlebot-unreal
Registering trainer turtlebot-unreal-noprior
Registering trainer turtlebot-a2c
Registering trainer turtlebot-a2c-noprior
Registering trainer dmhouse-a2c
Registering trainer dmhouse-unreal
Registering trainer dmhouse
Registering agent turtlebot
Registering agent turtlebot-noprior
Registering agent dmhouse
Registering agent dmhouse-a2c
Registering agent dmhouse-unreal
Registering agent turtlebot-a2c
Registering agent turtlebot-unreal
Registering agent turtlebot-a2c-noprior
Registering agent turtlebot-unreal-noprior
robot_visual_navigation/begin_compute_step: subscriber started
robot_visual_navigation/reset_state: subscriber started
Published
target got!
state has been cleared
[INFO] [1661943215.568396]: waiting for observation
[INFO] [1661943215.579759]: waiting for action
begin_compute_step: received request
observation to numpy
dtype_class <class 'numpy.uint8'>
dtype:  uint8
dtype:  uint8
/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/goalvisualnav_ros/src/visualnav_ros_agent_service/src/convert.py:78: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
  data = np.fromstring(msg.data, dtype=dtype).reshape(shape)
data.shape:  (360, 640, 3)
data.shape:  (360, 640, 3)
goal to numpy
dtype_class <class 'numpy.uint8'>
dtype:  uint8
dtype:  uint8
(360, 640, 3)
observation shape:  (360, 640, 3)
observation shape = (84, 84, 3)
observation size = 21168
dtype = uint8
goal shape = (84, 84, 3)
goal size = 21168
goal dtype = uint8

[ERROR] [1661934068.494090]: bad callback: <function create_send_response.<locals>.send_response at 0x7f5231a7b8>
Traceback (most recent call last):
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
    cb(msg)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/goalvisualnav_ros/src/visualnav_ros_agent_service/src/main.py", line 63, in send_response
    action = server.compute_step(request)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/goalvisualnav_ros/src/visualnav_ros_agent_service/src/main.py", line 52, in compute_step
    action = self.agent.act((observation, goal,))
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/agent.py", line 36, in act
    action = old_act(obs)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/trainer.py", line 353, in act
    action, self.states = self._step(obs, self.states)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/deep_rl/common/pytorch.py", line 62, in call
    to_tensor(kwargs, device))
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/trainer.py", line 342, in step
    policy_logits, _, states = model(observations, masks, states)
  File "/home/agv01/miniforge-pypy3/envs/visualnav/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/model.py", line 118, in forward
    features, states = self._forward_base(inputs, masks, states)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/model.py", line 149, in _forward_base
    return self.rnn(features, masks, states)
  File "/home/agv01/miniforge-pypy3/envs/visualnav/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/deep_rl/model/module.py", line 38, in forward
    return forward_masked_rnn_transposed(inputs, masks, states, self.inner)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/deep_rl/common/pytorch.py", line 221, in forward_masked_rnn_transposed
    outputs, states = forward_masked_rnn(inputs, masks, states, forward_rnn)
  File "/home/agv01/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/deep_rl/common/pytorch.py", line 113, in forward_masked_rnn
    mask_states(states, masks[:, start_idx])
  File "/home/agv01/miniforge-pypy3/envs/visualnav/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/agv01/miniforge-pypy3/envs/visualnav/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 689, in forward
    self.check_forward_args(input, hx, batch_sizes)
  File "/home/agv01/miniforge-pypy3/envs/visualnav/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 632, in check_forward_args
    self.check_input(input, batch_sizes)
  File "/home/agv01/miniforge-pypy3/envs/visualnav/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 207, in check_input
    self.input_size, input.size(-1)))
RuntimeError: input.size(-1) must be equal to input_size. Expected 518, got 517

[turtlesim2/goal_pick-3] process has finished cleanly
log file: /home/agv01/.ros/log/b475da3a-2903-11ed-8885-02421e8cbe4b/turtlesim2-goal_pick-3*.log
^C[turtlesim1/controller-2] killing on exit
[turtlesim1/agent_service-1] killing on exit
[INFO] [1661935536.123854]: received action None
action:  None
shutting down processing monitor...
... shutting down processing monitor complete
Done

So the error is 

[ERROR] [1661934068.494090]: bad callback: <function create_send_response.<locals>.send_response at 0x7f5231a7b8>
......
......
Traceback (most recent call last):`RuntimeError: input.size(-1) must be equal to input_size. Expected 518, got 517

Error Analysis: The target image and observation image saved in the function compute_step during running in ~/Documents/AGV_visualnav_dev/robot-visual-navigation/ros/src/ros_agent_service/src/main.py, so we guess the pictures are suitable with the codes and there are no business between the pictures and the error. image

Potentially relevant error codes located in the model.py (~/Documents/AGV_visualnav_dev/robot-goal-visual-navigation/python/model.py)

image

Running environment:

Hardware description: use Nvidia Jetson AGX Xavier and an AGV to run the code in ros folder

Software description: Ubuntu 18.04 Use miniforge(conda) to create python3.6 environment (/home/agv01/miniforge-pypy3/envs/visualnav/bin/python3.6) Pytorch version 1.10

image

The illustration of the model in the paper:

image

The question is whether the error is caused by our data format in captured image or the RNN / LSTM model code in pytorch provided in ros folder ? or, in order to run the real world part, are the codes we run correct ?

Thanks very much.

jkulhanek commented 2 years ago

I am sorry that you are experiencing the issue.

I believe the problem is in the number of actions for the environment.

Can you change the following line: https://github.com/jkulhanek/robot-visual-navigation/blob/6c1107ef8751b54e2b6b2241fb668eddacc4532e/python/agent.py#L55 Such that the second argument is True? I.e. replace the condition with True?

Let me know if it helped.

jianffu commented 2 years ago

Thanks for your timely reply! And, following your instruction by changing the corresponding code line, the error is solved. Then, the output becomes correct and the terminal has no error. The terminal outputs:

[INFO][XXXXXX]: received action 1
action: 1

I think we are not familiar with the code enough. Thank you for your sincere help！

jkulhanek commented 2 years ago

Thank you!

Fixed in commit 58403e7c6277c9828272a42a98116c397cce7cc8