search

mit-aera / FlightGoggles

A framework for photorealistic hardware-in-the-loop agile flight simulation using Unity3D and ROS. Developed by MIT AERA group.
http://flightgoggles.mit.edu
Other
402 stars 99 forks source link

Streaming images from AWS to local machine #37

Closed PHRABAL closed 5 years ago

PHRABAL commented 5 years ago

If we're running FG on a ec2 p3 server and want to render on a local machine, what are the minimum requirements of that local machine? Are they the same as stated when you run FG locally? GPU and 32GB RAM?

varunmurali1 commented 5 years ago

I am not quite sure I understand. If FG is running on aws, you shouldn't need a lot of resources on a local machine since the rendering is happening on aws. We do not specify a minimum requirement for local run, the specified GPU and RAM is the specifications of the machine we used to test and are not a minimum requirement. For local rendering, if you have a GPU with at least ~4 Gb VRAM, I would try the codebase. It might be slow, but since all the reporting is done in sim time this should not be a problem for experimental evaluation.

PHRABAL commented 5 years ago

That’s what I originally thought as well. Local machine hw is not important. I installed ROS Desktop in an Ubuntu virtual env on my Mac. Connection to ec2 p3 works. In rqt I see the time running, but no visualization, only a gray grid. Would my local machine at least need some type of GPU to render locally or would a CPU only Ubuntu setup with 16GB RAM be sufficient?

Winter-Guerra commented 5 years ago

Hi @PHRABAL, I would need some more information to diagnose the issue. What happens if you run the following commands on the AWS server?

If these commands succeed, then FlightGoggles is rendering correctly in the AWS instance. It would narrow down the issue to some sort of AWS/ROS connection issues. Please send the results of the following commands on your local machine:

If the second batch of commands do not work, then your AWS security configs are probably to blame. Or, you could be using the internal AWS IP address that is not accessible to the outside world.

danielnbarbosa commented 5 years ago

Hi @Winter-Guerra. I'm working with @PHRABAL on this same issue. I get framerates from your first set of commands, so looks like flightgoggles is working. I don't get anything from the second set of commands. I also suspect it is related to AWS security groups config. Currently I am allowing all TCP ports in from my IP and all ICMPv4 in from all IPs. However when I try to start the ROS server after setting ROS_MASTER_URI and ROS_IP as follows:

export ROS_MASTER_URI=http://34.xxx.xxx.xxx:11311
export ROS_IP=34.xxx.xxx.xxx

I get this error:

Unable to contact my own server at [http://34.xxx.xxx.xxx:44512/].
This usually means that the network is not configured properly.

A common cause is that the machine cannot ping itself.  Please check
for errors by running:

    ping 34.xxx.xxx.xxx

For more tips, please see

    http://www.ros.org/wiki/ROS/NetworkSetup

The traceback for the exception was written to the log file

ping 34.xxx.xxx.xxx does work to itself. Any ideas? Thanks so much for the help.

danielnbarbosa commented 5 years ago

@Winter-Guerra Here's the results of the commands you asked about:

ubuntu@ip-172-xxx-xxx-xxx:~$ ROS_MASTER_URI=http://34.xxx.xxx.xxx:11311 rostopic hz /uav/camera/left/image_rect_color
subscribed to [/uav/camera/left/image_rect_color]
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages

ubuntu@ip-172-xxx-xxx-xxx:~$ ROS_MASTER_URI=http://34.xxx.xxx.xxx:11311 rostopic hz /tf
subscribed to [/tf]
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
no new messages
Winter-Guerra commented 5 years ago

Hi all,

As a reminder, please don't post full IP addresses here. I've edited your comments and removed the IP addresses. Just to be sure, your setup looks like the following?

On AWS: export ROS_MASTER_URI=http://ec2_public_ip:11311 export ROS_IP=ec2_public_ip

On your local machine: export ROS_MASTER_URI=http://ec2_public_ip:11311 export ROS_IP=local_machine_public_ip

Winter-Guerra commented 5 years ago

You may also want to run roswtf on your local and ec2 machines after setting the environmental variables. It might be able to pick up on something that's wrong.

danielnbarbosa commented 5 years ago

Can you provide some guidance on what needs to be open with regards to EC2 security groups? Also, is the local machine expected to have a public IP address, will the server be calling back to it? Most people will have a local IP behind a NAT, or in the case of folks running ubuntu inside a VM on their mac even a double NAT.

danielnbarbosa commented 5 years ago

I've opened up TCP, UDP and ICMP and set the env vars per your guidelines and ROS still doesn't start up.

ubuntu@ip-172-31-61-192:~/.ros/log/cc7a0c12-2e6d-11e9-828d-0642468eebc4$ echo $ROS_MASTER_URI
http://34.xxx.xxx.xxx:11311
ubuntu@ip-172-31-61-192:~/.ros/log/cc7a0c12-2e6d-11e9-828d-0642468eebc4$ echo $ROS_IP
34.xxx.xxx.xxx
ubuntu@ip-172-31-61-192:~/.ros/log/cc7a0c12-2e6d-11e9-828d-0642468eebc4$ roslaunch flightgoggles core.launch
... logging to /home/ubuntu/.ros/log/a4729184-2e6e-11e9-828d-0642468eebc4/roslaunch-ip-172-31-61-192-2680.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http:/34.xxx.xxx.xxx:43798/

SUMMARY
========

PARAMETERS
 * /rosdistro: kinetic
 * /rosversion: 1.12.14
 * /uav/flightgoggles_imu/accelerometer_variance: 0.005
 * /uav/flightgoggles_imu/gyroscope_variance: 0.003
 * /uav/flightgoggles_lpf/gain_p: 35530.5758439
 * /uav/flightgoggles_lpf/gain_q: 266.57297629
 * /uav/flightgoggles_pid/gain_d_pitch: 0.3
 * /uav/flightgoggles_pid/gain_d_roll: 0.3
 * /uav/flightgoggles_pid/gain_d_yaw: 0.3
 * /uav/flightgoggles_pid/gain_i_pitch: 3.0
 * /uav/flightgoggles_pid/gain_i_roll: 3.0
 * /uav/flightgoggles_pid/gain_i_yaw: 3.0
 * /uav/flightgoggles_pid/gain_p_pitch: 9.0
 * /uav/flightgoggles_pid/gain_p_roll: 9.0
 * /uav/flightgoggles_pid/gain_p_yaw: 9.0
 * /uav/flightgoggles_pid/int_bound_pitch: 1000.0
 * /uav/flightgoggles_pid/int_bound_roll: 1000.0
 * /uav/flightgoggles_pid/int_bound_yaw: 1000.0
 * /uav/flightgoggles_uav_dynamics/angular_process_noise: 0.00025
 * /uav/flightgoggles_uav_dynamics/clockscale: 1.0
 * /uav/flightgoggles_uav_dynamics/drag_coefficient: 0.1
 * /uav/flightgoggles_uav_dynamics/ignore_collisions: False
 * /uav/flightgoggles_uav_dynamics/init_pose: [0.0, 0.0, 1.0, 0...
 * /uav/flightgoggles_uav_dynamics/linear_process_noise: 0.0005
 * /uav/flightgoggles_uav_dynamics/max_prop_speed: 2200.0
 * /uav/flightgoggles_uav_dynamics/moment_arm: 0.08
 * /uav/flightgoggles_uav_dynamics/motor_time_constant: 0.02
 * /uav/flightgoggles_uav_dynamics/thrust_coefficient: 1.91e-06
 * /uav/flightgoggles_uav_dynamics/torque_coefficient: 2.6e-07
 * /uav/flightgoggles_uav_dynamics/vehicle_inertia_xx: 0.0049
 * /uav/flightgoggles_uav_dynamics/vehicle_inertia_yy: 0.0049
 * /uav/flightgoggles_uav_dynamics/vehicle_inertia_zz: 0.0049
 * /uav/flightgoggles_uav_dynamics/vehicle_mass: 1.0

NODES
  /uav/
    camera_left_link (tf2_ros/static_transform_publisher)
    flightgogglesRenderer (flightgoggles/FlightGoggles.x86_64)
    flightgoggles_marker_visualizer (flightgoggles_marker_visualizer/flightgoggles_marker_visualizer)
    flightgoggles_ros_bridge (flightgoggles_ros_bridge/ROSClient)
    flightgoggles_uav_dynamics (flightgoggles_uav_dynamics/node)
    rviz (rviz/rviz)
    world_ned_link (tf2_ros/static_transform_publisher)

ERROR: unable to contact ROS master at [http://34.xxx.xxx.xxx:11311]
The traceback for the exception was written to the log file

ubuntu@ip-172-31-61-192:~/.ros/log/cc7a0c12-2e6d-11e9-828d-0642468eebc4$ cat roslaunch-ip-172-31-61-192-2619.log
[roslaunch][INFO] 2019-02-12 02:27:52,872: Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
...
[roslaunch.parent][INFO] 2019-02-12 02:27:52,877: starting roslaunch parent run
[roslaunch][INFO] 2019-02-12 02:27:52,877: loading roscore config file /opt/ros/kinetic/etc/ros/roscore.xml
[roslaunch][INFO] 2019-02-12 02:27:53,421: Added core node of type [rosout/rosout] in namespace [/]
[roslaunch.config][INFO] 2019-02-12 02:27:53,422: loading config file /home/ubuntu/catkin_ws/src/flightgoggles/flightgoggles/launch/core.launch
[roslaunch][INFO] 2019-02-12 02:27:53,455: Added node of type [flightgoggles_uav_dynamics/node] in namespace [/uav/]
[roslaunch][INFO] 2019-02-12 02:27:53,456: Added node of type [flightgoggles/FlightGoggles.x86_64] in namespace [/uav/]
[roslaunch][INFO] 2019-02-12 02:27:53,456: Added node of type [flightgoggles_ros_bridge/ROSClient] in namespace [/uav/]
[roslaunch][INFO] 2019-02-12 02:27:53,456: Added node of type [flightgoggles_marker_visualizer/flightgoggles_marker_visualizer] in namespace [/uav/]
[roslaunch][INFO] 2019-02-12 02:27:53,457: Added node of type [rviz/rviz] in namespace [/uav/]
[roslaunch][INFO] 2019-02-12 02:27:53,458: Added node of type [tf2_ros/static_transform_publisher] in namespace [/uav/]
[roslaunch][INFO] 2019-02-12 02:27:53,458: Added node of type [tf2_ros/static_transform_publisher] in namespace [/uav/]
[roslaunch][INFO] 2019-02-12 02:27:53,458: ... selected machine [] for node of type [flightgoggles_uav_dynamics/node]
[roslaunch][INFO] 2019-02-12 02:27:53,458: ... selected machine [] for node of type [flightgoggles/FlightGoggles.x86_64]
[roslaunch][INFO] 2019-02-12 02:27:53,458: ... selected machine [] for node of type [flightgoggles_ros_bridge/ROSClient]
[roslaunch][INFO] 2019-02-12 02:27:53,458: ... selected machine [] for node of type [flightgoggles_marker_visualizer/flightgoggles_marker_visualizer]
[roslaunch][INFO] 2019-02-12 02:27:53,458: ... selected machine [] for node of type [rviz/rviz]
[roslaunch][INFO] 2019-02-12 02:27:53,458: ... selected machine [] for node of type [tf2_ros/static_transform_publisher]
[roslaunch][INFO] 2019-02-12 02:27:53,458: ... selected machine [] for node of type [tf2_ros/static_transform_publisher]
[roslaunch.pmon][INFO] 2019-02-12 02:27:53,459: start_process_monitor: creating ProcessMonitor
[roslaunch.pmon][INFO] 2019-02-12 02:27:53,459: created process monitor <ProcessMonitor(ProcessMonitor-1, initial daemon)>
[roslaunch.pmon][INFO] 2019-02-12 02:27:53,460: start_process_monitor: ProcessMonitor started
[roslaunch.parent][INFO] 2019-02-12 02:27:53,460: starting parent XML-RPC server
[roslaunch.server][INFO] 2019-02-12 02:27:53,460: starting roslaunch XML-RPC server
[roslaunch.server][INFO] 2019-02-12 02:27:53,460: waiting for roslaunch XML-RPC server to initialize
[xmlrpc][INFO] 2019-02-12 02:27:53,460: XML-RPC server binding to 0.0.0.0:0
[xmlrpc][INFO] 2019-02-12 02:27:53,461: Started XML-RPC server [http://34.xxx.xxx.xxx:33967/]
[xmlrpc][INFO] 2019-02-12 02:27:53,461: xml rpc node: starting XML-RPC server
[roslaunch][INFO] 2019-02-12 02:27:53,472: started roslaunch server http://34.xxx.xxx.xxx:33967/
[roslaunch.parent][INFO] 2019-02-12 02:27:53,472: ... parent XML-RPC server started
[roslaunch][INFO] 2019-02-12 02:27:53,473: master.is_running[http://34.xxx.xxx.xxx:11311]
[roslaunch][ERROR] 2019-02-12 02:27:53,473: ERROR: unable to contact ROS master at [http://34.xxx.xxx.xxx:11311]
[roslaunch][ERROR] 2019-02-12 02:27:53,474: The traceback for the exception was written to the log file
[roslaunch][ERROR] 2019-02-12 02:27:53,474: Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/__init__.py", line 306, in main
    p.start()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/parent.py", line 279, in start
    self.runner.launch()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/launch.py", line 657, in launch
    self._setup()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/launch.py", line 632, in _setup
    launched = self._launch_master()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/launch.py", line 397, in _launch_master
    validate_master_launch(m, self.is_core, self.is_rostest)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/roslaunch/launch.py", line 105, in validate_master_launch
    raise RLException("ERROR: unable to contact ROS master at [%s]"%(m.uri))
RLException: ERROR: unable to contact ROS master at [http:/34.xxx.xxx.xxx:11311]

[rospy.core][INFO] 2019-02-12 02:27:53,474: signal_shutdown [atexit]
Winter-Guerra commented 5 years ago

I think that there is a rosmaster/roscore already running that is badly configured. roslaunch should say something about autostarting a new rosmaster. Perhaps try rosnode kill -a; killall -9 rosmaster; killall -9 roscore and try again? Also try roswtf because that provides lots of debug data.

danielnbarbosa commented 5 years ago

No, there were no other ps running. Also, interestingly roswtf (love that name BTW) outputs:

ubuntu@ip-172-31-81-64:~$ roswtf
Loaded plugin tf.tfwtf
No package or stack in context
================================================================================
Static checks summary:

Found 1 warning(s).
Warnings are things that may be just fine, but are sometimes at fault

WARNING ROS_IP may be incorrect: ROS_IP [18.xxx.xxx.xxx] does not appear to be a local IP address ['127.0.0.1', '172.31.81.64', '172.17.0.1'].

Found 1 error(s).

ERROR Not all paths in PYTHONPATH [/home/ubuntu/catkin_ws/devel/lib/python3/dist-packages:/opt/ros/kinetic/lib/python2.7/dist-packages:/home/ubuntu/src/cntk/bindings/python] point to a directory:
 * /home/ubuntu/src/cntk/bindings/python

================================================================================

ROS Master does not appear to be running.
Online graph checks will not be run.
ROS_MASTER_URI is [http://18.xxx.xxx.xxx:11311]

Seems like it actually wants an internal IP?

danielnbarbosa commented 5 years ago

Or maybe it's because the external IP doesn't show up in ifconfig on EC2 instances.

ubuntu@ip-172-31-81-64:~$ ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:30:01:d2:0b
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ens3      Link encap:Ethernet  HWaddr 16:f8:3f:bc:41:dc
          inet addr:172.31.81.64  Bcast:172.31.95.255  Mask:255.255.240.0
          inet6 addr: fe80::14f8:3fff:febc:41dc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:1631 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1221 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:196478 (196.4 KB)  TX bytes:187149 (187.1 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:6095 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6095 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:717137 (717.1 KB)  TX bytes:717137 (717.1 KB)
squilter commented 5 years ago

@danielnbarbosa the external IP is inet addr: under the ens3 entry

I agree that it probably has to do with the local machine being hidden behind a NAT. Can you try setting up a VPN and reporting back? https://answers.ros.org/question/11045/how-to-set-up-vpn-between-ros-machines/

danielnbarbosa commented 5 years ago

Well, things are looking better after setting up the VPN. I no longer get errors when running roswtf on both client and server. Also these commands now work on the client:

Still not able to get a GUI visualization of the drone when running rqt but this may just be my own ignorance, I'm not familiar with ROS. Is there something I need to do after starting rqt locally?

danielnbarbosa commented 5 years ago

Also FWIW, I see an error: Fixed Frame [map] does not exist when running rviz.

varunmurali1 commented 5 years ago

In rviz, under the displays pane, there is a drop down for fixed frame in the global options. Could you please set that to world instead of map?

Winter-Guerra commented 5 years ago

@danielnbarbosa you could also run rviz -d "$(rospack find flightgoggles)/rviz/third_person_drone_follower.rviz". That will spawn an rviz window with the correct configs and should start showing an image.

danielnbarbosa commented 5 years ago

@varunmurali1 that fixed the error, but still no image. @Winter-Guerra it's loading the config but still no image. here's what i see after running your command, this is after running roslaunch flightgoggles core.launch on the server:

screen shot 2019-02-12 at 2 47 08 pm screen shot 2019-02-12 at 2 47 33 pm

danielnbarbosa commented 5 years ago

FWIW, here's the output of roswtf on the client after running rviz -d "$(rospack find flightgoggles)/rviz/third_person_drone_follower.rviz"

Loaded plugin tf.tfwtf
No package or stack in context
================================================================================
Static checks summary:

No errors or warnings
================================================================================
Beginning tests of your ROS graph. These may take awhile...
analyzing graph...
... done analyzing graph
running graph rules...
... done running graph rules
running tf checks, this will take a second...
... tf checks complete

Online checks summary:

Found 3 warning(s).
Warnings are things that may be just fine, but are sometimes at fault

WARNING The following node subscriptions are unconnected:
 * /uav/flightgoggles_uav_dynamics:
   * /uav/input/rateThrust

WARNING The following nodes are unexpectedly connected:
 * /uav/flightgoggles_uav_dynamics->/roswtf_4643_1550011978950 (/tf)
 * /uav/flightgoggles_uav_dynamics->/roswtf_4551_1550011895918 (/tf)

WARNING No tf messages
varunmurali1 commented 5 years ago

What is the output of rostopic hz /uav/camera/left/image_rect_color and rostopic hz /uav/camera/left/ir_beacons? Could you check in rviz while running our config, if the image topic under Image in the displays pane is set to /bounding_box_camera/RGB just to ensure it is pointed to the right topic?

danielnbarbosa commented 5 years ago

@varunmurali1 Those two rostopic commands both keep returning no new messages, which is strange because they were working prior. The image topic is indeed set to /bounding_box_camera/RGB.

Winter-Guerra commented 5 years ago

Hi @danielnbarbosa, it looks like the dynamics and render engine were waiting for control input before starting. Sorry about that, I've pushed a patch that fixes this issue. Here are some commands that should get you up and running.

On AWS:

export ROS_MASTER_URI=http://ec2_public_ip:11311
export ROS_IP=ec2_public_ip

cd ~/catkin_ws/src
wstool update
roslaunch flightgoggles core.launch

On your local machine:

export ROS_MASTER_URI=http://ec2_public_ip:11311
export ROS_IP=local_machine_public_ip

# Requires that FlightGoggles is also installed locally
cd ~/catkin_ws/src
wstool update
# This will run teleop and rviz
roslaunch flightgoggles teleopAWSFromLocalMachine.launch
danielnbarbosa commented 5 years ago

That did the trick. Thanks!

screen shot 2019-02-12 at 7 02 08 pm

varunmurali1 commented 5 years ago

@danielnbarbosa glad it now works for you!

danielnbarbosa commented 5 years ago

Thanks for being so proactive @Winter-Guerra @varunmurali1 @squilter.

mhaboali commented 4 years ago

Well, things are looking better after setting up the VPN. I no longer get errors when running roswtf on both client and server. Also these commands now work on the client:

  • rostopic hz /uav/camera/left/image_rect_color
  • rostopic hz /tf

Still not able to get a GUI visualization of the drone when running rqt but this may just be my own ignorance, I'm not familiar with ROS. Is there something I need to do after starting rqt locally?

Hi @danielnbarbosa,

Could you explain in some details how did you manage to establish the VPN connection between your robot and the AWS instance? when I followed the steps described here, I managed to see the topics but all topics are empty and I'm completely stuck.

Thanks!