ROS 2 interoperability issues

vmayoral commented 2 years ago

I'm unable to run https://github.com/BerkeleyAutomation/FogROS2/blob/main/fogros2_examples/launch/talker.launch.py successfully and obtain basic interoperability. Neither between edge-cloud, nor between edge-edge.

From my inspection, VPN works just fine but there's something wrong in the setup. ROS 2 CLI daemons aren't able to see simple Nodes launched (manually) across machines in the VPN.

This deserves further attention.

jeffi commented 2 years ago

@vmayoral can you try again on the humble branch? I've made a few changes and I'm now able to run the talker example without issue. If you do run into an issue the ssh changes I made should make it easier to find the root cause.

vmayoral commented 2 years ago

@jeffi, gave it a try again. Failed twice with humble branch:

Attempt 1 :x:	Attempt 2 ❌

Here're a few additional observations:

👍 there's definitely an improvement with the prompt, so @jeffi's changes definitely added good.
I think we should polish the 101 experience (the talker launchfile I demonstrate above) before exposing this to others.
humble branch is inconsistent from a ROS 2 perspective, it should upon rolling in here https://github.com/BerkeleyAutomation/FogROS2/blob/humble/Dockerfile#L1, not on top of Galactic (Humble is branching out Rolling). I believe @KDharmarajanDev took care of this in main.
we're missing in humble branch many of the changes introduced in main (sorry I sent them there, I just didn't know that humble was the actual "good" one, since generally, development happens in main and then you branch things out when ready, as appropriate). I think it'd be nice to cherry-pick many of the changes introduced in main regarding the README polishing or just rebase things together.
I think it'd be nice to default to whatever development repo we're sticking with (right now, default is main).

jeffi commented 2 years ago

We branched humble since it will undergo many changes that are inconsistent with the paper. We branched recently, so it should be straightforward to merge everything onto the humble branch. Anything on main that is inconsistent with the paper, we may need to back out. Sorry for not being clear about this earlier.

What are the steps you're taking to reproduce the issues you see? Also are you using an EC2 instance near you? Or us-west-1?

vmayoral commented 2 years ago

Been using us-west-1. This is known to work in the past and I would expect bigger latencies, but still interoperability.

Do you guys get issues with machines in Europe?

El El mié, 13 abr 2022 a las 18:44, Jeff Ichnowski @.***> escribió:

We branched humble since it will undergo many changes that are inconsistent with the paper. We branched recently, so it should be straightforward to merge everything onto the humble branch. Anything on main that is inconsistent with the paper, we may need to back out. Sorry for not being clear about this earlier.

What are the steps you're taking to reproduce the issues you see? Also are you using an EC2 instance near you? Or us-west-1?

— Reply to this email directly, view it on GitHub https://github.com/BerkeleyAutomation/FogROS2/issues/19#issuecomment-1098267575, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKPYDVWCAIRVOKTBD5JBOTVE32VNANCNFSM5TASODDQ . You are receiving this because you were mentioned.Message ID: @.***>

jeffi commented 2 years ago

I think some DDS changes have occurred. So it might be the issue. Can you try with a closer instance? That’s best practice anyways.

I tried connecting from Berkeley to eu-west-1 and didn't run any issues.

KeplerC commented 2 years ago

I just ran the humble branch with docker, and it works fine on my end

vmayoral commented 2 years ago

All right folks, gave it yet another try with the humble branch and failed again ❌ :

git clone --recurse-submodules https://github.com/BerkeleyAutomation/FogROS2 -b humble
cd FogROS2
docker build -t fogros2:latest .
docker run -it --rm --net=host --cap-add=NET_ADMIN fogros2
aws configure
source install/setup.bash
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp 
export CYCLONEDDS_URI=file://$(pwd)/install/share/fogros2/configs/cyclonedds.xml

# edited src/fogros2/fogros2_examples/launch/talker.launch.py, and switched to "eu-west-1", tried various combinations
ros2 launch fogros2_examples talker.launch.py

`eu-west-1` ❌

AMI's not available in this location:

root@xilinx:/home/root/fog_ws# ros2 launch fogros2_examples talker.launch.py
[INFO] [launch]: All log files can be found below /root/.ros/log/2022-04-14-17-18-25-804926-xilinx-709
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [root]: init
using ROS workspace:  /home/root/fog_ws
creating EC2 instance
security group id is ['sg-03aff72c987f1fb2b']
[ERROR] [launch]: Caught exception in launch (see debug for traceback): Caught exception when trying to load file of format [py]: An error occurred (InvalidAMIID.NotFound) when calling the RunInstances operation: The image id '[ami-09175f2ca3c3dc67c]' does not exist

I'd be interested to know if someone manages to reproduce this in Europe.

`us-west-1` ❌

[INFO] [launch]: All log files can be found below /home/ubuntu/.ros/log/2022-04-14-17-27-39-176746-ip-172-31-20-210-8020
[INFO] [launch]: Default logging verbosity is set to INFO
action added
[INFO] [talker-1]: process started with pid [8232]
[talker-1] 1649957263.505701 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7410 failed with retcode -12
[talker-1] 1649957263.505735 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7412 failed with retcode -12
[talker-1] 1649957263.505751 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7414 failed with retcode -12
[talker-1] 1649957263.505764 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7416 failed with retcode -12
[talker-1] 1649957263.505777 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7418 failed with retcode -12
[talker-1] 1649957263.505790 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7420 failed with retcode -12
[talker-1] 1649957263.505803 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7422 failed with retcode -12
[talker-1] 1649957263.505816 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7424 failed with retcode -12
[talker-1] 1649957263.505840 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7426 failed with retcode -12
[talker-1] 1649957263.605819 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7410 failed with retcode -12
[talker-1] 1649957263.605855 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7412 failed with retcode -12
[talker-1] 1649957263.605869 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7414 failed with retcode -12
[talker-1] 1649957263.605881 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7416 failed with retcode -12
[talker-1] 1649957263.605894 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7418 failed with retcode -12
[talker-1] 1649957263.605906 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7420 failed with retcode -12
[talker-1] 1649957263.605918 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7422 failed with retcode -12
[talker-1] 1649957263.605929 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7424 failed with retcode -12
[talker-1] 1649957263.605942 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7426 failed with retcode -12
[talker-1] [WARN] [1649957264.059897731] [minimal_publisher]: Publishing: "Hello World: 0"
[talker-1] [WARN] [1649957264.547799905] [minimal_publisher]: Publishing: "Hello World: 1"
[talker-1] [WARN] [1649957265.047765659] [minimal_publisher]: Publishing: "Hello World: 2"
[talker-1] [WARN] [1649957265.547783128] [minimal_publisher]: Publishing: "Hello World: 3"
[talker-1] [WARN] [1649957266.047788410] [minimal_publisher]: Publishing: "Hello World: 4"
[talker-1] [WARN] [1649957266.547765959] [minimal_publisher]: Publishing: "Hello World: 5"
[talker-1] [WARN] [1649957267.047778546] [minimal_publisher]: Publishing: "Hello World: 6"
[talker-1] [WARN] [1649957267.547725197] [minimal_publisher]: Publishing: "Hello World: 7"
[talker-1] [WARN] [1649957268.047791400] [minimal_publisher]: Publishing: "Hello World: 8"
[talker-1] [WARN] [1649957268.547799940] [minimal_publisher]: Publishing: "Hello World: 9"
[talker-1] [WARN] [1649957269.047773415] [minimal_publisher]: Publishing: "Hello World: 10"
[talker-1] [WARN] [1649957269.547797331] [minimal_publisher]: Publishing: "Hello World: 11"
[talker-1] [WARN] [1649957270.047731141] [minimal_publisher]: Publishing: "Hello World: 12"
[talker-1] [WARN] [1649957270.547734673] [minimal_publisher]: Publishing: "Hello World: 13"
[talker-1] [WARN] [1649957271.047857520] [minimal_publisher]: Publishing: "Hello World: 14"
[talker-1] [WARN] [1649957271.547791794] [minimal_publisher]: Publishing: "Hello World: 15"
[talker-1] 1649957271.605965 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7410 failed with retcode -12
[talker-1] 1649957271.606008 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7412 failed with retcode -12
[talker-1] 1649957271.606022 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7414 failed with retcode -12
[talker-1] 1649957271.606033 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7416 failed with retcode -12
[talker-1] 1649957271.606045 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7418 failed with retcode -12
[talker-1] 1649957271.606057 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7420 failed with retcode -12
[talker-1] 1649957271.606069 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7422 failed with retcode -12
[talker-1] 1649957271.606081 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7424 failed with retcode -12
[talker-1] 1649957271.606093 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7426 failed with retcode -12
[talker-1] [WARN] [1649957272.047780478] [minimal_publisher]: Publishing: "Hello World: 16"
[talker-1] [WARN] [1649957272.547810903] [minimal_publisher]: Publishing: "Hello World: 17"
[talker-1] [WARN] [1649957273.047775785] [minimal_publisher]: Publishing: "Hello World: 18"
[talker-1] [WARN] [1649957273.547864499] [minimal_publisher]: Publishing: "Hello World: 19"
[talker-1] [WARN] [1649957274.047790936] [minimal_publisher]: Publishing: "Hello World: 20"
[talker-1] [WARN] [1649957274.547792087] [minimal_publisher]: Publishing: "Hello World: 21"
[talker-1] [WARN] [1649957275.047782828] [minimal_publisher]: Publishing: "Hello World: 22"
[talker-1] [WARN] [1649957275.547805925] [minimal_publisher]: Publishing: "Hello World: 23"
[talker-1] [WARN] [1649957276.047801592] [minimal_publisher]: Publishing: "Hello World: 24"
[talker-1] [WARN] [1649957276.547817732] [minimal_publisher]: Publishing: "Hello World: 25"
[talker-1] [WARN] [1649957277.047784636] [minimal_publisher]: Publishing: "Hello World: 26"
[talker-1] [WARN] [1649957277.548027508] [minimal_publisher]: Publishing: "Hello World: 27"
[talker-1] [WARN] [1649957278.047788750] [minimal_publisher]: Publishing: "Hello World: 28"
[talker-1] [WARN] [1649957278.547805068] [minimal_publisher]: Publishing: "Hello World: 29"
[talker-1] [WARN] [1649957279.047773950] [minimal_publisher]: Publishing: "Hello World: 30"
[talker-1] [WARN] [1649957279.547744832] [minimal_publisher]: Publishing: "Hello World: 31"
[talker-1] 1649957279.606107 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7410 failed with retcode -12
[talker-1] 1649957279.606138 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7412 failed with retcode -12
[talker-1] 1649957279.606152 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7414 failed with retcode -12
[talker-1] 1649957279.606163 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7416 failed with retcode -12
[talker-1] 1649957279.606194 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7418 failed with retcode -12
[talker-1] 1649957279.606212 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7420 failed with retcode -12
[talker-1] 1649957279.606224 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7422 failed with retcode -12
[talker-1] 1649957279.606236 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7424 failed with retcode -12
[talker-1] 1649957279.606248 [0]        tev: ddsi_udp_conn_write to udp/10.0.0.1:7426 failed with retcode -12
[talker-1] [WARN] [1649957280.047784027] [minimal_publisher]: Publishing: "Hello World: 32"
[talker-1] [WARN] [1649957280.547778110] [minimal_publisher]: Publishing: "Hello World: 33"
[talker-1] [WARN] [1649957281.047787163] [minimal_publisher]: Publishing: "Hello World: 34"
[talker-1] [WARN] [1649957281.547907442] [minimal_publisher]: Publishing: "Hello World: 35"
[talker-1] [WARN] [1649957282.047792322] [minimal_publisher]: Publishing: "Hello World: 36"
[talker-1] [WARN] [1649957282.547796226] [minimal_publisher]: Publishing: "Hello World: 37"
[talker-1] [WARN] [1649957283.047783401] [minimal_publisher]: Publishing: "Hello World: 38"
[talker-1] [WARN] [1649957283.547781548] [minimal_publisher]: Publishing: "Hello World: 39"

Additional observations

I can see there's been some decent cleanup in the humble branch but from what I can see at #15, we're still missing quite a bit of things to release into Humble. Critical aspects include:

Remove duplicated code in FogROS2 launch extensions fogros2_launch (ping @jeffi and @KeplerC, there's currently lots of duplicated code and we should only include the extensions)

Right now I can see we're duplicating official ROS 2 packages, which will block this from being accepted.

KeplerC commented 2 years ago

I repeated the steps that @vmayoral mentioned. I had the same problem whenever I start fogros container the first time.

Then my theory is that if the VPN's wg0 interface exists, but /etc/wireguard/wg0 file does not exist(because the container is new), then the interface is not deleted and restarted. Then the setup does not succeed.

The fix is in #22

jeffi commented 2 years ago

(Just getting the issue up to date from conversations over Slack and email)

The issue appears to stem from ROS_DOMAIN_ID. When we set it on the robot, the setting is not transmitted to the cloud instance. The result is that robot and cloud were talking on different port ranges, and thus never to each other. The workaround for now is to leave ROS_DOMAIN_ID unset on the robot (to match the cloud). The fix will be to match the ROS_DOMAIN_ID setting on the cloud.

vmayoral commented 2 years ago

Tried latest changes in humble in a clean, Docker-bases setup and failed ❌ :

Error dump

```bash root@xilinx:/home/root/fog_ws# ros2 launch fogros2_examples talker.launch.py [INFO] [launch]: All log files can be found below /root/.ros/log/2022-04-16-06-52-48-253448-xilinx-69 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [root]: init using ROS workspace: /home/root/fog_ws creating EC2 instance security group id is ['sg-0e92c69af4995c85c'] i-04e0451b9131a77c3 EC2 instance is created with ip address: 13.57.190.164 Hit:1 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease Get:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Get:4 http://repo.ros2.org/ubuntu/main focal InRelease [4,676 B] Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] Get:6 http://packages.ros.org/ros2/ubuntu focal InRelease [4,676 B] Get:7 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,712 kB] Get:8 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main Translation-en [320 kB] Get:9 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.9 kB] Get:10 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [917 kB] Get:11 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [131 kB] Get:12 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [528 B] Get:13 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [918 kB] Get:14 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [205 kB] Get:15 http://repo.ros2.org/ubuntu/main focal/main arm64 Packages [987 kB] Get:16 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.6 kB] Get:17 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] Get:18 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,336 B] Get:19 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] Get:20 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] Get:21 http://repo.ros2.org/ubuntu/main focal/main amd64 Packages [1,012 kB] Get:22 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] Get:23 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] Get:24 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.5 kB] Get:25 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [808 B] Get:26 http://packages.ros.org/ros2/ubuntu focal/main amd64 Packages [1,012 kB] Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,385 kB] Get:28 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [240 kB] Get:29 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.0 kB] Get:30 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [859 kB] Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [122 kB] Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [532 B] Get:33 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [698 kB] Get:34 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [124 kB] Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.3 kB] Fetched 11.2 MB in 2s (4,975 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... The following packages will be upgraded: locales 1 upgraded, 0 newly installed, 0 to remove and 173 not upgraded. Need to get 3,871 kB of archives. After this operation, 0 B of additional disk space will be used. Get:1 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 locales all 2.31-0ubuntu9.7 [3,871 kB] debconf: unable to initialize frontend: Dialog debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (This frontend requires a controlling tty.) debconf: falling back to frontend: Teletype dpkg-preconfigure: unable to re-open stdin: Fetched 3,871 kB in 0s (15.4 MB/s) (Reading database ... 172136 files and directories currently installed.) Preparing to unpack .../locales_2.31-0ubuntu9.7_all.deb ... Unpacking locales (2.31-0ubuntu9.7) over (2.31-0ubuntu9.2) ... Setting up locales (2.31-0ubuntu9.7) ... debconf: unable to initialize frontend: Dialog debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) debconf: falling back to frontend: Readline Generating locales (this might take a while)... en_US.ISO-8859-1... done en_US.UTF-8... done Generation complete. Processing triggers for man-db (2.9.1-1) ... Generating locales (this might take a while)... en_US.ISO-8859-1... done en_US.UTF-8... done Generation complete. Reading package lists... Building dependency tree... Reading state information... software-properties-common is already the newest version (0.99.9.8). 0 upgraded, 0 newly installed, 0 to remove and 173 not upgraded. 'universe' distribution component is already enabled for all sources. Hit:1 http://repo.ros2.org/ubuntu/main focal InRelease Hit:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Hit:5 http://packages.ros.org/ros2/ubuntu focal InRelease Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease Fetched 222 kB in 1s (425 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... lsb-release is already the newest version (11.1.0ubuntu2). curl is already the newest version (7.68.0-1ubuntu2.7). gnupg is already the newest version (2.2.19-3ubuntu2.1). 0 upgraded, 0 newly installed, 0 to remove and 173 not upgraded. Hit:1 http://repo.ros2.org/ubuntu/main focal InRelease Hit:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Hit:5 http://packages.ros.org/ros2/ubuntu focal InRelease Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease Fetched 222 kB in 1s (423 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... ros-rolling-desktop is already the newest version (0.9.3-1focal.20220204.201458). 0 upgraded, 0 newly installed, 0 to remove and 173 not upgraded. Reading package lists... Building dependency tree... Reading state information... The following additional packages will be installed: python3-bson python3-bson-ext python3-tornado ros-rolling-rosapi ros-rolling-rosapi-msgs ros-rolling-rosbridge-library ros-rolling-rosbridge-msgs ros-rolling-rosbridge-server Suggested packages: python3-pycurl python-tornado-doc The following NEW packages will be installed: python3-bson python3-bson-ext python3-tornado ros-rolling-rosapi ros-rolling-rosapi-msgs ros-rolling-rosbridge-library ros-rolling-rosbridge-msgs ros-rolling-rosbridge-server ros-rolling-rosbridge-suite 0 upgraded, 9 newly installed, 0 to remove and 173 not upgraded. Need to get 743 kB of archives. After this operation, 8,006 kB of additional disk space will be used. Get:1 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosapi-msgs amd64 1.1.2-1focal.20220121.214852 [249 kB] Get:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 python3-bson amd64 3.10.1-0ubuntu2 [34.8 kB] Get:3 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosbridge-library amd64 1.1.2-1focal.20220122.073542 [44.2 kB] Get:4 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosapi amd64 1.1.2-1focal.20220126.191225 [18.6 kB] Get:5 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosbridge-msgs amd64 1.1.2-1focal.20220121.213906 [43.0 kB] Get:6 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosbridge-server amd64 1.1.2-1focal.20220126.193440 [19.8 kB] Get:7 http://repo.ros2.org/ubuntu/main focal/main amd64 ros-rolling-rosbridge-suite amd64 1.1.2-1focal.20220126.193929 [6,600 B] Get:8 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 python3-bson-ext amd64 3.10.1-0ubuntu2 [22.3 kB] Get:9 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 python3-tornado amd64 6.0.3+really5.1.1-3 [304 kB] debconf: unable to initialize frontend: Dialog debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (This frontend requires a controlling tty.) debconf: falling back to frontend: Teletype dpkg-preconfigure: unable to re-open stdin: Fetched 743 kB in 0s (6,784 kB/s) Selecting previously unselected package python3-bson. (Reading database ... 172136 files and directories currently installed.) Preparing to unpack .../0-python3-bson_3.10.1-0ubuntu2_amd64.deb ... Unpacking python3-bson (3.10.1-0ubuntu2) ... Selecting previously unselected package python3-bson-ext. Preparing to unpack .../1-python3-bson-ext_3.10.1-0ubuntu2_amd64.deb ... Unpacking python3-bson-ext (3.10.1-0ubuntu2) ... Selecting previously unselected package python3-tornado. Preparing to unpack .../2-python3-tornado_6.0.3+really5.1.1-3_amd64.deb ... Unpacking python3-tornado (6.0.3+really5.1.1-3) ... Selecting previously unselected package ros-rolling-rosapi-msgs. Preparing to unpack .../3-ros-rolling-rosapi-msgs_1.1.2-1focal.20220121.214852_amd64.deb ... Unpacking ros-rolling-rosapi-msgs (1.1.2-1focal.20220121.214852) ... Selecting previously unselected package ros-rolling-rosbridge-library. Preparing to unpack .../4-ros-rolling-rosbridge-library_1.1.2-1focal.20220122.073542_amd64.deb ... Unpacking ros-rolling-rosbridge-library (1.1.2-1focal.20220122.073542) ... Selecting previously unselected package ros-rolling-rosapi. Preparing to unpack .../5-ros-rolling-rosapi_1.1.2-1focal.20220126.191225_amd64.deb ... Unpacking ros-rolling-rosapi (1.1.2-1focal.20220126.191225) ... Selecting previously unselected package ros-rolling-rosbridge-msgs. Preparing to unpack .../6-ros-rolling-rosbridge-msgs_1.1.2-1focal.20220121.213906_amd64.deb ... Unpacking ros-rolling-rosbridge-msgs (1.1.2-1focal.20220121.213906) ... Selecting previously unselected package ros-rolling-rosbridge-server. Preparing to unpack .../7-ros-rolling-rosbridge-server_1.1.2-1focal.20220126.193440_amd64.deb ... Unpacking ros-rolling-rosbridge-server (1.1.2-1focal.20220126.193440) ... Selecting previously unselected package ros-rolling-rosbridge-suite. Preparing to unpack .../8-ros-rolling-rosbridge-suite_1.1.2-1focal.20220126.193929_amd64.deb ... Unpacking ros-rolling-rosbridge-suite (1.1.2-1focal.20220126.193929) ... Setting up python3-tornado (6.0.3+really5.1.1-3) ... Setting up ros-rolling-rosapi-msgs (1.1.2-1focal.20220121.214852) ... Setting up python3-bson (3.10.1-0ubuntu2) ... Setting up python3-bson-ext (3.10.1-0ubuntu2) ... Setting up ros-rolling-rosbridge-msgs (1.1.2-1focal.20220121.213906) ... Setting up ros-rolling-rosbridge-library (1.1.2-1focal.20220122.073542) ... Setting up ros-rolling-rosapi (1.1.2-1focal.20220126.191225) ... Setting up ros-rolling-rosbridge-server (1.1.2-1focal.20220126.193440) ... Setting up ros-rolling-rosbridge-suite (1.1.2-1focal.20220126.193929) ... Processing triggers for libc-bin (2.31-0ubuntu9.2) ... ssh -o StrictHostKeyChecking=no -i /tmp/fogros/272/FogROSKEYAWS167.pem ubuntu@13.57.190.164 "source /opt/ros/rolling/setup.bash && ros2 launch rosbridge_server rosbridge_websocket_launch.xml &" Warning: Permanently added '13.57.190.164' (ECDSA) to the list of known hosts. Warning: apt-key output should not be parsed (stdout is not a terminal) OK Hit:1 http://repo.ros2.org/ubuntu/main focal InRelease Hit:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal InRelease Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] Hit:5 http://packages.ros.org/ros2/ubuntu focal InRelease Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease Fetched 222 kB in 1s (428 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... The following additional packages will be installed: python3-colcon-installed-package-information python3-colcon-override-check The following NEW packages will be installed: python3-colcon-installed-package-information python3-colcon-override-check The following packages will be upgraded: python3-colcon-common-extensions 1 upgraded, 2 newly installed, 0 to remove and 172 not upgraded. Need to get 15.5 kB of archives. After this operation, 74.8 kB of additional disk space will be used. Get:1 http://repo.ros2.org/ubuntu/main focal/main amd64 python3-colcon-common-extensions all 0.3.0-1 [2,824 B] Get:2 http://repo.ros2.org/ubuntu/main focal/main amd64 python3-colcon-installed-package-information all 0.0.1-1 [7,512 B] Get:3 http://repo.ros2.org/ubuntu/main focal/main amd64 python3-colcon-override-check all 0.0.1-1 [5,172 B] debconf: unable to initialize frontend: Dialog debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (This frontend requires a controlling tty.) debconf: falling back to frontend: Teletype dpkg-preconfigure: unable to re-open stdin: Fetched 15.5 kB in 0s (1,363 kB/s) (Reading database ... 173133 files and directories currently installed.) Preparing to unpack .../python3-colcon-common-extensions_0.3.0-1_all.deb ... Unpacking python3-colcon-common-extensions (0.3.0-1) over (0.2.1-1) ... Selecting previously unselected package python3-colcon-installed-package-information. Preparing to unpack .../python3-colcon-installed-package-information_0.0.1-1_all.deb ... Unpacking python3-colcon-installed-package-information (0.0.1-1) ... Selecting previously unselected package python3-colcon-override-check. Preparing to unpack .../python3-colcon-override-check_0.0.1-1_all.deb ... Unpacking python3-colcon-override-check (0.0.1-1) ... Setting up python3-colcon-common-extensions (0.3.0-1) ... Setting up python3-colcon-installed-package-information (0.0.1-1) ... Setting up python3-colcon-override-check (0.0.1-1) ... [INFO] [launch]: All log files can be found below /home/ubuntu/.ros/log/2022-04-16-06-56-37-215579-ip-172-31-31-128-5219 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [rosbridge_websocket-1]: process started with pid [5484] [INFO] [rosapi_node-2]: process started with pid [5486] Reading package lists... Building dependency tree... Reading state information... unzip is already the newest version (6.0-25ubuntu1). wireguard is already the newest version (1.0.20200513-1~20.04.2). The following additional packages will be installed: bridge-utils containerd pigz runc ubuntu-fan Suggested packages: ifupdown aufs-tools cgroupfs-mount | cgroup-lite debootstrap docker-doc rinse zfs-fuse | zfsutils The following NEW packages will be installed: bridge-utils containerd docker.io pigz runc ubuntu-fan 0 upgraded, 6 newly installed, 0 to remove and 172 not upgraded. Need to get 74.2 MB of archives. After this operation, 360 MB of additional disk space will be used. Get:1 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/universe amd64 pigz amd64 2.4-1 [57.4 kB] Get:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 bridge-utils amd64 1.6-2ubuntu1 [30.5 kB] Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 runc amd64 1.0.1-0ubuntu2~20.04.1 [4,155 kB] Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 containerd amd64 1.5.5-0ubuntu3~20.04.2 [33.0 MB] Get:5 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.7-0ubuntu5~20.04.2 [36.9 MB] Get:6 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 ubuntu-fan all 0.12.13 [34.5 kB] debconf: unable to initialize frontend: Dialog debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (This frontend requires a controlling tty.) debconf: falling back to frontend: Teletype dpkg-preconfigure: unable to re-open stdin: [rosapi_node-2] [WARN] [1650092206.223525877] [rcl.logging_rosout]: Publisher already registered for provided node name. If this is due to multiple nodes with the same name then all logs for that logger name will go out over the existing publisher. As soon as any node with that name is destructed it will unregister the publisher, preventing any further logs for that name from being published on the rosout topic. Fetched 74.2 MB in 4s (20.7 MB/s) [rosbridge_websocket-1] [INFO] [1650092206.311298513] [rosbridge_websocket]: Rosbridge WebSocket server started on port 9090 Selecting previously unselected package pigz. (Reading database ... 173168 files and directories currently installed.) Preparing to unpack .../0-pigz_2.4-1_amd64.deb ... Unpacking pigz (2.4-1) ... Selecting previously unselected package bridge-utils. Preparing to unpack .../1-bridge-utils_1.6-2ubuntu1_amd64.deb ... Unpacking bridge-utils (1.6-2ubuntu1) ... Selecting previously unselected package runc. Preparing to unpack .../2-runc_1.0.1-0ubuntu2~20.04.1_amd64.deb ... Unpacking runc (1.0.1-0ubuntu2~20.04.1) ... Selecting previously unselected package containerd. Preparing to unpack .../3-containerd_1.5.5-0ubuntu3~20.04.2_amd64.deb ... Unpacking containerd (1.5.5-0ubuntu3~20.04.2) ... Selecting previously unselected package docker.io. Preparing to unpack .../4-docker.io_20.10.7-0ubuntu5~20.04.2_amd64.deb ... Unpacking docker.io (20.10.7-0ubuntu5~20.04.2) ... Selecting previously unselected package ubuntu-fan. Preparing to unpack .../5-ubuntu-fan_0.12.13_all.deb ... Unpacking ubuntu-fan (0.12.13) ... Setting up runc (1.0.1-0ubuntu2~20.04.1) ... Setting up bridge-utils (1.6-2ubuntu1) ... debconf: unable to initialize frontend: Dialog debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) debconf: falling back to frontend: Readline Setting up pigz (2.4-1) ... Setting up containerd (1.5.5-0ubuntu3~20.04.2) ... Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service. Setting up ubuntu-fan (0.12.13) ... Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service → /lib/systemd/system/ubuntu-fan.service. Setting up docker.io (20.10.7-0ubuntu5~20.04.2) ... debconf: unable to initialize frontend: Dialog debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) debconf: falling back to frontend: Readline Adding group `docker' (GID 131) ... Done. Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service. Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket. Processing triggers for systemd (245.4-4ubuntu3.13) ... Processing triggers for man-db (2.9.1-1) ... Reading package lists... Building dependency tree... Reading state information... python3-pip is already the newest version (20.0.2-5ubuntu1.6). 0 upgraded, 0 newly installed, 0 to remove and 172 not upgraded. Requirement already satisfied: wgconfig in /usr/local/lib/python3.8/dist-packages (0.2.2) Requirement already satisfied: boto3 in /usr/local/lib/python3.8/dist-packages (1.21.6) Requirement already satisfied: paramiko in /usr/local/lib/python3.8/dist-packages (2.9.2) Requirement already satisfied: scp in /usr/local/lib/python3.8/dist-packages (0.14.4) Requirement already satisfied: botocore<1.25.0,>=1.24.6 in /usr/local/lib/python3.8/dist-packages (from boto3) (1.24.6) Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /usr/local/lib/python3.8/dist-packages (from boto3) (0.5.1) Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.8/dist-packages (from boto3) (0.10.0) Requirement already satisfied: pynacl>=1.0.1 in /usr/lib/python3/dist-packages (from paramiko) (1.3.0) Requirement already satisfied: bcrypt>=3.1.3 in /usr/local/lib/python3.8/dist-packages (from paramiko) (3.2.0) Requirement already satisfied: cryptography>=2.5 in /usr/lib/python3/dist-packages (from paramiko) (2.8) Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.25.0,>=1.24.6->boto3) (1.25.8) Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3/dist-packages (from botocore<1.25.0,>=1.24.6->boto3) (2.7.3) Requirement already satisfied: six>=1.4.1 in /usr/lib/python3/dist-packages (from bcrypt>=3.1.3->paramiko) (1.14.0) Requirement already satisfied: cffi>=1.1 in /usr/local/lib/python3.8/dist-packages (from bcrypt>=3.1.3->paramiko) (1.15.0) Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko) (2.21) /home/root fog_ws removing old workspace mv: cannot move 'fog_ws' to a subdirectory of itself, 'fog_ws/fog_ws' successfully extracted new workspace [ERROR] [launch]: Caught exception in launch (see debug for traceback): No module named 'fogros2.actions' ```

I'll be taking a few days off but will re-engage with the discussion next week!

mjd3 commented 2 years ago

Superseded by #36

BerkeleyAutomation / FogROS2