search

wil3 / gymfc

A universal flight control tuning framework
http://wfk.io/neuroflight/
MIT License
402 stars 101 forks source link

test_axis.py Timeout communicating with flight control plugin #40

Closed xabierolaz closed 4 years ago

xabierolaz commented 4 years ago

Whenever I run text_axis.py

python3 test_start_sim.py ../modules/digitaltwins/nf1/model.sdf

Gazebo opens with the nf1 in the middle, but shells wont get pass from here:

(env) cuda@eim-alu-83252:~/projects/gymfc/tests$ python3 test_axis.py ~/projects/gymfc/modules/digitaltwins/nf1/model.sdf
Sending motor control signals to port  9527
Gazebo Model Path = /home/cuda/local/share/gazebo-10/models::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/models:/home/cuda/projects/gymfc/modules/digitaltwins
Gazebo Plugin Path = /home/cuda/local/lib/gazebo-10/plugins::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/plugins/build:/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build
Starting gzserver with process ID= 3028
Roll Left
-----------------------------------

After 1 minute, this message appears and gazebo ends.

Timeout communicating with flight control plugin.

Simulation Stats
-----------------
steps                  0
packets_dropped        0
time_start_seconds     1586362388.8515196
time_lapse_hours       0.016432642539342245

Killing Gazebo process with ID= 3028
Killing Gazebo process with ID= 3029
Timeout communicating with flight control plugin.

Watching the youtube video in your Channel, I see that after all these hyphens, this should appear, but wont get pass from there. Captura

wil3 commented 4 years ago

99% of the time its because you don't have the plugins built and in the correct location. You get a time out because there are no motor plugins to talk to. The thesis branch is still WIP in progress so there is still stuff missing.

Take a look here for instructions https://github.com/wil3/gymfc#directory-layout

So youll need to update the directory to be something like, /modules/digitaltwins/nf1/plugin/bulid/

If you are using these plugins https://github.com/wil3/gymfc-aircraft-plugins youll need to create soft links to the built IMU and motor model plugins (libgazebo_imu_plugin.so and libgazebo_motor_model.so).

wil3 commented 4 years ago

There's actually an open issue to make this more clear, https://github.com/wil3/gymfc/issues/37

SwapnilPande commented 4 years ago

@xabierolaz What output do you get when you pass the --verbose tag to test_start_sim.py?

xabierolaz commented 4 years ago

@wil3 thanks, will have a look at #37 and directory layout instructions

@SwapnilPande this is what I get passing the --verbose

(env) cuda@eim-alu-83252:~/projects/gymfc/tests$ python3 test_axis.py ~/projects/gymfc/modules/digitaltwins/nf1/model.sdf --verbose
Sending motor control signals to port  9722
Gazebo Model Path = /home/cuda/local/share/gazebo-10/models::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/models:/home/cuda/projects/gymfc/modules/digitaltwins
Gazebo Plugin Path = /home/cuda/local/lib/gazebo-10/plugins::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/plugins/build:/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build
Starting gzserver with process ID= 12609
Roll Left
-----------------------------------
Gazebo multi-robot simulator, version 10.1.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11375
[Msg] Publicized address: 172.18.83.252
[Dbg] [DARTModel.cc:72] Initializing DART model attitude_control_training_rig
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'pivot' and joint 'base_joint'.
[Err] [DARTJoint.cc:195] DARTJoint: SetAnchor is not implemented.
[Dbg] [FlightControllerPlugin.cpp:225] Binding on port 9722
[Dbg] [FlightControllerPlugin.cpp:408] CoT link=battery
[Dbg] [FlightControllerPlugin.cpp:410] Got COT from plugin 0 0 0.058
[Dbg] [FlightControllerPlugin.cpp:413] Num motors 4
[Dbg] [FlightControllerPlugin.cpp:443] Inserting digital twin from SDF, /home/cuda/projects/gymfc/modules/digitaltwins/nf1/model.sdf.
[Dbg] [DARTModel.cc:72] Initializing DART model nf1
[Dbg] [DARTModel.cc:128] Building DART BodyNode for link 'frame' with a free joint.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'battery' and joint 'battery_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_1' and joint 'motor_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_2' and joint 'motor_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_3' and joint 'motor_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_4' and joint 'motor_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_1' and joint 'prop_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_2' and joint 'prop_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_3' and joint 'prop_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_4' and joint 'prop_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'fc_stack' and joint 'fc_stack_joint'.
[Err] [Plugin.hh:187] Failed to load plugin libgazebo_motor_model.so: libgazebo_motor_model.so: cannot open shared object file: No such file or directory
[Err] [Plugin.hh:187] Failed to load plugin libgazebo_motor_model.so: libgazebo_motor_model.so: cannot open shared object file: No such file or directory
[Err] [Plugin.hh:187] Failed to load plugin libgazebo_motor_model.so: libgazebo_motor_model.so: cannot open shared object file: No such file or directory
[Err] [Plugin.hh:187] Failed to load plugin libgazebo_motor_model.so: libgazebo_motor_model.so: cannot open shared object file: No such file or directory
[Err] [Plugin.hh:187] Failed to load plugin libgazebo_imu_plugin.so: libgazebo_imu_plugin.so: cannot open shared object file: No such file or directory
[Dbg] [FlightControllerPlugin.cpp:527] Setting link to center battery
[Dbg] [FlightControllerPlugin.cpp:542] Aircraft model fixed to world
Timeout communicating with flight control plugin.

Simulation Stats
-----------------
steps                  0
packets_dropped        0
time_start_seconds     1586517499.5778537
time_lapse_hours       0.016418274839719138

Killing Gazebo process with ID= 12609
/bin/sh: 1: kill: No such process

Killing Gazebo process with ID= 12610
Timeout communicating with flight control plugin.
SwapnilPande commented 4 years ago

Yeah, so as @wil3 mentioned earlier, it can't find the motor model plugin. I've created a PR (#42) with additional documentation on how to setup the aircraft model. Until it is merged, you can try following the instructions here in my fork of the repo.

wil3 commented 4 years ago

@xabierolaz did building and linking the aircraft-plugins resolve the error? Can this be closed?

xabierolaz commented 4 years ago

Followed all steps building and linking the aircraft plugins and following the directory structure, and getting this now when running test_axis.py:

Might be something related to SetAnchor?

(env) cuda@eim-alu-83252:~/projects/gymfc/tests$ python3 test_axis.py ../modules/digitaltwins/nf1/model.sdf --verbose
Sending motor control signals to port  9177
Gazebo Model Path = /home/cuda/local/share/gazebo-10/models::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/models:/home/cuda/projects/gymfc/modules/digitaltwins
Gazebo Plugin Path = /home/cuda/local/lib/gazebo-10/plugins::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/plugins/build:/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build
Starting gzserver with process ID= 27257
Roll Left
-----------------------------------
Gazebo multi-robot simulator, version 10.1.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11842
[Msg] Publicized address: 172.18.83.252
[Dbg] [DARTModel.cc:72] Initializing DART model attitude_control_training_rig
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'pivot' and joint 'base_joint'.
[Err] [DARTJoint.cc:195] DARTJoint: SetAnchor is not implemented.
[Dbg] [FlightControllerPlugin.cpp:225] Binding on port 9177
[Dbg] [FlightControllerPlugin.cpp:408] CoT link=battery
[Dbg] [FlightControllerPlugin.cpp:410] Got COT from plugin 0 0 0.058
[Dbg] [FlightControllerPlugin.cpp:413] Num motors 4
[Dbg] [FlightControllerPlugin.cpp:443] Inserting digital twin from SDF, ../modules/digitaltwins/nf1/model.sdf.
[Dbg] [DARTModel.cc:72] Initializing DART model nf1
[Dbg] [DARTModel.cc:128] Building DART BodyNode for link 'frame' with a free joint.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'battery' and joint 'battery_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_1' and joint 'motor_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_2' and joint 'motor_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_3' and joint 'motor_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_4' and joint 'motor_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_1' and joint 'prop_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_2' and joint 'prop_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_3' and joint 'prop_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_4' and joint 'prop_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'fc_stack' and joint 'fc_stack_joint'.
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=0 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=1 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=2 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=3 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_imu_plugin.cpp:55] Loading IMU sensor
[Dbg] [FlightControllerPlugin.cpp:527] Setting link to center battery
[Dbg] [FlightControllerPlugin.cpp:542] Aircraft model fixed to world
Timeout communicating with flight control plugin.

Simulation Stats
-----------------
steps                  0
packets_dropped        0
time_start_seconds     1586774205.728632
time_lapse_hours       0.016450484659936693

Killing Gazebo process with ID= 27257
Killing Gazebo process with ID= 27258
Timeout communicating with flight control plugin.
(env) cuda@eim-alu-83252:~/projects/gymfc/tests$
wil3 commented 4 years ago

What's the output of, ls -l /home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build

Show me the build contents of the aircraft plugin repo.

xabierolaz commented 4 years ago

This is the output

### cuda@eim-alu-83252:~$ ls -l /home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build
total 16
lrwxrwxrwx 1 cuda cuda 106 Apr 13 10:25 libcontrol_msgs.so -> /home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/gymfc-aircraft-plugins/build/libcontrol_msgs.so
lrwxrwxrwx 1 cuda cuda 111 Apr 13 10:25 libgazebo_imu_plugin.so -> /home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/gymfc-aircraft-plugins/build/libgazebo_imu_plugin.so
lrwxrwxrwx 1 cuda cuda 112 Apr 13 10:25 libgazebo_motor_model.so -> /home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/gymfc-aircraft-plugins/build/libgazebo_motor_model.so
lrwxrwxrwx 1 cuda cuda 105 Apr 13 10:25 libsensor_msgs.so -> /home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/gymfc-aircraft-plugins/build/libsensor_msgs.so
cuda@eim-alu-83252:~$

And the content of the airclaft plugin repo's build folder Captura

Steps I followed:

1-Created these directories nf1/ model.config model.sdf plugins/ build/

2- git clone inside nf1/plugins

3- inside gymfc-aircraft-plugins mkdir build cd build cmake ../ make (no errors)

4- symlink cp --symbolic-link ~/projects/gymfc/modules/digitaltwins/nf1/plugins/gymfc-aircraft-plugins/build/*.so ~/projects/gymfc/modules/digitaltwins/nf1/plugins/build

wil3 commented 4 years ago

Thanks @xabierolaz. Everything looks fine but you do have something things different than how I have it setup so could you try a couple things? I don't think it should cause problems but I've never cloned the gymfc-aircraft-plugin repo inside a digital twin. To rule out all possibilities could you move it to ~/projects? I'm also not familiar with that method to create symbolic links, your ls output ls looks correct though.

Inside ~/projects/gymfc/modules/digitaltwins/nf1/plugins/build could you first remove all files in the directory and then execute, 

ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_imu_plugin.so libgazebo_imu_plugin.so
ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_motor_model.so libgazebo_motor_model.so

My build directory only has these two files.

wil3 commented 4 years ago

Reviewing the code, this error occurs when the python environment can not communicate with the Gazebo environment over the UDP network. The flight control plugin will wait until it gets data back from the motors and IMU, if those plugins don't exist it waits indefinitely and times out. This is usually the problem as I previously mentioned. If the above does not help it may require uncommenting some debug statements in the FlightControllerPlugin.cpp plugin to see if its possibly a networking issue.

I did test out test_axis.py with the thesis branch and nf1 model and was able to run it without problems.

xabierolaz commented 4 years ago

Went to a previous snapshot where gymfc was clean and tried both with @SwapnilPande method and deleting (method 1)

and trying your method from scratch (method 2)

To sum up: Both ways throw same output, with the setanchor error

Method 1 (Steps 1 to 9)

1- Created /plugins/build directories inside nf1 folder 2- git cloned gymfc-aircraft-plugins this time inside projects folder 3- inside /projects/gymfc-aircraft-plugins: mkdir build cd build cmake ../ make 4- went to build folder/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build/ Folder is empty, but when I use this method to create symbolic links,

5- cp --symbolic-link ~/projects/gymfc-aircraft-plugins/build/*.so ~/projects/gymfc/modules/digitaltwins/nf1/plugins/build

BTW, this is the method @SwapnilPande mentioned in its repo https://github.com/SwapnilPande/gymfc/#setting-up-an-aircraft-model

when running this, 4 files are created there, so I'm guessing both libcontrol_msgs.so and libsensor_msgs.so shouldn't be there? Captura

6- Deleted all 4 files 7-Run cuda@eim-alu-83252:~/projects/gymfc/modules/digitaltwins/nf1/plugins/build$ ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_imu_plugin.so libgazebo_imu_plugin.so

cuda@eim-alu-83252:~/projects/gymfc/modules/digitaltwins/nf1/plugins/build$ ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_motor_model.so libgazebo_motor_model.so 8-This time just the 2 files you mentioned are created, imu and motor Captura

9-Proceed to run test_axis.py

(env) cuda@eim-alu-83252:~/projects/gymfc/tests$ python3 test_axis.py ../modules/digitaltwins/nf1/model.sdf --verbose
Sending motor control signals to port  9323
Gazebo Model Path = /home/cuda/local/share/gazebo-10/models::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/models:/home/cuda/projects/gymfc/modules/digitaltwins
Gazebo Plugin Path = /home/cuda/local/lib/gazebo-10/plugins::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/plugins/build:/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build
Starting gzserver with process ID= 2603
Roll Left
-----------------------------------
Gazebo multi-robot simulator, version 10.1.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11853
[Msg] Publicized address: 172.18.83.252
[Dbg] [DARTModel.cc:72] Initializing DART model attitude_control_training_rig
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'pivot' and joint 'base_joint'.
[Err] [DARTJoint.cc:195] DARTJoint: SetAnchor is not implemented.
[Dbg] [FlightControllerPlugin.cpp:225] Binding on port 9323
[Dbg] [FlightControllerPlugin.cpp:408] CoT link=battery
[Dbg] [FlightControllerPlugin.cpp:410] Got COT from plugin 0 0 0.058
[Dbg] [FlightControllerPlugin.cpp:413] Num motors 4
[Dbg] [FlightControllerPlugin.cpp:443] Inserting digital twin from SDF, ../modules/digitaltwins/nf1/model.sdf.
[Dbg] [DARTModel.cc:72] Initializing DART model nf1
[Dbg] [DARTModel.cc:128] Building DART BodyNode for link 'frame' with a free joint.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'battery' and joint 'battery_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_1' and joint 'motor_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_2' and joint 'motor_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_3' and joint 'motor_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_4' and joint 'motor_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_1' and joint 'prop_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_2' and joint 'prop_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_3' and joint 'prop_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_4' and joint 'prop_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'fc_stack' and joint 'fc_stack_joint'.
[Dbg] [FlightControllerPlugin.cpp:527] Setting link to center battery
[Dbg] [FlightControllerPlugin.cpp:542] Aircraft model fixed to world
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=0 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=1 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=2 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=3 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_imu_plugin.cpp:55] Loading IMU sensor
Timeout communicating with flight control plugin.

Simulation Stats
-----------------
steps                  0
packets_dropped        0
time_start_seconds     1586855447.330785
time_lapse_hours       0.01643754972351922

Killing Gazebo process with ID= 2603
Killing Gazebo process with ID= 2604
Timeout communicating with flight control plugin.

Method 2

10- As this didn't seem to work, tried with your symlink method

went to a previous snapshot, this time didnt run the symlink of @SwapnilPande and run yours instead (build folder was empty and the 2 files you mentioned were created this time)

ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_imu_plugin.so libgazebo_imu_plugin.so
ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_motor_model.so libgazebo_motor_model.so

Output:

(env) cuda@eim-alu-83252:~/projects/gymfc/tests$ python3 test_axis.py ../modules/digitaltwins/nf1/model.sdf --verbose
Sending motor control signals to port  9645
Gazebo Model Path = /home/cuda/local/share/gazebo-10/models::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/models:/home/cuda/projects/gymfc/modules/digitaltwins
Gazebo Plugin Path = /home/cuda/local/lib/gazebo-10/plugins::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/plugins/build:/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build
Starting gzserver with process ID= 1066
Roll Left
-----------------------------------
Gazebo multi-robot simulator, version 10.1.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11514
[Msg] Publicized address: 172.18.83.252
[Dbg] [DARTModel.cc:72] Initializing DART model attitude_control_training_rig
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'pivot' and joint 'base_joint'.
[Err] [DARTJoint.cc:195] DARTJoint: SetAnchor is not implemented.
[Dbg] [FlightControllerPlugin.cpp:225] Binding on port 9645
[Dbg] [FlightControllerPlugin.cpp:408] CoT link=battery
[Dbg] [FlightControllerPlugin.cpp:410] Got COT from plugin 0 0 0.058
[Dbg] [FlightControllerPlugin.cpp:413] Num motors 4
[Dbg] [FlightControllerPlugin.cpp:443] Inserting digital twin from SDF, ../modules/digitaltwins/nf1/model.sdf.
[Dbg] [DARTModel.cc:72] Initializing DART model nf1
[Dbg] [DARTModel.cc:128] Building DART BodyNode for link 'frame' with a free joint.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'battery' and joint 'battery_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_1' and joint 'motor_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_2' and joint 'motor_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_3' and joint 'motor_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_4' and joint 'motor_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_1' and joint 'prop_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_2' and joint 'prop_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_3' and joint 'prop_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_4' and joint 'prop_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'fc_stack' and joint 'fc_stack_joint'.
[Dbg] [FlightControllerPlugin.cpp:527] Setting link to center battery
[Dbg] [FlightControllerPlugin.cpp:542] Aircraft model fixed to world
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=0 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=1 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=2 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=3 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_imu_plugin.cpp:55] Loading IMU sensor
Timeout communicating with flight control plugin.

Simulation Stats
-----------------
steps                  0
packets_dropped        0
time_start_seconds     1586856066.7606745
time_lapse_hours       0.016447839670711093

Killing Gazebo process with ID= 1066
Killing Gazebo process with ID= 1067
Timeout communicating with flight control plugin.
wil3 commented 4 years ago

I'm not familiar with that window manager you are using, you mentioned using a VM, what OS are you running?

I suggest uncommenting the gzdbg statements in this function, rebuilding and providing the output, https://github.com/wil3/gymfc/blob/master/gymfc/envs/assets/gazebo/plugins/FlightControllerPlugin.cpp#L593

Your problem isn't obvious, its going to take some deep diving into the source code to find out whats going on with your environment.

xabierolaz commented 4 years ago

I'm using Ubuntu 18.04.3 LTS with the xfce desktop environment, thats why it looks that way Captura

It's indeed a VM with these specifications: 16GB RAM 4 cores Nvidia Quadro 6000 (it was built so it can use the gpu)

I will be commenting output here by tomorrow, as Im rebuilding everything taking consideration of all the feedback you gave me

xabierolaz commented 4 years ago

Re-building process: (at the end, got same setanchor error and output with those gzdbg lines uncommented)

1- went to a clean state with just gazebo and dart installed 2-Installed gymfc and noticed this error I didnt have noticed before when doing pip3 install -e .

error: invalid command 'bdist_wheel

(env) cuda@eim-alu-83252:~/projects/gymfc$ pip3 install -e . Obtaining file:///home/cuda/projects/gymfc Collecting gym (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/77/48/c43b8a72b916cc70896aa431b0fc00d1481ae34e28dc55e2144f4c77916b/gym-0.17.1.tar.gz (1.6MB) 100% |################################| 1.6MB 614kB/s Collecting numpy (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/07/08/a549ba8b061005bb629b76adc000f3caaaf881028b963c2e18f811c6edc1/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl (20.2MB) 100% |################################| 20.2MB 58kB/s Collecting protobuf (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/57/02/5432412c162989260fab61fa65e0a490c1872739eb91a659896e4d554b26/protobuf-3.11.3-cp36-cp36m-manylinux1_x86_64.whl (1.3MB) 100% |################################| 1.3MB 883kB/s Collecting psutil>=5.3.0 (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/c4/b8/3512f0e93e0db23a71d82485ba256071ebef99b227351f0f5540f744af41/psutil-5.7.0.tar.gz (449kB) 100% |################################| 450kB 2.4MB/s Collecting cloudpickle<1.4.0,>=1.2.0 (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/ea/0b/189cd3c19faf362ff2df5f301456c6cf8571ef6684644cfdfdbff293825c/cloudpickle-1.3.0-py2.py3-none-any.whl Collecting pyglet<=1.5.0,>=1.4.0 (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/70/ca/20aee170afe6011e295e34b27ad7d7ccd795faba581dd3c6f7cec237f561/pyglet-1.5.0-py2.py3-none-any.whl (1.0MB) 100% |################################| 1.0MB 1.1MB/s Collecting scipy (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/dc/29/162476fd44203116e7980cfbd9352eef9db37c49445d1fec35509022f6aa/scipy-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (26.1MB) 100% |################################| 26.1MB 44kB/s Collecting six (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from protobuf->gymfc==0.1.2) Collecting future (from pyglet<=1.5.0,>=1.4.0->gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB) 100% |################################| 829kB 1.3MB/s Building wheels for collected packages: gym, psutil, future Running setup.py bdist_wheel for gym ... error Complete output from command /home/cuda/projects/gymfc/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-tvmb034u/gym/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmps_139zw1pip-wheel- --python-tag cp36: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for gym Running setup.py clean for gym Running setup.py bdist_wheel for psutil ... error Complete output from command /home/cuda/projects/gymfc/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-tvmb034u/psutil/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpasg4d_p5pip-wheel- --python-tag cp36: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for psutil Running setup.py clean for psutil Running setup.py bdist_wheel for future ... error Complete output from command /home/cuda/projects/gymfc/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-tvmb034u/future/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpwxvavi5ipip-wheel- --python-tag cp36: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for future Running setup.py clean for future Failed to build gym psutil future Installing collected packages: cloudpickle, numpy, future, pyglet, scipy, six, gym, protobuf, psutil, gymfc Running setup.py install for future ... done Running setup.py install for gym ... done Running setup.py install for psutil ... done Running setup.py develop for gymfc Successfully installed cloudpickle-1.3.0 future-0.18.2 gym-0.17.1 gymfc numpy-1.18.2 protobuf-3.11.3 psutil-5.7.0 pyglet-1.5.0 scipy-1.4.1 six-1.14.0

Did some research and gets fixed with pip install wheel before doing pip3 install -e . As gymfc seems to install no matter the error, dont know if I should ignore it or try to fix it.

However, this time I did pip install wheel before installing gymfc in case that could be the reason.

So, steps: mkdir projects && cd projects git clone https://github.com/wil3/gymfc.git //downloaded gymfc sudo apt-get install python3-venv -y //python3 -m venv dependencies cd gymfc && python3 -m venv env //inside gymfc folder, created a virtual environment called env source env/bin/activate //activated environment pip install wheel pip3 install -e .

no 'bdist_wheel error this time, as Im doing pip install wheel before (env) cuda@eim-alu-83252:~/projects/gymfc$ pip3 install -e . Obtaining file:///home/cuda/projects/gymfc Collecting gym (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/77/48/c43b8a72b916cc70896aa431b0fc00d1481ae34e28dc55e2144f4c77916b/gym-0.17.1.tar.gz (1.6MB) 100% |################################| 1.6MB 723kB/s Collecting numpy (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/07/08/a549ba8b061005bb629b76adc000f3caaaf881028b963c2e18f811c6edc1/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl (20.2MB) 100% |################################| 20.2MB 58kB/s Collecting protobuf (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/57/02/5432412c162989260fab61fa65e0a490c1872739eb91a659896e4d554b26/protobuf-3.11.3-cp36-cp36m-manylinux1_x86_64.whl (1.3MB) 100% |################################| 1.3MB 971kB/s Collecting psutil>=5.3.0 (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/c4/b8/3512f0e93e0db23a71d82485ba256071ebef99b227351f0f5540f744af41/psutil-5.7.0.tar.gz (449kB) 100% |################################| 450kB 2.6MB/s Collecting cloudpickle<1.4.0,>=1.2.0 (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/ea/0b/189cd3c19faf362ff2df5f301456c6cf8571ef6684644cfdfdbff293825c/cloudpickle-1.3.0-py2.py3-none-any.whl Collecting pyglet<=1.5.0,>=1.4.0 (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/70/ca/20aee170afe6011e295e34b27ad7d7ccd795faba581dd3c6f7cec237f561/pyglet-1.5.0-py2.py3-none-any.whl (1.0MB) 100% |################################| 1.0MB 1.2MB/s Collecting scipy (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/dc/29/162476fd44203116e7980cfbd9352eef9db37c49445d1fec35509022f6aa/scipy-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (26.1MB) 100% |################################| 26.1MB 44kB/s Collecting six (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from protobuf->gymfc==0.1.2) Collecting future (from pyglet<=1.5.0,>=1.4.0->gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB) 100% |################################| 829kB 1.5MB/s Building wheels for collected packages: gym, psutil, future Running setup.py bdist_wheel for gym ... done Stored in directory: /home/cuda/.cache/pip/wheels/c0/84/61/523b92d88787ae29689b3cc08cf445d8d8186d7fbe1acbf87b Running setup.py bdist_wheel for psutil ... done Stored in directory: /home/cuda/.cache/pip/wheels/d7/69/b4/3200b95828d1f0ddb3cb5699083717f4fdbd9b4223d0644c57 Running setup.py bdist_wheel for future ... done Stored in directory: /home/cuda/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e Successfully built gym psutil future Installing collected packages: cloudpickle, numpy, future, pyglet, scipy, six, gym, protobuf, psutil, gymfc Running setup.py develop for gymfc Successfully installed cloudpickle-1.3.0 future-0.18.2 gym-0.17.1 gymfc numpy-1.18.2 protobuf-3.11.3 psutil-5.7.0 pyglet-1.5.0 scipy-1.4.1 six-1.14.0

cd ~/projects/gymfc/gymfc/envs/assets/gazebo/plugins uncommented gzb lines you told me inside FlightControllerPlugin.cpp .../build_plugin.sh as it's uncommented, building throws some errors and cannot be completed, most of them about not declaring f variables (im guessing you meant just to modify the .cpp and run test_axis.py, not rebuilding gymfc completely?)

In this case, I will build build_plugin.sh without modifying the FlightController.cpp

changed gymfc.ini to be pointing to new gazebo as usual added nf1 folder with it's directory layout structure cloned aircraft plugin repo inside projects folder ~/projects Build the plugins. From the root directory of the repo,~/projects/gymfc-aircraft-plugins run: mkdir build && cd build

cmake ../ (no errors)

-- The C compiler identification is GNU 7.5.0 -- The CXX compiler identification is GNU 7.5.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Boost version: 1.65.1 -- Found the following Boost libraries: -- system -- thread -- timer -- chrono -- date_time -- atomic -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Checking for module 'bullet>=2.82' -- Found bullet, version 2.87 -- Found Simbody: /usr/include/simbody -- Found CCD: /usr/include (found version "2.0") -- Found FCL: /usr/include (found version "0.5.0") -- Found ASSIMP: /usr/include (found version "4.1.0") -- Found DART: /usr/local/include (Required is at least version "6.6") found components: dart -- Boost version: 1.65.1 -- Found the following Boost libraries: -- thread -- system -- filesystem -- program_options -- regex -- iostreams -- date_time -- chrono -- atomic -- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread (found version "3.0.0") -- Boost version: 1.65.1 -- Looking for OGRE... -- OGRE_PREFIX_WATCH changed. -- Checking for module 'OGRE' -- Found OGRE, version 1.9.0 -- Found Ogre Ghadamon (1.9.0) -- Found OGRE: optimized;/usr/lib/x86_64-linux-gnu/libOgreMain.so;debug;/usr/lib/x86_64-linux-gnu/libOgreMain.so -- Looking for OGRE_Paging... -- Found OGRE_Paging: optimized;/usr/lib/x86_64-linux-gnu/libOgrePaging.so;debug;/usr/lib/x86_64-linux-gnu/libOgrePaging.so -- Looking for OGRE_Terrain... -- Found OGRE_Terrain: optimized;/usr/lib/x86_64-linux-gnu/libOgreTerrain.so;debug;/usr/lib/x86_64-linux-gnu/libOgreTerrain.so -- Looking for OGRE_Property... -- Found OGRE_Property: optimized;/usr/lib/x86_64-linux-gnu/libOgreProperty.so;debug;/usr/lib/x86_64-linux-gnu/libOgreProperty.so -- Looking for OGRE_RTShaderSystem... -- Found OGRE_RTShaderSystem: optimized;/usr/lib/x86_64-linux-gnu/libOgreRTShaderSystem.so;debug;/usr/lib/x86_64-linux-gnu/libOgreRTShaderSystem.so -- Looking for OGRE_Volume... -- Found OGRE_Volume: optimized;/usr/lib/x86_64-linux-gnu/libOgreVolume.so;debug;/usr/lib/x86_64-linux-gnu/libOgreVolume.so -- Looking for OGRE_Overlay... -- Found OGRE_Overlay: optimized;/usr/lib/x86_64-linux-gnu/libOgreOverlay.so;debug;/usr/lib/x86_64-linux-gnu/libOgreOverlay.so -- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread;-lpthread (found suitable version "3.0.0", minimum required is "2.3.0") -- Config-file not installed for ZeroMQ -- checking for pkg-config -- Checking for module 'libzmq >= 4' -- Found libzmq , version 4.2.5 -- Found ZeroMQ: TRUE (Required is at least version "4") -- Checking for module 'uuid' -- Found uuid, version 2.31.1 -- Found UUID: TRUE -- Checking for module 'tinyxml2' -- Found tinyxml2, version 6.0.0 -- Looking for dlfcn.h - found -- Looking for libdl - found -- Found DL: TRUE -- FreeImage.pc not found, we will search for FreeImage_INCLUDE_DIRS and FreeImage_LIBRARIES -- Checking for module 'gts' -- Found gts, version 0.7.6 -- Found GTS: TRUE -- Checking for module 'libswscale' -- Found libswscale, version 4.8.100 -- Found SWSCALE: TRUE -- Checking for module 'libavdevice >= 56.4.100' -- Found libavdevice , version 57.10.100 -- Found AVDEVICE: TRUE (Required is at least version "56.4.100") -- Checking for module 'libavformat' -- Found libavformat, version 57.83.100 -- Found AVFORMAT: TRUE -- Checking for module 'libavcodec' -- Found libavcodec, version 57.107.100 -- Found AVCODEC: TRUE -- Checking for module 'libavutil' -- Found libavutil, version 55.78.100 -- Found AVUTIL: TRUE -- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "7.58.0") -- Checking for module 'jsoncpp' -- Found jsoncpp, version 1.7.4 -- Found JSONCPP: TRUE -- Checking for module 'yaml-0.1' -- Found yaml-0.1, version 0.1.7 -- Found YAML: TRUE -- Checking for module 'libzip' -- Found libzip, version 1.1.2 -- Found ZIP: TRUE -- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread;-lpthread;-lpthread (found version "3.0.0") -- Checking for module 'protobuf' -- Found protobuf, version 3.0.0 Gazebo version: 10.1 -- Configuring done -- Generating done -- Build files have been written to: /home/cuda/projects/gymfc-aircraft-plugins/build

make (no errors)

cuda@eim-alu-83252:~/projects/gymfc-aircraft-plugins/build$ make [ 5%] Running C++ protocol buffer compiler on /home/cuda/local/include/gazebo-10/gazebo/msgs/proto/quaternion.proto [ 10%] Running C++ protocol buffer compiler on msgs/Float.proto [ 15%] Running C++ protocol buffer compiler on msgs/Imu.proto [ 20%] Running C++ protocol buffer compiler on msgs/MotorSpeed.proto [ 25%] Running C++ protocol buffer compiler on msgs/EscSensor.proto [ 30%] Running C++ protocol buffer compiler on /home/cuda/local/include/gazebo-10/gazebo/msgs/proto/vector3d.proto Scanning dependencies of target sensor_msgs [ 35%] Building CXX object CMakeFiles/sensor_msgs.dir/Float.pb.cc.o [ 40%] Building CXX object CMakeFiles/sensor_msgs.dir/Imu.pb.cc.o [ 45%] Building CXX object CMakeFiles/sensor_msgs.dir/MotorSpeed.pb.cc.o [ 50%] Building CXX object CMakeFiles/sensor_msgs.dir/EscSensor.pb.cc.o [ 55%] Building CXX object CMakeFiles/sensor_msgs.dir/vector3d.pb.cc.o [ 60%] Building CXX object CMakeFiles/sensor_msgs.dir/quaternion.pb.cc.o [ 65%] Linking CXX shared library libsensor_msgs.so [ 65%] Built target sensor_msgs [ 70%] Running C++ protocol buffer compiler on msgs/MotorCommand.proto Scanning dependencies of target control_msgs [ 75%] Building CXX object CMakeFiles/control_msgs.dir/MotorCommand.pb.cc.o [ 80%] Linking CXX shared library libcontrol_msgs.so [ 80%] Built target control_msgs Scanning dependencies of target gazebo_imu_plugin [ 85%] Building CXX object CMakeFiles/gazebo_imu_plugin.dir/src/gazebo_imu_plugin.cpp.o [ 90%] Linking CXX shared library libgazebo_imu_plugin.so [ 90%] Built target gazebo_imu_plugin Scanning dependencies of target gazebo_motor_model [ 95%] Building CXX object CMakeFiles/gazebo_motor_model.dir/src/gazebo_motor_model.cpp.o [100%] Linking CXX shared library libgazebo_motor_model.so [100%] Built target gazebo_motor_model

Tried to link with, but wont let me as it says those file are already there build folder is again fillefdwith all these files (as in the image I posted)

Observation: /home/cuda/projects/gymfc-aircraft-plugins/build/ creates all these files when building with cmake and make, shouldn't just those 2 .so files be created instead? 

ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_imu_plugin.so libgazebo_imu_plugin.so
ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_motor_model.so libgazebo_motor_model.so

However, as test_axis.py is still outputing same error, deleted /home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build/ content and linked anyway so both .so files are created again

Proceed to change the .cpp now, uncommenting those gzdbg lines

ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_imu_plugin.so libgazebo_imu_plugin.so ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_motor_model.so libgazebo_motor_model.so

text_axis.py output with those gzdb lines uncommented:

Result:

same output as before, with the SetAnchor error

(env) cuda@eim-alu-83252:~/projects/gymfc/tests$ python3 test_axis.py ~/projects/gymfc/modules/digitaltwins/nf1/model.sdf --verbose Sending motor control signals to port 9095 Gazebo Model Path = /home/cuda/local/share/gazebo-10/models::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/models:/home/cuda/projects/gymfc/modules/digitaltwins Gazebo Plugin Path = /home/cuda/local/lib/gazebo-10/plugins::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/plugins/build:/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build Starting gzserver with process ID= 12207 Roll Left ----------------------------------- Gazebo multi-robot simulator, version 10.1.0 Copyright (C) 2012 Open Source Robotics Foundation. Released under the Apache 2 License. http://gazebosim.org [Msg] Waiting for master. [Msg] Connected to gazebo master @ http://127.0.0.1:11638 [Msg] Publicized address: 172.18.83.252 [Dbg] [DARTModel.cc:72] Initializing DART model attitude_control_training_rig [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'pivot' and joint 'base_joint'. [Err] [DARTJoint.cc:195] DARTJoint: SetAnchor is not implemented. [Dbg] [FlightControllerPlugin.cpp:225] Binding on port 9095 [Dbg] [FlightControllerPlugin.cpp:408] CoT link=battery [Dbg] [FlightControllerPlugin.cpp:410] Got COT from plugin 0 0 0.058 [Dbg] [FlightControllerPlugin.cpp:413] Num motors 4 [Dbg] [FlightControllerPlugin.cpp:443] Inserting digital twin from SDF, /home/cuda/projects/gymfc/modules/digitaltwins/nf1/model.sdf. [Dbg] [DARTModel.cc:72] Initializing DART model nf1 [Dbg] [DARTModel.cc:128] Building DART BodyNode for link 'frame' with a free joint. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'battery' and joint 'battery_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_1' and joint 'motor_1_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_2' and joint 'motor_2_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_3' and joint 'motor_3_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_4' and joint 'motor_4_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_1' and joint 'prop_1_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_2' and joint 'prop_2_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_3' and joint 'prop_3_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_4' and joint 'prop_4_joint'. [Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'fc_stack' and joint 'fc_stack_joint'. [Dbg] [FlightControllerPlugin.cpp:527] Setting link to center battery [Dbg] [FlightControllerPlugin.cpp:542] Aircraft model fixed to world [Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=0 Subscribed to /aircraft/command/motor [Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=1 Subscribed to /aircraft/command/motor [Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=2 Subscribed to /aircraft/command/motor [Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=3 Subscribed to /aircraft/command/motor [Dbg] [gazebo_imu_plugin.cpp:55] Loading IMU sensor Timeout communicating with flight control plugin. Simulation Stats ----------------- steps 0 packets_dropped 0 time_start_seconds 1587024379.6890924 time_lapse_hours 0.01643631484773424 Killing Gazebo process with ID= 12207 Killing Gazebo process with ID= 12208 Timeout communicating with flight control plugin.

wil3 commented 4 years ago

If you aren't seeing the debug statements that you uncommented that could mean the packets sent from python are never getting to the plugin for some reason. You'll need to trace the plugin to see where it's getting too. If you aren't receiving any packets there's a networking problem somewhere on your setup. You can use Wireshark to capture the UDP packets being sent from the Python code. The log messages tell you which UDP port is used to you can filter the packet stream on that. If you need help making sense of the Wireshark output upload it here and I'll take a look. I've never tested it within a VM so I cant say if that's a source of the problem.

Other option is to work off of the docker images that have been tested to work.

wil3 commented 4 years ago

Following your steps above you built the plugin and the made the code change. You made sure to rebuild the GymFC plugin after the code change right? (The build_plugin.sh script)

wil3 commented 4 years ago

@xabierolaz Sorry I've been on mobile, was able to read through your post in greater detail. You write,

cd ~/projects/gymfc/gymfc/envs/assets/gazebo/plugins uncommented gzb lines you told me inside FlightControllerPlugin.cpp .../build_plugin.sh as it's uncommented, building throws some errors and cannot be completed, most of them about not declaring f variables (im guessing you meant just to modify the .cpp and run test_axis.py, not rebuilding gymfc completely?) In this case, I will build build_plugin.sh without modifying the FlightController.cpp

The purpose of this test was to trace in FlightController.cpp where you are getting so it can help you figure out what's going on in your setup. Since you didn't make any changes there's nothing new here to use to help debug. If uncommenting gzdbg statements cause error then put your own in that just print out where you are in the code. Most importantly you need to find out if you are getting here. If you aren't then somehow your client and server are not communicating.

xabierolaz commented 4 years ago

Okay, I was uncommenting the https://github.com/wil3/gymfc/blob/master/gymfc/envs/assets/gazebo/plugins/FlightControllerPlugin.cpp#L593 loop, not the https://github.com/wil3/gymfc/blob/master/gymfc/envs/assets/gazebo/plugins/FlightControllerPlugin.cpp#L749 so I will uncomment that too, rebuild it and copy the output here

EDIT: uncommented those loops, built plugin.sh, same output, so it must be not getting there

Some considerations before using Wireshark to trace the packages

1-Is it okay to pip install wheel before pip3 install -e . gymfc? should be gymfc able to get installed without any bdist_wheel errors ?

2- Should the virtual environment be installed in ~/PROJECTS/gymfc/ or in gymfc ~/projects/GYMFC/(inside repository) ?

Code:

(env) cuda@eim-alu-83252:~/projects/gymfc/tests$ python3 test_axis.py ../modules/digitaltwins/nf1/model.sdf --verbose
Sending motor control signals to port  9308
Gazebo Model Path = /home/cuda/local/share/gazebo-10/models::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/models:/home/cuda/projects/gymfc/modules/digitaltwins
Gazebo Plugin Path = /home/cuda/local/lib/gazebo-10/plugins::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/plugins/build:/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build
Starting gzserver with process ID= 7494
Roll Left
-----------------------------------
Gazebo multi-robot simulator, version 10.1.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11801
[Msg] Publicized address: 172.18.83.252
[Dbg] [DARTModel.cc:72] Initializing DART model attitude_control_training_rig
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'pivot' and joint 'base_joint'.
[Err] [DARTJoint.cc:195] DARTJoint: SetAnchor is not implemented.
[Dbg] [FlightControllerPlugin.cpp:225] Binding on port 9308
[Dbg] [FlightControllerPlugin.cpp:408] CoT link=battery
[Dbg] [FlightControllerPlugin.cpp:410] Got COT from plugin 0 0 0.058
[Dbg] [FlightControllerPlugin.cpp:413] Num motors 4
[Dbg] [FlightControllerPlugin.cpp:443] Inserting digital twin from SDF, ../modules/digitaltwins/nf1/model.sdf.
[Dbg] [DARTModel.cc:72] Initializing DART model nf1
[Dbg] [DARTModel.cc:128] Building DART BodyNode for link 'frame' with a free joint.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'battery' and joint 'battery_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_1' and joint 'motor_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_2' and joint 'motor_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_3' and joint 'motor_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'motor_4' and joint 'motor_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_1' and joint 'prop_1_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_2' and joint 'prop_2_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_3' and joint 'prop_3_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'prop_4' and joint 'prop_4_joint'.
[Dbg] [DARTModel.cc:166] Building DART BodyNode for link 'fc_stack' and joint 'fc_stack_joint'.
[Dbg] [FlightControllerPlugin.cpp:527] Setting link to center battery
[Dbg] [FlightControllerPlugin.cpp:542] Aircraft model fixed to world
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=0 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=1 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=2 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_motor_model.cpp:204] Loading Motor number=3 Subscribed to /aircraft/command/motor
[Dbg] [gazebo_imu_plugin.cpp:55] Loading IMU sensor
Timeout communicating with flight control plugin.

Simulation Stats
-----------------
steps                  0
packets_dropped        0
time_start_seconds     1587120479.7972784
time_lapse_hours       0.016429428259531656

Killing Gazebo process with ID= 7494
Killing Gazebo process with ID= 7495
Timeout communicating with flight control plugin.
wil3 commented 4 years ago
  1. It should be. I've never scene that error before and gymfc doesn't use wheels (but its dependencies may). I can double check this weekend.
  2. Doesnt matter where virtual environments are installed, they can be anywhere. I typically name them env and put them in the root of projects (which is why its in .gitignore).

As a sanity check I recommend printing out a debug statement in the plugin as soon as it starts to make sure you are using the new one you are modifying. Maybe like right after this statement, "[Dbg] [FlightControllerPlugin.cpp:225] Binding on port 9308".

Wireshark can verify the python side is sending UDP packets, just filter the UDP port by the port specified in the logs, (i.e., 9308 in the example above). The init sets the hostname to localhost, also make sure thats being resolved. On your machine if you just ping localhost it'll do the translation.

xabierolaz commented 4 years ago

Quick update, installing wheel is not needed and therefore there are no errors when installing gymfc with pip3 install -e . without using any virtual environment. I will look forward to this while I try Wireshark and commenting that line in the .cpp

xabierolaz commented 4 years ago

Tried to run the docker, but my virtual machines docker engine seems unable to run even a simple hello word ,

cuda@eim-alu-83252:~$ sudo docker run hello-world docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"proc\\\" to rootfs \\\"/var/lib/docker/overlay2/e80f3eab4e46d1de8e110f0674df5b39b7aba8e6e647def2bc5455fd9d2dc671/merged\\\" at \\\"/proc\\\" caused \\\"permission denied\\\"\"": unknown. ERRO[0000] error waiting for container: context canceled so I am starting from scratch with a physic computer instead