TSC21
commented
7 years ago /dev/ttyACM0:57600 -> try to increase to baudrate to 921600. Also set SERIAL1_BAUD to 921.
Closed gui2dev closed 6 years ago
TSC21
commented
7 years ago /dev/ttyACM0:57600 -> try to increase to baudrate to 921600. Also set SERIAL1_BAUD to 921.
gui2dev
commented
7 years ago Thanks for the answer. I just tried it, without success. And I guess it's SERIAL0_BAUD that I should change as I'm connected to my pixhawk 2 USB port ?
gui2dev
commented
7 years ago Putting waypoint plugin output level to DEBUG, I got something like this:
[DEBUG] [1506422968.584678109]: WP: send item #232 F:0 C:115 p: 135.000000 0.000000 -1.000000 0.000000 x: 0.000000 y: 0.000000 z: 0.000000
[DEBUG] [1506422968.593246869]: WP: FCU requested waypoint 232
[DEBUG] [1506422968.593594694]: WP: send item #232 F:0 C:115 p: 135.000000 0.000000
-1.000000 0.000000 x: 0.000000 y: 0.000000 z: 0.000000
[DEBUG] [1506422968.598412674]: WP: FCU requested waypoint 233
[DEBUG] [1506422968.598799423]: WP: send item #233 F:3 C: 16 p: 0.000000 0.000000 0.000000
0.000000 x: 45.051233 y: 5.800261 z: 100.000000
[DEBUG] [1506422968.604025764]: WP: Received INVALID_SEQUENCE ack
[DEBUG] [1506422969.610167177]: WP: FCU requested waypoint 234
[DEBUG] [1506422969.610542945]: WP: send item #234 F:0 C:115 p: 135.000000 0.000000
-1.000000 0.000000 x: 0.000000 y: 0.000000 z: 0.000000
[DEBUG] [1506422969.616371093]: WP: FCU requested waypoint 234
[DEBUG] [1506422969.616760118]: WP: send item #234 F:0 C:115 p: 135.000000 0.000000
-1.000000 0.000000 x: 0.000000 y: 0.000000 z: 0.000000
[DEBUG] [1506422969.621158504]: WP: FCU requested waypoint 235
[DEBUG] [1506422969.621405778]: WP: send item #235 F:3 C: 16 p: 0.000000 0.000000 0.000000
0.000000 x: 45.051282 y: 5.800336 z: 100.000000
[DEBUG] [1506422969.626836776]: WP: Received INVALID_SEQUENCE ackSo my first guess is that's transfer is somehow ok, but just very slow.
vooon
commented
7 years ago Lets calculate, 605 waypoints it is: 605 (MISSION_INTEM + MISSION_REQUEST) + MISSION_REQUEST_LIST + MISSION_COUT bytes to exchange.
605 * (38 + 10 + 5 + 10) + 3 + 10 + 5 + 10 = 38143 bytes, for 115200 you can pass about 10 kiB/sec.
So transmission takes at least 3 sec. Hm that's look okay, 10x less than default timeout (30 sec).
Launch htop and look on CPU load.
vooon
commented
7 years ago Double requesting of same wp item seems invalid, check wiring.
TSC21
commented
7 years ago 115200 you can pass about 10 kiB/sec.
Is serial connection is set to 57600.
TSC21
commented
7 years ago Thanks for the answer. I just tried it, without success. And I guess it's SERIAL0_BAUD that I should change as I'm connected to my pixhawk 2 USB port ?
Try both.
gui2dev
commented
7 years ago Actually, it does not seems to be linked with connectivity at whatever speed I have the same issue while connected to the joule, but not when connected to another machine (my laptop).
It's really as if each first try of sending WP times out.
Setting WP_TIMEOUT_MS to 25 let me upload all the waypoints with only one timeout.
What I also don't understand is that the waypoint plugin seems to be not happy with my first WP (takeoff) as I got this INFO:
[ INFO] [1506423994.490924413]: WP: item #0 F:0 C: 16 p: 0.000000 0.000000 0.000000 0.000000 x: 45.051647 y: 5.801180 z: 328.070007
When my first waypoint should be a takeoff:
currentWps.clear(); [...] mavros_msgs::Waypoint wpTakeOff; wpTakeOff.frame = 3; wpTakeOff.command = 22; // MAV_CMD_NAV_TAKEOFF; wpTakeOff.param4 = 0.0; wpTakeOff.x_lat = 0.0; wpTakeOff.y_long = 0.0; wpTakeOff.z_alt = global_alt; wpTakeOff.autocontinue = true; currentWps.push_back(wpTakeOff);
TSC21
commented
7 years ago @JMare could any of your changes have influence here?
gui2dev
commented
7 years ago The only think I added to the original package is some lines that calls a service with the waypoint sequence number on ITEM_MISSION_REACHED form the the appropriate handler
JMare
commented
7 years ago So @gui2dev first off waypoint 0 is home position so your takeoff waypoint should be in position 1.
This is curious. The handling of WP: Received INVALID_SEQUENCE ack is something that we added in #798 , but the previous behaviour was to abort the transfer in this circumstance.
And the order of the debug messages seems strange here.
@gui2dev you mention you have modified the wp plugin, can you show us the branch you are working off?
Its worth noting that parameter and waypoint transfers are surprisingly intolerant of any significant link loss. From my testing even 10% link loss makes significant transfers basically impossible.
Are you able to show us the code that is calling the WaypointPush service?
[ WARN] [1506408178.311390820]: WP: timeout, retries left 2
[ WARN] [1506408180.320554139]: WP: timeout, retries left 2
[ WARN] [1506408182.331194313]: WP: timeout, retries left 2
[ WARN] [1506408184.343025752]: WP: timeout, retries left 2
[ WARN] [1506408186.354143266]: WP: timeout, retries left 2
[ WARN] [1506408188.365744648]: WP: timeout, retries left 2
[ WARN] [1506408190.375305900]: WP: timeout, retries left 2
[ WARN] [1506408192.383309896]: WP: timeout, retries left 2
[ WARN] [1506408194.392562067]: WP: timeout, retries left 2I don't know why this is happening but it looks bad.
@JamesStewy what do you think of this
JamesStewy
commented
7 years ago the order of the debug messages seems strange here.
@gui2dev, do the DEBUG log messages consistently follow the pattern you posted? The pattern being:
Its worth noting that parameter and waypoint transfers are surprisingly intolerant of any significant link loss. From my testing even 10% link loss makes significant transfers basically impossible.
I also have to echo this. We changed the waypoint module to ignore the INVALID_SEQUENCE ack to increase the fault tolerance, as, despite the retry mechanism implemented at both ends, dropping a single message could cause the waypoint transfer to fail completely. Just looking at that small snippet of debug logs suggests there are messages that are failing to be sent. If this test is run on a mavros version without our changes I would suspect the transfer would fail long before you get to the 232nd waypoint.
gui2dev
commented
7 years ago @jmare I issued a pull request -- #826
@JamesStewy the pattern is somehow the same, sometimes one waypoint is accepted without receivingINVALID_SEQUENCE ack. I first worked on a version that would not include the INVALID_SEQUENCE ignore. At that time transfer just failed after first waypoint.
After updating my sources, transfer would just timeout on LIST_TIMEOUT_MS, so I started to play with the timeouts duration to allow a full transfer.
Increasing LIST_TIMEOUT_MS and WP_TIMEOUT_MS would do it, but it was awfully slow (5 minutes or more) and sometimes causes an HEARTBEAT drop. Lowering WP_TIMEOUT_MS to 25 is my best option so far.
As it only happens with the intel joule but not when using SITL, I am starting to think that it's somehow kernel-related and how /dev/ttyACM0 is managed but that sounds pretty odd.
JMare
commented
7 years ago @gui2dev It seems more likely to me that the issue only occurs on low powered computers, than it being specific to the joule.
If you share the code you are using to call the WaypointPush service from I could test it on a Raspberry Pi or Odroid and see if I can replicate.
gui2dev
commented
7 years ago @JMare that's the other option. You can get the code here: https://github.com/gui2dev/mavros
lamping7
commented
7 years ago Just to share my experience, our team had no issues when we used the Joule in the past with:
We were using a Pixhawk 2.1 connected via UART on the GPIO @ 921600 baud. We didn't push nearly as many WPs though.
gui2dev
commented
7 years ago @lamping7 Did you use pullups?
lamping7
commented
7 years ago Did you used pullups?
@gui2dev I'm not sure what you're asking. I used pins 22/24 for TX/RX and pin 36 for GND.
gui2dev
commented
7 years ago @lamping7 GPIO levels are at 1.8V right? Or maybe you were not using the dev board.
lamping7
commented
7 years ago @gui2dev From my understanding, the Joule itself is 1.8v, but the dev kit expansion board is 3.3v. We didn't use a level shifter, like one you'd use for the ODROID XU4, for example. We have the dev kit, if that wasn't clear.
EDIT: Yup. See -> https://communities.intel.com/thread/111850
gui2dev
commented
7 years ago @lamping7, so we have a similar use of the joule + dev kit except you're using serial while I'm using usb. We are using gpio 55 (j13 15) for trigging purposes and high level is definitely 1.8V. Maybe 22/24 is 3.3V.
lamping7
commented
7 years ago @gui2dev Checkout the Intel post above. We didn't use any of the other GPIO, but again, we had no communication issues the way we had it setup once we figured out how to do it. You need the Ubuntu flavor packed with the Joule specific kernel in order to use the UART on /dev/ttyS1 for the pins I listed previously.
Another, but unrelated, issue that we found was that using a USB type C hub (we tried two different manufacturers) connected to the Joule introduced odd EMI to the system, causing GPS signal loss and compass errors.
gui2dev
commented
7 years ago @lamping7 Thanks for the link. I relied on an intel reference for GPIO Signal Termination and Conditioning We are using the type A with a usb hub and we did not observe EMI issues (pixhawk 2.1).
gui2dev
commented
7 years ago @lamping7 I am also running a joule flavored kernel (latest I guess) along with latest bios.
lamping7
commented
7 years ago @gui2dev No problem. The Joule is known for terrible documentation. I'd see if your problem exists with that hardware interface. Should be a quick test if you have the correct software, which you say you do. Good luck.
Also, we switched to the type A and had no issues with interference.
gui2dev
commented
7 years ago @lamping7 allright. By any chance did you had any connectivity issue when using usb connection to pixhawk ?
lamping7
commented
7 years ago @gui2dev We used a USB FTDI serial converter connected from the Joule USB A port to the telem2 on the Pixhawk at first and had no issues @ 921600 baud.
TSC21
commented
7 years ago @gui2dev is this solved to you already?
gui2dev
commented
7 years ago sorry @TSC21 I've been too busy to make tests again. It really looks as if it's power related, I'll confirm asap, but feel free to close the issue if I'm taking too long.
TSC21
commented
7 years ago We can wait some days. We just usually don't like to keep the bug tracking stalled for long (except with feature requests).
TSC21
commented
6 years ago @gui2dev I'm closing this as stale. Let us know if this still persists.
Issue details
I'm using mavros on an intel joule. I am experiencing very high latency and timeouts with waypoints and param plugins. My first guess is that it's hardware related, because changing the computer that executes the launch file seems to solve the issue. In addition to that, uploading waypoints from UGS seems flawless. For now my only way around that issue is to increase LIST_TIMEOUT_MS to 600 000 to be able to load a waypoint table of about 605 elements.
MAVROS version and platform
Mavros: 0.21.2 ROS: Kinetic Ubuntu: 16.04
Autopilot type and version
[X] ArduPilot [ ] PX4
Version: 3.5
Node logs
Diagnostics
Check ID