search

bluerobotics / BlueOS-Water-Linked-DVL

MIT License
4 stars 12 forks source link

ArduSub timeouts when rangefinder value > 2.5m #44

Open clydemcqueen opened 3 months ago

clydemcqueen commented 3 months ago

I have noticed that RFND.Dist values jump between something reasonable and 0 around when the ROV is > ~2.5m away from the seafloor.

When the ROV is close to the seafloor, the driver is sending DISTANCE_SENSOR messages at ~9Hz. Above ~2.5m the rate drops to ~4.5 Hz, so the gap between messages increases to ~220ms. This is easy to spot in the logs.

image

What is surprising is that the RNFD.Dist values show periodic 0 values when this happens, which AFAICT only happens if the 500ms timeout is hit.

https://github.com/ArduPilot/ardupilot/blob/ecd9694e47678384b083823c09899d4232c4cfa1/libraries/AP_RangeFinder/AP_RangeFinder_MAVLink.cpp#L79

Example RFND graph:

image

If the DISTANCE_SENSOR messages are arriving every ~250ms, this should never happen. I suspect that some DISTANCE_SENSOR messages are delayed by up to ~250ms, so a 500ms gap appears every ~7.5s. I do not have enough data to pinpoint the source of this delay.

Am I off-track here? Have others seen this behavior?

clydemcqueen commented 3 months ago

Using MAVExplorer.py on the tlog file:

MAVExplorer.py '2024-08-21 14-10-00.tlog'
condition DISTANCE_SENSOR._header.srcSystem==255
graph diff(DISTANCE_SENSOR._timestamp,0)

(Graph is zoomed into the same region as above.)

This clearly shows periodic delays in DISTANCE_SENSOR messages from the driver recorded in QGC. These periodic delays are present at all depths, but the delay is significant when the ROV is > 2.5m above the seafloor. I suspect that a similar delays are also experienced by the autopilot, which is why I am seeing periodic timeouts.

image

clydemcqueen commented 2 months ago

This also affects VISION_POSITION_DELTA messages, which makes sense.

I am seeing this in logs on both DVL-equipped ROVs in the Seattle Aquarium fleet. So, I suspect that this is a bug in the extension or BlueOS somewhere. I'm curious if others are seeing this?

Not critical, but a puzzle....

Williangalvani commented 2 months ago

Hi @clydemcqueen, I suspect this issue is caused by mavlink-router, and that it is related to the "MyGCS 255: Heartbeat lost". Unfortunately mavp2p support is problematic right now, but we are also working on our own router which should fix this issue.

clydemcqueen commented 2 months ago

@Williangalvani LMK if you need help testing