search

LORD-MicroStrain / microstrain_inertial

ROS driver for all of MicroStrain's current G and C series products. To learn more visit
https://www.microstrain.com/inertial
102 stars 75 forks source link

Invalid NMEA sentences and RTCM write error #332

Open taskbjorn opened 6 months ago

taskbjorn commented 6 months ago

Describe the bug On my setup, NMEA sentences published by the GQ7 are occasionally empty or do not pass the checksum. This usually happens after some time (ranging from minutes to hours, but happens more often with high NMEA data rates), the NTRIP client log shows either of these two blocks of warnings, relating to:

Some time after the first warnings appear, I can read a truncated error message from the microstrain_inertial_driver_node regarding a SIGHUP signal being received by the MIP SDK, and immediately after the following error message is flooded to the console:

[microstrain_inertial_driver_node-8] [ERROR] [1716317808.192041445] [gnss_ins]: Failed to write RTCM to device

From there onwards, the node is unable to process RTCM data and does not recover.

To Reproduce Steps to reproduce the behavior:

  1. Run the microstrain_inertial_driver_node lifecycle node from the microstrain_inertial_driver package.
  2. Run the ntrip_ros.py node from the ntrip_client package.
  3. Wait for some time (a few minutes to a few hours).
  4. Observe the ntrip_ros log for NMEA warnings.
  5. Observe the microstrain_inertial_driver_node for RTCM write errors.

Expected behavior I did not expect to read empty or corrupted NMEA sentences from the GQ7 AUX port. I also did not expect the ROS node to stop receiving RTCM data altogether with no attempt to recover.

Environment:

Modifications No modifications to source code, using the latest package ros-humble-microstrain-inertial-driver from the ROS Humble Hawksbill distribution.

Launch Parameters See attached params.yml file.

Additional context I am connecting to the GQ7 via two USB to DB9 cables, going from two USB 2.0 ports of my Linux machine running both ROS nodes to the MAIN and AUX ports of my GQ7. When ntrip_interface_enableand nmea_message_config are disabled , the sensor works perfectly.

I have tried the following:

  1. Switching to different ports did not make a difference.
  2. Changing baudrates on the MAIN and AUX ports did not make a difference.
  3. Removing all published NMEA sentences but one (gnss1_nmea_gga_data_rate) reduced the rate at which warnings appear, but did not eliminate the issue.
robbiefish commented 5 months ago

Hi @taskbjorn,

I am taking a look at this now, but am currently unable to reproduce this.

I will note that these logs

[ntrip_ros.py-9] [WARN] [1716296096.108954746] [ntrip_client]: Invalid NMEA sentence, not sending to server
[ntrip_ros.py-9] [WARN] [1716296867.293565107] [ntrip_client]: Found packet, but checksums didn't match
[ntrip_ros.py-9] [WARN] [1716296867.294513096] [ntrip_client]: Expected Checksum: 0x25B607
[ntrip_ros.py-9] [WARN] [1716296867.295463915] [ntrip_client]: Actual Checksum: 0x28B382

Are not very well formed, but are saying that we received an Invalid NMEA sentence, and then, we received an RTCM message from the NTRIP caster that has an invalid checksum.

This makes me suspicious that this problem could be with the NTRIP caster. Can you provide any information about the caster you are using? Obviously not the username and password, but other information would be helpful. If you are uncomfortable sharing that information on Github, please feel free to reach out to me via email

taskbjorn commented 5 months ago

Thank you for looking into this @robbiefish - I have sent you an email with more details on my NTRIP caster, but feel free to come back here if troubleshooting moves away from the NTRIP caster itself and if the conversation can be helpful to other users.