search

LORD-MicroStrain / microstrain_inertial

ROS driver for all of MicroStrain's current G and C series products. To learn more visit
https://www.microstrain.com/inertial
100 stars 75 forks source link

3DM-GQ7 reconnect issue with latest firmware on arm #197

Closed ghfue closed 11 months ago

ghfue commented 1 year ago

Describe the bug Attempting to run the driver a second time gives a failure of "Unable to communicate with device".

To Reproduce Steps to reproduce the behavior:

  1. Launch Node with command: ros2 launch microstrain_inertial_driver microstrain_launch.py configure:=true activate:=true

Output:

[INFO] [launch]: All log files can be found below <folder name>
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [microstrain_inertial_driver_node-1]: process started with pid [2732]
[microstrain_inertial_driver_node-1] [INFO] [1668797810.973011808] [microstrain_inertial_driver]: Using MSCL Version: 64.2.2
[microstrain_inertial_driver_node-1] [INFO] [1668797810.973597376] [microstrain_inertial_driver]: Attempting to open serial port </dev/ttyACM0> at <115200>
[microstrain_inertial_driver_node-1] [INFO] [1668797810.979025440] [microstrain_inertial_driver]: Setting to Idle: Stopping data streams and/or waking from sleep
[microstrain_inertial_driver_node-1] [INFO] [1668797811.004981472] [microstrain_inertial_driver]: 
[microstrain_inertial_driver_node-1]       #######################
[microstrain_inertial_driver_node-1]       Model Name:    3DM-GQ7
[microstrain_inertial_driver_node-1]       Serial Number: <serial number here>
[microstrain_inertial_driver_node-1]       #######################
[microstrain_inertial_driver_node-1]     
[microstrain_inertial_driver_node-1] [INFO] [1668797811.006662304] [microstrain_inertial_driver]: Setting Declination Source
[microstrain_inertial_driver_node-1] [INFO] [1668797811.010175200] [microstrain_inertial_driver]: Setting GNSS1 antenna offset to [0.000000, -0.700000, -1.000000]
[microstrain_inertial_driver_node-1] [INFO] [1668797811.014007264] [microstrain_inertial_driver]: Setting GNSS2 antenna offset to [0.000000, 0.700000, -1.000000]
[microstrain_inertial_driver_node-1] [INFO] [1668797811.018020256] [microstrain_inertial_driver]: Setting RTK dongle enable to 0
[microstrain_inertial_driver_node-1] [INFO] [1668797811.020923904] [microstrain_inertial_driver]: Note: The device does not support the vehicle dynamics mode command.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.021020352] [microstrain_inertial_driver]: Setting PPS source to 0X01
[microstrain_inertial_driver_node-1] [INFO] [1668797811.021908032] [microstrain_inertial_driver]: Note: The device does not support the heading source command.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.021986208] [microstrain_inertial_driver]: Note: The device does not support the filter autoinitialization command.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.022024704] [microstrain_inertial_driver]: Setting autoadaptive options to: level = 2, time_limit = 15000
[microstrain_inertial_driver_node-1] [INFO] [1668797811.024114880] [microstrain_inertial_driver]: Filter aiding gnss pos/vel = 1
[microstrain_inertial_driver_node-1] [INFO] [1668797811.026137376] [microstrain_inertial_driver]: Filter aiding gnss heading = 1
[microstrain_inertial_driver_node-1] [INFO] [1668797811.028123136] [microstrain_inertial_driver]: Filter aiding altimeter = 0
[microstrain_inertial_driver_node-1] [INFO] [1668797811.030141728] [microstrain_inertial_driver]: Filter aiding odometer = 0
[microstrain_inertial_driver_node-1] [INFO] [1668797811.032135712] [microstrain_inertial_driver]: Filter aiding magnetometer = 0
[microstrain_inertial_driver_node-1] [INFO] [1668797811.034139840] [microstrain_inertial_driver]: Filter aiding external heading = 0
[microstrain_inertial_driver_node-1] [INFO] [1668797811.038197920] [microstrain_inertial_driver]: Setting wheeled vehicle contraint enable to 0
[microstrain_inertial_driver_node-1] [INFO] [1668797811.040413408] [microstrain_inertial_driver]: Setting vertical gyro contraint enable to 0
[microstrain_inertial_driver_node-1] [INFO] [1668797811.042433920] [microstrain_inertial_driver]: Setting GNSS antenna calibration to: enable = 0, max_offset = 0.100000
[microstrain_inertial_driver_node-1] [WARN] [1668797811.047491136] [microstrain_inertial_driver]: Attempted to stream MIP field with descriptor 0x8214 at a rate of 10.000000 hz, but the device reported that it does not support it
[microstrain_inertial_driver_node-1] [INFO] [1668797811.050859072] [microstrain_inertial_driver]: Setting sensor2vehicle frame transformation with euler angles [0.000000, 0.000000, 0.000000]
[microstrain_inertial_driver_node-1] [INFO] [1668797811.051746624] [microstrain_inertial_driver]: Resetting the filter after the configuration is complete.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.054474720] [microstrain_inertial_driver]: Publishing raw IMU data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.120580576] [microstrain_inertial_driver]: Publishing Magnetometer data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.121051712] [microstrain_inertial_driver]: Publishing GNSS1 NavSatFix data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.121373216] [microstrain_inertial_driver]: Publishing GNSS1 odom data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.123012512] [microstrain_inertial_driver]: Publishing GNSS1 time reference data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.123264032] [microstrain_inertial_driver]: Publishing GNSS1 fix info data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.126664672] [microstrain_inertial_driver]: Publishing GNSS1 aiding status
[microstrain_inertial_driver_node-1] [INFO] [1668797811.126923360] [microstrain_inertial_driver]: Publishing GNSS2 NavSatFix data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.127187328] [microstrain_inertial_driver]: Publishing GNSS2 odom data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.127471616] [microstrain_inertial_driver]: Publishing GNSS2 time reference data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.127662656] [microstrain_inertial_driver]: Publishing GNSS2 fix info data.
[microstrain_inertial_driver_node-1] [INFO] [1668797811.127835136] [microstrain_inertial_driver]: Publishing GNSS2 aiding status
[microstrain_inertial_driver_node-1] [INFO] [1668797811.128007008] [microstrain_inertial_driver]: Publishing Filter status data
[microstrain_inertial_driver_node-1] [INFO] [1668797811.128332640] [microstrain_inertial_driver]: Publishing Filter heading message
[microstrain_inertial_driver_node-1] [INFO] [1668797811.128566560] [microstrain_inertial_driver]: Publishing Filter heading state message
[microstrain_inertial_driver_node-1] [INFO] [1668797811.128773248] [microstrain_inertial_driver]: Publishing Filter odometry message
[microstrain_inertial_driver_node-1] [INFO] [1668797811.129083168] [microstrain_inertial_driver]: Publishing Filtered IMU data
[microstrain_inertial_driver_node-1] [INFO] [1668797811.129343136] [microstrain_inertial_driver]: Publishing Dual Antenna Status message
[microstrain_inertial_driver_node-1] [INFO] [1668797811.139277600] [microstrain_inertial_driver]: Setting spin rate to <200.000000> hz
[microstrain_inertial_driver_node-1] [INFO] [1668797811.140269312] [microstrain_inertial_driver]: Resuming the device data streams
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[microstrain_inertial_driver_node-1] [INFO] [1668797952.948428608] [rclcpp]: signal_handler(signal_value=2)
[INFO] [microstrain_inertial_driver_node-1]: process has finished cleanly [pid 2732]
  1. Kill the node with a ctrl-c
  2. Relaunch the node with the same command: ros2 launch microstrain_inertial_driver microstrain_launch.py configure:=true activate:=true

Output:

[INFO] [launch]: All log files can be found below <folder name>
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [microstrain_inertial_driver_node-1]: process started with pid [2780]
[microstrain_inertial_driver_node-1] [INFO] [1668797956.910888832] [microstrain_inertial_driver]: Using MSCL Version: 64.2.2
[microstrain_inertial_driver_node-1] [INFO] [1668797956.911410752] [microstrain_inertial_driver]: Attempting to open serial port </dev/ttyACM0> at <115200>
[microstrain_inertial_driver_node-1] [INFO] [1668797956.915561792] [microstrain_inertial_driver]: Setting to Idle: Stopping data streams and/or waking from sleep
[microstrain_inertial_driver_node-1] [WARN] [1668797957.166696480] [microstrain_inertial_driver]: Unable to communicate with device. Waiting a second before setting to idle again, as the device may just be streaming
[microstrain_inertial_driver_node-1] [INFO] [1668797957.667370400] [microstrain_inertial_driver]: Setting to Idle: Stopping data streams and/or waking from sleep
[microstrain_inertial_driver_node-1] [WARN] [1668797957.918024544] [microstrain_inertial_driver]: Unable to communicate with device. Waiting a second before setting to idle again, as the device may just be streaming
[microstrain_inertial_driver_node-1] [INFO] [1668797958.418596928] [microstrain_inertial_driver]: Setting to Idle: Stopping data streams and/or waking from sleep
[microstrain_inertial_driver_node-1] [WARN] [1668797958.669483008] [microstrain_inertial_driver]: Unable to communicate with device. Waiting a second before setting to idle again, as the device may just be streaming

Expected behavior Successfully connect.

Environment (please complete the following information):

Modifications No source code modifications.

Launch Parameters No modified params.

Additional context I was able to successfully reconnect with a 3DM-GQ7 with a prior firmware with x86, so I'm not sure if it is firmware or architecture related.

vvel-tracpilot commented 1 year ago

We are experiencing the same issues with a 3DMGX5-AHRS:

Running the same version of the node and the same sensor on a different machine does not show this behavior:

We are building the node from source without any modifications to the source.

robbiefish commented 1 year ago

This is interesting behavior. @ghfue, are you also using a Nvidia Jetson Orin? We have heard from others that there has been trouble connecting a GQ7 (on old firmware) to an Orin.

I unfortunately do not have the ability to test this right now, as I do not have an Orin. Could you try running the following debug steps?

  1. After the first successful run of the driver, does the device file still appear to be streaming data? You can test this by running cat /dev/ttyACM0, or xxd /dev/ttyACM0. If either command prints data to the screen, the device is still streaming
    1. If the device is streaming after the first stop, does it continue streaming after the second attempt to run the driver? It should not if the setToIdle command succeeded
  2. If you follow dmesg with dmesg -w as you kill the node the first time, do you see any alarming logs related to the device or USB?
ghfue commented 1 year ago

Yes, this is on a Nvidia Jetson Orin with Jetpack 5.0.2 (thanks @vvel-tracpilot for including the additional information).

  1. After successfully running the driver, the device is still streaming data. i. It does not continue streaming after the second attempt to run the driver.
  2. dmesg doesn't show any alarming logs when starting or stopping the node.
vvel-tracpilot commented 1 year ago

We are rebuilding our setup right now, so i cannot run any new tests at the moment. But from the testing I already did before I saw the same as @ghfue re dmesg output, though I will check again when we get everything up and running again.

vvel-tracpilot commented 1 year ago

We have everything set up again. Annotated dmesg output:

<< imu connected >>
[ 4043.127176] usb 1-4.4: new full-speed USB device number 29 using tegra-xusb
[ 4043.419174] usb 1-4.4: device descriptor read/64, error -32
[ 4043.643938] usb 1-4.4: New USB device found, idVendor=0483, idProduct=5740, bcdDevice= 2.00
[ 4043.643952] usb 1-4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 4043.643960] usb 1-4.4: Product: Lord Inertial Sensor
[ 4043.643966] usb 1-4.4: Manufacturer: Lord Microstrain
[ 4043.643972] usb 1-4.4: SerialNumber: 0000_6253.119680
[ 4043.646773] cdc_acm 1-4.4:1.0: ttyACM0: USB ACM device
<< ros driver launched >>
<< ros driver terminated >>
<< ros driver launched >>
<< ros driver failed >>
<<imu disconnected >>
[ 4200.857288] usb 1-4.4: USB disconnect, device number 29
github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 2 weeks with no activity. If the issue is still not resolved, please leave a comment describing what is still not working

ghfue commented 1 year ago

For the bot, the issue is still unresolved.

ghfue commented 1 year ago

Looks like this was already reported on #172 but got closed due to being stale. This was on a Xavier running Noetic, so still Ubuntu 20.04 but not on the Orin.

tocococa commented 1 year ago

@ghfue I also had issues running a 3DM-GQ7 from an Nvidia Jetson Nano (see issue #171). I ended using a Rpi3 as a bridge with the MSCL library and a ROS node written by myself, as I could not get the Jetson to communicate reliably with the 3DM-GQ7.

Acerox41 commented 1 year ago

Same behavior here using the Jetson AGX Orin (Ubuntu 20.04, jetpack 5.1-b147) and 3DM-GQ7 (current firmware). After first successful roslaunch of the node, device does not streaming after Ctrl-C. Second roslaunch results in attempting to open the serial port at all available baudrates until it fails: MIP SDK: Failed to poll serial port (4): Interrupt system call. [microstrain_inertial_driver-2] escalating to SIGTERM

In the end I have to replug 3DM-GQ7.

Any advices?

Acerox41 commented 12 months ago

Problem for Jetson Devices is solved with update to r35.4 release. See here: https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html#package-management-tool It fixes a bug related to UART receiving data in the L4T 35.3.1 release.

jeaninevbrr commented 11 months ago

Thanks @Acerox41, that worked :) I was still at 35.2.1 having the exact same issue.