Failed to communicate with the device

[Jaetriel]

Hello, I am trying to use the Microstrain 3DMGX5-25 IMU with a UART connection on an Nvidia Jetson Xavier AGX host machine but when I run the ROS node I get the following error:

[ INFO] [1620302780.977140772]: Attempting to open serial port </dev/ttyTHS0> at <921600> [FATAL] [1620302781.235342058]: Error: Failed to communicate with the device. terminate called after throwing an instance of 'mscl::Error_Communication' what(): Failed to communicate with the device. [gx5/ros_mscl_node-2] process has died [pid 7780, exit code -6, cmd /home/gl/catkin_ws/devel/lib/ros_mscl/ros_mscl_node name:=ros_mscl_node log:=/home/gl/.ros/log/77015876-ae63-11eb-a1af-02426bc5af84/gx5-ros_mscl_node-2.log]. log file: /home/gl/.ros/log/77015876-ae63-11eb-a1af-02426bc5af84/gx5-ros_mscl_node-2*.log

I have tried using the default baudrate of 115200 and this 921600( after checking with stty -F /dev/ttyTHS0 command that the port supports 921600 baud rate ) but I still cannot get the ROS node to launch, I tried it yesterday with a USB connection and it was working as expected.

Edit: I have tried using software to stream the output on the serial ports both when using the UART pins on the Jetson Xavier and when using just a USB connection (the software was PuTTy) and both times I get seemingly the same data ( obviously unreadable) at 115200 baud rate, so an actual connection is established but the ROS node still doesn't run using the UART connection

[nathanmillermicrostrain]

Hi,

It is possible that you have configured the message output to overrun the 115200 baud rate. Do you have a windows computer with our SensorConnect software installed (it can be found here https://www.microstrain.com/software/sensorconnect)?

The ROS driver can be configured to save the settings in the device and, depending on the data rate that you selected, you could be saturating the available baudrate (115200). If the ROS node can't do command and control due to this issue, you can get problems like you are seeing above. You can use the SensorConnect software to remove the message format and save the settings, this will ensure it doesn't output anything at power-on, then try to connect it to the Xavier and run the ROS driver again. I would caution that you need to take into account the baudrate when setting the data rates you want from the driver. Using SensorConnect you can also change the baudrate of the device to 921600 and save it, so that you can stream at a higher rate in the driver.

Let me know if that fixes your issue!

Nathan

[Jaetriel]

Hello Nathan,

I am almost 100% sure that the serial port on the Jetson can support up to 921600 baud rate, and I tried that as well with the same results, but I did go into SensorConnect on a Windows machine and set the baud rate to 115200 manually but that still didnt change a thing. I also tried setting the IMU to not stream data immediately on startup as I saw in one of the previous issue that that was a fix for somebody but it didn't work for me either.

Edit 2: I did some more testing, and I am able to read the data from the IMU over serial connection on my laptop with Realterm I tried as binary, hex, ascii etc. It also shows up as ascii on the Nvidia Jetson so I guess it's okay, so I thought maybe the problem is the Jetson transmitting data, but connecting it through serial to the laptop I was also able to send and receive data with Realterm and PuTTy, the only idea I have now is that it might be wrong encoding/format of the data being sent from the Nvidia Jetson to the IMU perhaps or something else I am missing about sending the data to the IMU

[nathanmillermicrostrain]

Hmmm... it's unfortunate that those steps didn't solve your problem. Here's a couple of things to try:

1. You can try to extend the command timeout by adding the following line of code at line 310 of ros_mscl/src/microstrain_3dm.cpp:

   //Set the command timeout to 1 second m_inertial_device->timeout(1000);

2. If that doesn't work, let's make sure you can command the device over serial. Connect with a terminal program that allows you to send raw hex and try the following:

• Send this hex (an idle command): 75 65 01 02 02 02 E1 C7
• You should get the following back from the device: 75 65 01 04 04 F1 02 00 D6 6C

You should see that the device LED is pulsating as well. If you don't get a response, then something might be wrong with the serial cable TX line to the device or some other hardware or computer-side configuration. If you have a second serial cable to test in this case, that would be helpful. If not, you could buzz-out the cable using the following as a guide (note: you will be buzzing the mating cable to our device, so the pins will be mirrored):

If you do get the correct response, then it is a bit puzzling that the driver isn't working. Let's start with these steps and see where that takes us and we'll look at other options if needed.

[Jaetriel]

Hello, @nathanmillerparker thank you for the reply. I added the code you suggested to the node but that changed nothing for me, just getting the same response after some delay. As for transmitting the hex commands, I tried a few things. I found out that the UART pins on the Nvidia Xavier we are using only receive and transmit 3.3V and I can only give the IMU 5V of power, which I am starting to suspect that when using a serial connection the IMU needs more than 5V of power supplied to it? The only way I was able to transmit a hex command to the unit was with an RS232-USB connector which provided a higher RX-TX voltage of around 10V while I was still using 5V power to the pin 3 of the IMU so our only assumption at this point is that we don't have enough power, but I'm not sure if we just provide more power on pin 6 to the IMU would that be sufficient to have enough voltage on RX and TX as well or we probably need some converter on the Nvidia to step its RX and TX pins from 3.3V

Also just to be sure, when using the serial communication, do we absolutely need an external power source or can we connect it to a 5V of the Nvidia?

[nathanmillermicrostrain]

Hello,

Yes, you cannot drive the serial lines of the GX5-25 with TTL-level serial, it requires RS232-level signals. You would need an RS232 level shifter between your TTL lines and our device. The 5V power to the device is fine, it is the signal level that is the issue.

You could use one of the following links to purchase a small level shifter (note: these were taken from a google search):

https://www.amazon.com/MAX3232-Connector-Converter-Equipment-Upgrades/dp/B07PFB4MHR/ref=pd_lpo_147_t_0/137-3424728-7287655?_encoding=UTF8&pd_rd_i=B07PFB4MHR&pd_rd_r=0a7924f9-5947-47b6-b9c3-aac87881001d&pd_rd_w=MSmT9&pd_rd_wg=E3tUc&pf_rd_p=a0d6e967-6561-454c-84f8-2ce2c92b79a6&pf_rd_r=S7FSN657S8P8PP09EFV7&psc=1&refRID=S7FSN657S8P8PP09EFV7

https://www.sparkfun.com/products/449

[Jaetriel]

I saw on the pinout for the Nvidia Jetson that the Rx and Tx lines work at 3.3V, am I correct in assuming that the Rx and Tx for the Microstrain need at least 5V since they are RS232? Which would mean if we use the level shifter to 5V it should be fine

[nathanmillermicrostrain]

You will need an RS232 level shifter as you have to go from 0V logic low, 5V logic high to -3 to -15V logic low, +3 to +15v logic high. 0V is not a valid logic low level in RS232:

https://en.wikipedia.org/wiki/RS-232#Voltage_levels

[Jaetriel]

Thank you for being so helpful @nathanmillerparker , I'll take a look at the links you shared and will report back if the level shifter solves the issues

[nathanmillermicrostrain]

No problem... let me know how it goes!

[Jaetriel]

So we managed to get a hold of the level shifter, connected it and now I can again see that I receive messages from the IMU ( indecipherable for me, probably some sort of ASCII i assume ), I also tried sending it hex commands via the command line to set to idle and resume and those work as well. When trying to launch the ROS node though, we still get some errors and weird behaviour. Sometimes we still just get the 'Failed to communicate' error, but the code also executed a bit further and managed to print out a few more things

[ INFO] [1620744131.840086865]: Attempting to open serial port </dev/ttyTHS0> at <115200> [ INFO] [1620744131.881445507]: Model Name: 3DMGX5-AHRS [ INFO] [1620744131.881629483]: Serial Number: 6253.119680 [ INFO] [1620744131.904563643]: Setting to Idle: Stopping data streams and/or waking from sleep [ INFO] [1620744131.904854088]: Setting IMU data to stream at 500 hz [ INFO] [1620744131.912631560]: Setting Declination Source [FATAL] [1620744132.167130580]: Error: Failed to communicate with the device. [gx5/ros_mscl_node-2] process has finished cleanly log file: /home/gl/.ros/log/10882e0a-b267-11eb-9c3d-009e997002a0/gx5-ros_mscl_node-2*.log

After this I tried changing the device_setup parameter to use the default settings of the device and this was the result:

[ INFO] [1620744367.323939886]: Attempting to open serial port </dev/ttyTHS0> at <115200> [ INFO] [1620744367.360626861]: Model Name: 3DMGX5-AHRS [ INFO] [1620744367.360929818]: Serial Number: 6253.119680 [ INFO] [1620744367.394750743]: Publishing IMU data. [ INFO] [1620744367.397020222]: Publishing Magnetometer data. [ INFO] [1620744367.399165535]: Publishing Filter data. [ INFO] [1620744367.449077333]: Setting spin rate to <1000> [ INFO] [1620744367.461594508]: Starting Data Parsing

The topics were published but when I subscribed and even tried rostopic hz there were no messages being delivered.

[nathanmillermicrostrain]

Ok, so this is actually good progress. One of the reasons that you may be getting the "failed to communicate" error is that 115200 is insufficient to send all of the IMU data at 500 Hz (not taking into account the filter data.) If you need that data rate, you will need to increase the baudrate.

When you disabled device_setup, the driver started running successfully (which is a good thing), but because the device did not have a stored data format, you weren't getting any data. When device_setup is enabled, the driver sets up the messages from the device that are required to populate the output ROS topics.

So, the only issue is why the driver seemed to fail after the "Setting Declination Source" line above. Can you shoot over your launch file so that I can test here?

[Jaetriel]

Please try this link for a zip of my launch file: link to google drive

What you said about the device_setup parameter makes total sense, maybe it would be a good idea if I can somehow store parameters on the device itself that I set up from SensorConnect on a Windows device that would tell it how to format and output the data, but either way it would be nice to get the node running in both modes. After zipping the launch file I tried reducing the frequency down to 100hz for raw and 10hz for filtered data but the results seemed to be the same, I will also have a look at increasing the baud rate. It might also be worth mentioning that I made some changes to the code according to your recommendations in #52 to get filtered data but with compensated acceleration values.

[nathanmillermicrostrain]

Ok, so I was able to get your launch file up and running without a problem using an RS232-USB serial converter to a Linux laptop, so you should be able to use the settings you have (although note that you will likely be dropping data at 115200.) I'm not sure exactly what the issue would be at this point, but it is likely related to the hardware that you have. I know several people have been successful in getting Jetson devices up and going, so it's not an issue of if this should work, just something isn't quite right. Is the GND common to the RS232 level shifter and the Jetson?

[Jaetriel]

There is something slightly weird going on, we got a USB-RS232 serial converter as well and it worked on the Jetson just fine with the default 100hz/10hz rates and 115200 baudrate but with our serial connection going to the jetson's UART pins with the level shifter we still fail to load the node even though we are able to read from the port and send hex commands, the only thing that happens when I launch the node is that the IMU goes from transmitting to idle for some reason even though the node just fails. The GND we are using for the level shifter comes from a pin on the Jetson and from the level shifter it goes to the IMU as well so it's all common. It seems that the only possible options left are software issues either on the Jetson side, or on the IMU side, are there any special options or parameters we have to set differently for the mscl library when we want to use a pure serial connection with the IMU and not go through USB-serial connectors?

[nathanmillermicrostrain]

There is nothing to change in the device or the driver when using serial, except for the baudrate (common to both) and the port (for the driver.) From everything above, it appears that the driver is able to communicate with the device up to a point, then it fails to communicate. Are you ok with using the RS232-USB converter or do you need to use the serial port? If you need serial, are there serial buffer sizes that you can configure on the Jetson? Are there any other low-level configuration options (bios-level) for the serial port that might affect this? If the serial port IRQ can't be serviced and the FIFO isn't big enough, it is conceivable that you would drop a byte, which could cause a communication error. I don't know enough about the Jetson's UART to know if this is a valid concern, but this can happen on microcontrollers that have a small receive FIFO that cannot be serviced due to other processing.

Unfortunately, the only other thing that comes to mind at this point is to scope the serial TX and RX lines and see if: 1) the voltages are within spec and 2) the bytes being transmitted are what you'd expect. It is tedious to do this, so I wouldn't recommend it if the converter will work for you.

[Jaetriel]

Unfortunately I think this saga has to continue for us because we would really like to be able to bypass going through the USB by using the serial connection. Yea it seems to me as well that there is an established communication which gets lost at some point, when trying to set the Declination Source. This could maybe mean that the data sent/received is corrupted or incomplete in some way? My guess at this point, because we are quite sure our wiring is okay, is that there's either some software/configuration step we are missing on the Nvidia ( granted that I have installed the MSCL library and that is enough to use the ROS node for the IMU without any additional drivers needed for linux and as you say nothing in the code needs to be changed except port and baudrate ) or just some mismatch or something we're missing on a deeper level with the RS232/UART situation and going between the two, because frankly I am not well-versed at all when it comes to these kinds of protocols.

[nathanmillermicrostrain]

So... I scrounged up a Jetson Xavier AGX, but it will take me a little bit to get everything configured. In the interim, if you figure it out please let me know. I'll let you know if I have success once I get everything together.

[Jaetriel]

Wow you are really amazing! Thank you for going through all the trouble, it will be very interesting to see what you find, I will inform you if anything changes on our side as well

[Jaetriel]

Kind of a side question, which may also be relevant, what are the default settings on the IMU for parity bits and flow control? the Jetson by default uses no parity and no flow control with only 1 stop bit, so is it possible to change those settings for the IMU?

[nathanmillermicrostrain]

It is the same for the device and there are no settings to change it on the device side.... no parity and 1 stop bit is the norm. My Jetson is set up, but I'm waiting on a piece of hardware to arrive to do my testing.... will let you know when I have some results!

[Jaetriel]

Hey, I haven't got much progress per se, just some research and observations.From what I can see looking at the hex being sent by the IMU using Realterm on my Windows machine, it looks like the packets can be variable in size ( which makes sense because for example one sensor might have lower output frequency ) but the messages are reasonably big. I will try to find out the buffer size on the Nvidia and see if i could even manipulate it somehow, but if it turns out that is the issue and we can't increase the buffer in any way, I guess the solution would be to code some sort of custom parser that would split up the messages coming from the sensor?

[nathanmillermicrostrain]

I have tried one of our CX5 devices (which has TTL I/O) and it worked fine on my test Xavier (I didn't change any low-level settings)... still waiting on the level shifter to test the GX5. Will let you know how that test goes.

[nathanmillermicrostrain]

Ok... I was able to get everything working using the Sparkfun converter, but it required setting the IMU data rate to 100 Hz. A better level shifter that is capable of 921600 would be needed if you wanted to push the rates higher. I've included a picture of the setup that I used:

[Jaetriel]

That is pretty amazing, but leaves me thinking that maybe we do have something wrong in our physical connection. Could you give me some more details as to how you’ve connected yours? More specifically I think your Tx from the Xavier goes to the Rx of the shifter and then the ‘null modem’ connection means that Rx from the shifter gets amplified and goes directly into Rx of the sensor and the Tx and Rx of the sensor also have the GND connected to them right? Also are you using an external power source for the IMU? Edit: disregard my comment about the GND connected to Tx and Rx I double checked that null modem connection is just a crossed Rx and Tx but Im guessing in your case it's mostly used as a male-male adapter? Also, by get it running you mean you managed to launch the ROS node as well without any issues I imagine? We'll get our hands on the SparkFun level shifter as well really soon to try that out, because currently we are using a different one:

[nathanmillermicrostrain]

Both DB9 cables are designed to be the "device side" and be connected to a computer, so the null modem cable allows them to talk to each other. The sparkfun converter is limited to 115200 baud and if you go above that, it errors at the same spot you were having trouble with using your level shifter. Did you try your level shifter with 100 hz IMU data, it might work?

And, yes, at 115200, with 100 hz data, the ROS driver ran as expected.

[Jaetriel]

Yea we tried the level shifter with 100 hz and lower IMU data at baud rates lower than 115200 as well but even at 1hz IMU data I get the same results as well. I will attach a picture of our setup and a diagram along with it

[nathanmillermicrostrain]

I'm not sure if this will help because I haven't looked into the requirements for the MAX232 level shifter, but I had to use the 3.3V from the Jetson for VCC on the Sparkfun converter for communications to work.

[Jaetriel]

We tried the 3.3V as well with the same results. Can I ask which version of the Jetson OS you are using have you installed any special drivers or made any configuration to it, and maybe what firmware version you use on your IMU that might be relevant as well?

[nathanmillermicrostrain]

The only thing that I did was disable nvgetty and rebooted as described here: https://www.jetsonhacks.com/2019/10/10/jetson-nano-uart/

The device firmware shouldn't have an effect on this particular issue. Let me know if you have success with this or a different level shifter... I think that is really the crux of the issue.

[Jaetriel]

Soo, in a weird twist of events, we managed to get the node working after setting the publish_gnss1 and publish_gnss2 params in the launch file to false, although I stll have to do further testing, but it's pretty weird cause I didn't think changes in the launch file would be necessary

[nathanmillermicrostrain]

Hmmm... that should have 0 affect on the ARHS product. The driver checks if the device supports GNSS data and requires that to be true along with the publish flags to be set to do any setup... that is really strange. So that I understand, you got it working at 100 or 500 Hz?

[Jaetriel]

We got it working at 100hz, and I had the same thought as you that it should be checking what features the device supports so I printed out the values of the bools support_gnss1 and support_gnss2 and they are both 0 so the if statements in the code don't even get triggered. We're trying to set it up and get it running on another Xavier right now just for reference

[Jaetriel]

So, we managed to get it working on a second Xavier as well, and I think I might have some sort of idea what the issue could be related to. I'm quite sure before we got the serial connection working on the first Nvidia we did a factory settings reset in SensorConnect, and before we did the reset I had set some Sampling settings in there so that I can visualize the data a bit in SensorConnect. Now I went back and set the Sampling channels again, like accelerometer,attitude etc to sample at max sampling rate 1khz and 500 hz and started getting the errors again, I turned off the sampling from the channels and now the node is running, I tried sampling from the channels in SensorConnect but at 100hz and it was okay again. Quite weird behaviour but perhaps there's a mismatch between the sampling commands the IMU had in its memory from SensorConnect ( or maybe the sampling rate more specifically ) and the commands coming from the code when I run the node? And sidenote by the way, do you know of any level shifters that would be capable of operating at a higher baud rate so we could put to at least like 500hz raw data output?

[github-actions[bot]]

This issue is stale because it has been open for 2 weeks with no activity. If the issue is still not resolved, please leave a comment describing what is still not working

[github-actions[bot]]

This issue was closed because it has been inactive for 2 weeks since being marked as stale. If the issue is still not resolved, please reopen the issue, and leave a comment describing what is still not working