Investigate xsens termination output of the wireless suit

[yeshasvitirupachuri]

Seems wireless suit termination is giving the issue observed in the demo as noted in https://github.com/dic-iit/lab-events-demos/issues/99#issuecomment-498228455.

The main problem at the moment is that when the suit is crashing we do not see any termination output. So, we need to investigate the xsens driver termination process. The termination at the xsens wearable device level need not be double checked as it is being tracked and fixed in https://github.com/robotology-playground/wearables/issues/37. We need to put debug code and check the termination of the xsens driver.

@lrapetti @kouroshD

[lrapetti]

Today, during the tests with @prashanthr05, @kouroshD experience the same termination with the wired suit.

[lrapetti]

This behaviour may be due to the stand-by mode described in the documentation:

This condition may be detected by the following the sensors LED status:

[GiulioRomualdi]

Today I tried with @kouroshD to run VisualStudio debugger to understand which is the cause of the crash. Unfortunately, when we run the debugger the Xsens SDK returns an error related to the impossibility of using a debugger.

@traversaro @lrapetti @Yeshasvitvs @kouroshD @diegoferigo

[lrapetti]

Unfortunately, when we run the debugger the Xsens SDK returns an error related to the impossibility of using a debugger.

I think the SDK is installed in release. I notice we also have a pendrive with written DEBUG, I am not sure about the content since it was never used, but maybe we have also the licence for the debug version of the SDK

[lrapetti]

Anyway, that condition happens after terminating not correctly the Xsens suit (wirless). I think the best procedure, is to send stopAcquisition command before terminating the device. In case it is not terminated correctly, the best procedure is to restart the station (unplug power and usb), and eventually turn off and on some of the sensor, in this way I think somehow the status is reset and you can notice it by the fact that when starting the device it will take a while again searching for the sensors and pairing them.

[diegoferigo]

If you want to debug software on windows (good luck!) keep in mind that ALL the linked libraries must be compiled in debug. This means you need to link against the debug version of the SDK.

[traversaro]

If you want to debug software on windows (good luck!) keep in mind that ALL the linked libraries must be compiled in debug. This means you need to link against the debug version of the SDK.

If the Find<cmake>.cmake script for using the SDK is done appropriately, this should be automatically handled by CMake. Unfortunately this does not seem the case for https://github.com/robotology/wearables/blob/master/cmake/FindXsensXME.cmake . For an example of Find<package>.cmake that handles correctly Debug and Release configurations, check https://github.com/robotology/ycm/blob/v0.10.4/find-modules/FindLibOVR.cmake#L83 .

[kouroshD]

I totally found three pendrive, one we use normally, one is written Release and another one Developer. Non of them was written DEBUG! As @diegoferigo mentioned all the libraries should be compiled in debug. But we cannot simply compile it in debug mode, also with RelWithDebInfo type it returns an error related to privacy/permission while running.

[lrapetti]

I totally found three pendrive, one we use normally, one is written Release and another one Developer.

I think the total number of pendrive is three, so there is no debug one.

[traversaro]

I totally found three pendrive, one we use normally, one is written Release and another one Developer.

I think the total number of pendrive is three, so there is no debug one.

Are you sure that the fact that the license key has the "Release" label is related at all with the "Release" compile configuration? Is is possible that with "Release" they just mean "Deployment"? Or perhaps instead "Developer" can be used with "Debug" libraries, and it is not a problem at all if the key does not have a "Debug" label?

[lrapetti]

Are you sure that the fact that the license key has the "Release" label is related at all with the "Release" compile configuration? Is is possible that with "Release" they just mean "Deployment"? Or perhaps instead "Developer" can be used with "Debug" libraries, and it is not a problem at all if the key does not have a "Debug" label?

No, I am not sure at all. We are currently using the one without any label, and I was just mentioning that I knew there were other two pen drive but I have no idea about the content and the usage (maybe they are not even related to the SDK but to the MVN Analyze software we no longer have the license).

[kouroshD]

Is our problem related to https://base.xsens.com/hc/en-us/community/posts/360000076634-MT-Manager-crashes-after-confirming-wireless-connection ?

[lrapetti]

Is our problem related to https://base.xsens.com/hc/en-us/community/posts/360000076634-MT-Manager-crashes-after-confirming-wireless-connection ?

Probably not since they seem to be using the MTV Analizer software, not directly the SDK, and anyway our SDK should be newer then the fixed version (of the software) they are pointing.

[lrapetti]

Note that in general I think we are experiencing crashing in two different moment. In both case the device stops without any message:

• accessing data: this is the failure I have mostly experienced, the suit start fine, calibrate and start acquisition. The failure happens when we try to access the data from the wearable device port (with yarp read ... or launching a device that read that port).
• start Acquisition: in this case the failure is experienced right after the startAcquisition command is passed to the rpc port.

In my opinion it can be that the two failure are related to the same problem, and it is just that in the first case the device is stucked until we try to access it, and then it crash.

[lrapetti]

Yesterday I did some tests and I have some notes:

termination A possible cause can be the wrong termination of the application and of the device.

• when we stop the device with the wired suit we hear the bip signal which indicates the device is being turned off. With the wireless suit instead, after stopping the device, the sensors and the station are still blinking as if they are measuring (fast symmetric on-off with the awinda station).
• I try to check then how the device is terminated, the first thing I have noticed is that cleanAndClose function terminating all the xsens processes is never called both using wireless and wired suit. I try to trigger the call to that function with the wirelss suit and I have verified that it disconnect all the sensors so that they are back in the scanning status (the first status they enter after removing them from the station), however this doesn't seem to avoid the temination.

debug I tried to add some debug code to identify the source of the failure (not properly running in debug since not available for xsens at the moment), given that it terminates without giving any error message (and neither it gives segmentation fault). This is the sequence of the call I see after start Acquisiiton:

• in startAcquisition in wearable device
• in startAcquisition in device driver
• start the acquisition in the xsens mvn
• at this stage we see the thread processDataSample running.
• pause the thread for 100ms (we should check if in case of wireless this wating time should be longer!)
• acquisition succesfully started in device driver
• the status of the sensors is set to OK
• acquisition succesfully started in wearable device.
• at this point it is called getSensors giving the following warning:

  [WARNING]XsensSuit : Selected sensor type ( 0 ) is not supported by XsensSuit
  [WARNING]XsensSuit : Selected sensor type ( 1 ) is not supported by XsensSuit
  [WARNING]XsensSuit : Selected sensor type ( 2 ) is not supported by XsensSuit
  [WARNING]XsensSuit : Selected sensor type ( 3 ) is not supported by XsensSuit
  [WARNING]XsensSuit : Selected sensor type ( 5 ) is not supported by XsensSuit
  [WARNING]XsensSuit : Selected sensor type ( 10 ) is not supported by XsensSuit
  [WARNING]XsensSuit : Selected sensor type ( 11 ) is not supported by XsensSuit
  [WARNING]XsensSuit : Selected sensor type ( 12 ) is not supported by XsensSuit
  [WARNING]XsensSuit : Selected sensor type ( 14 ) is not supported by XsensSuit

  that is due to the fact that the Xsens stream only some type of sensors (sensor type ordering can be retrieved from here).

• at this point, after the warning messages above, we observe the application crash.

[yeshasvitirupachuri]

A possible problem is with the sampling-rate parameter in the configuration file. The current value of 120 is ok for the wired suit as its maximum sampling rate is 240. However, for the wireless suit, the maximum sampling rate is 60. Check the suit specifications. I think we have been running the wireless suit always with the sampling-rate value set to 120. @lrapetti please confirm this. Possible changing it to 60 may prevent the termination.

[lrapetti]

I think we have been running the wireless suit always with the sampling-rate value set to 120. @lrapetti please confirm this. Possible changing it to 60 may prevent the termination.

If I remember correctly, even if we set it to 120, it is automatically setting the value to the maximum speed it can reach (or at least this what I remember to have seen from the device print out messages in https://github.com/robotology/wearables/blob/master/XSensMVN/src/XSensMVNDriverImpl.cpp#L426).

[yeshasvitirupachuri]

@lrapetti you are right, I double checked with both the suits and the maximum sampling rate is used in both the cases

Wired

Wireless

One observation here is that the sampling rate value of 120 we are setting from the configuration file is not passed correctly to the configuration of the driver.

[lrapetti]

One observation here is that the sampling rate value of 120 we are setting from the configuration file is not passed correctly to the configuration of the driver.

It was due to a small bug, fixed by https://github.com/robotology/wearables/pull/67

[GiulioRomualdi]

I was checking the code trying to understand what it is going on, and I found this: https://github.com/robotology/wearables/blob/cc7fb16e12ceceff66ae740f09939ba32ef89328/devices/XsensSuit/src/XsensSuit.cpp#L408-L409

Do you think it may cause problems if data are not available?

[GiulioRomualdi]

It seems there is an SDK also for linux. This will simplify the implementation and the testing for our lab.

@kouroshD @Yeshasvitvs @lrapetti @DanielePucci

[lrapetti]

It seems there is an SDK also for linux. This will simplify the implementation and the testing for our lab.

Cool! We should understand which version of the SDK is available, and the compatibility with the hardware/firmware we have. In particular the SDK version we are using is xsens MVN SDK 2018.0.3

[GiulioRomualdi]

It seems there is an SDK also for linux. This will simplify the implementation and the testing for our lab.

Cool! We should understand which version of the SDK is available, and the compatibility with the hardware/firmware we have. In particular the SDK version we are using is xsens MVN SDK 2018.0.3

As far as I understood the SDK can be easily downloaded from here. I think we can try to install a see what happen

[S-Dafarra]

This issue was almost causing a major failure during the scientific evaluation demo (https://github.com/dic-iit/lab-events-demos/issues/123). We have to use a miscalibrated suit to avoid risking the demo. During the preparation, this crash occurred several times. Since the wired suit seems broken, this issue now is highly blocking. How can we proceed? Is it worth testing the Linux SDK? Can we add thousands of prints to understand what crashes? Can we check what @GiulioRomualdi suggested in this comment?

[lrapetti]

Is it worth testing the Linux SDK?

I think it is worth testing it, even because it would be handy, in general, to have it working on Linux.

Can we add thousands of prints to understand what crashes? Can we check what @GiulioRomualdi suggested in this comment?

Most likely it is the point where it fails at the first acquisition, given that it seems not to be failing at during the startAcquistion (see https://github.com/robotology/human-dynamics-estimation/issues/412). We can double check it is the exact point, and then we should understand why data are not available from xsens driver (eventually contact them).