data preparation

[yonamoto]

Hello.

Thank you for sharing TLIO algorithm. I'm very interested in IMU and working on trying it out. I have a question about datasets. This github doesn't contain sample datasets, so I think I need to prepare them. However, I don't know the details of the required data well from the code.

Could you tell me what kinds of data need when learning and how to make these data more clearly?

[raabuchanan]

I am also trying to understand the data formats. I've started documenting it here: https://github.com/ori-drs/TLIO#file-formats

[yonamoto]

Thank you for replying and sending your documentations! I have some questions.

・Did you already run these codes by using your datasets? ・If you use public datasets, what kinds of datasets do you use?

I would like to know because I try to execute code by using public datasets.

[raabuchanan]

@yonamoto No I haven't yet run this code, I'm still working on understanding the data format. I am trying to use the Newer College Dataset If you would like to collaborate on this, you can get my email from my GitHub profile.

[CathIAS]

Thanks for the question. Yes, unfortunately the dataset and the pre-trained model are not available at this point. You would need calibrated IMU data and ground-truth odometry for training. You need raw IMU data and factory calibration for running the filter.

We've received quite a lot of inquiries regarding data format, it would be a good idea to have a documentation. @raabuchanan feel free to go ahead and create a pull request on the doc. I'll make additions onto it if anything is not clear. For the file formats, the following info could help (you can also add these into the PR).

my_timestamps_p.txt: VIO timestamps. Used to select data time range [t] Note: skipped first 20 frames imu_measurements.txt: IMU data, raw and calibrated using VIO calibration [t, acc_raw (3), acc_cal (3), gyr_raw (3), gyr_cal (3), has_vio] Note: has been interpolated evenly between images. Every timestamp in my_timestamps_p.txt will have a corresponding timestamp in this file (has_vio==1) evolving_state.txt: VIO evolving states at IMU rate. [t, q_wxyz (4), p (3), v (3)] Note: not interpolated, raw IMU timestamps. Every timestamp in my_timestamps_p.txt will have a corresponding timestamp in this file calib_state.txt: VIO calibration states at image rate (used in data_io.py) [t, acc_scale_inv (9), gyr_scale_inv (9), gyro_g_sense (9), b_acc (3), b_gyr (3)]

Thanks, Wenxin

[nbgit10]

Many thanks @CathIAS and @raabuchanan for your explanations. I can train and run both network and filter successfully.

I am still wondering about the coordinate systems in use. Are you having the raw data in a local (IMU) ENU frame or maybe NED or even ECEF still?

[CathIAS]

@nbgit10 The raw data is in the IMU local frame. The estimated states are in a gravity aligned frame, not necessarily north up.

[nbgit10]

Many thanks for your quick reply, @CathIAS. Makes sense that the IMU measurements are in local frame. What about the VIO reference data and the AHRS filter? In which coordinate system is position in VIO and which rotation is expressed by the quaternion?

I currently have my VIO data in an ENU frame initialized in the IMU at t=0. Hence my GT-position is relative to the starting point in ENU and the quaternion gives me the rotation from the IMU frame to this ENU frame. The AHRS filter estimates the same rotation. My understanding of the paper is that your referenced world frame is similar to such an ENU frame setup and the (local) gravity aligned frame can be obtained by rotation along the yaw axis. Is this understanding correct? Sorry for bothering you again with this.

[CathIAS]

That's correct. The only difference between your setup and the paper seems to be the yaw position relative to the geographic north. The algorithm can take any initial yaw and initial position, defined by the VIO estimates.

[smorantg2]

Hello, first of all thanks for the documentation.

My intention is to use the ADVIO dataset (https://github.com/AaltoVision/ADVIO) with your code, but despite all the information I'm still a little bit lost. I have ground truth (t, px,py,pz, qw,qx,qy,qz) and IMU raw (t, acc_raw(3), gyro_raw(3), magnetometer_raw(3)). timestamps of GT and IMU are not the same, but I could interpolate them at any frequency needed. I think it would also be correct to obtain ground truth velocity using position and timestamps.

My questions are:

1. Is there any way I could use TLIO code with this dataset? Without calibrations? After reading the docs I think with ADVIO I would be missing _calibstate.txt, atttitude.txt and the _calib part of imu_measurements.txt

2. Is it necessary to interpolate all t's to 1000Hz?

[raabuchanan]

Hi I'm not sure the answer to your first question, as the VIO algorithm I use gives me an estimate of the biases which I then use for training. However, I can say that it's working relatively well for me at 400 Hz IMU data.

[CathIAS]

Hi @smorantg2, thanks for your questions.

1. Yes, you can use the code with any dataset as long as you have the ground truth and IMU data. If you do not have the calibration, it would mean that you have to manually set the calibration to a default (e.g. zero), or estimate the calibration beforehand. Read the file reader to understand how calib_state.txt is being used, and replace with your calibration there. atttitude.txt is the ahrs attitude filter which is only used for comparison, which you don't need. You would want to write your own dataloader with your data format. You do need to interpolate the data to have the same timestamps, preferably equally spaced, and having the data for training as calibrated as possible.
2. You can use any frequency you like. You need to set the parameters in main_net.py correspondingly.

[Andrew001207]

Hi everyone, Could anyone please give me a hint? How does one compute VIO calibrated IMU data? In most datasets VIO is given as a sequence of positions and quaernions, representing heading of the object, so to get VIO - based IMU data, I have to compute accelerations, which led to such results by somehow taking a diff of this path? If it's correct, where do I look for any information related to this operation?

[smorantg2]

Hi @CathIAS thanks a lot for the help! I really want this code to work for my data.

I'm now trying with RoNIN dataset, so maybe it is easier for you to understand my problems. I'm using synced data, so I'm able to generate _my_timestamps.txt, imu_measurements.txt and evolvingstate.txt.. RoNIN provides some bias info about acce and gyro, so I've used it to get acce <-> acce_uncalib and gyro <-> gyro_uncalib. Therefore the final data.hdf5 file lacks _integration_q_wxyz_s, filter_q_wxyz_s and offline_calibs. (To do so I've commented lines 157:185 & 208:214 in _gen_fbdata.py and 88:93 in _datasetfb.py)

Frequency is 200Hz and I assume _hasvio is always == 1 because gt and imu are synchronized.

Unfortunately, checking the trajectory estimated using the network outputs I'm still not getting good results and I would very much appreciate your help. So I have some questions:

1 - What's the difference between --imu_freq, --imu_base_freq and --sample_freq? Should I use 200Hz for all three of them? 2 - Why do loss calculations change after epoch 10? Sometimes I get negative loss with that second calculation. 3 - Am I doing something wrong regarding how I'm using RoNIN data? If you've used it, could you please tell me how?

Thanks a lot in advance!

[nbgit10]

My understanding might be incomplete so take it with a grain of salt:

• imu_freq: Frequency of your IMU data [Hz] (for you 200)
• imu_base_freq: Frequency of your GT data [Hz] (for you 200)
• sample_freq: Frequency [Hz] at which network gets called during inference (i.e. test/eval with RoNIN) (20 seems to be alright but try out different values for your specific setup, e.g. 10, 20, 50, 100, 200).

Loss is mentioned in the paper, it is first MSE, then log likelihood loss. You can also use a mixture of both after some epochs. Log-likelihood can be negative if likelihood-function < 1, so that is to be expected.

I think you should not uncomment the filter values etc as they are used for RoNIN. Remember, RoNIN or actually the NN presented in this Repo/Paper relies on an external rotations estimate, which I think was stored in the filter_q variable.

I also have synced data and just set calib_state.txt statically to zeros and ones, and imu_raw = imu_calib since our IMU is already calibrated well and we cannot estimate bias/scale factors further with our method. For the AHRS-values we just use another estimate we have. If you cannot get an AHRS estimate for whatever reason, you should rather use the provided SC-EKF setup instead of RoNIN as it integrates rotation based on the gyroscope values (propagation model) and does not require an external rotation estimate.

On the results side: The paper uses an intense amount of data and only 10% for validation and testing each combined with quite a large neural network. Furthermore, RoNIN alone is much worse than the SC-EKF but the results are (for me) not great anyhow - but still much better than a naive open-loop integration of the IMU signal. After some sensitivity analysis I have noticed that the amount of data used is by far the limiting factor and I don't have enough data to get really good results (2h training data compared to 48h in the paper). Rather try to compare to other state of the art dead reckoning solutions running on your data than to the results presented in the paper.

[CathIAS]

@smorantg2 @nbgit10

Let me know in case this explanation https://github.com/CathIAS/TLIO/issues/20#issuecomment-795692797 on --imu_freq, --imu_base_freq and --sample_freq is not clear.

At epoch 10, the loss function changes to both MSE and NLL, you will be able to modify this in get_loss function in losses.py.

I have not directly used RoNIN dataset, but I think it to be possible to achieve a fair result with it. Using RoNIN dataset to run main_net.py I would expect a result on par with what the RoNIN paper reports. Do not use sample_freq as 200 as a first try, that would be too many identical samples. Set it to 20. You do not need integration_q_wxyz_s, filter_q_wxyz_s and offline_calib_s to run the network so you're on the right track. Check your data. Do an integration of the calibrated gyroscope data and compare to ground truth to see if the data makes sense. Try using ground truth rotation during test time to see if you can obtain a good result with that first. Make use of the plot functions in the code, and look into the estimated displacements comparing to the ground truth, and see if the network is doing reasonable things. Trajectory is the final step by integrating the network outputs.

I'm sorry for this delay.

[CathIAS]

@Andrew001207 Getting perfect IMU data usually happens in a simulator environment. With rotation and position, you can do interpolation, however this requires subsampling, and you cannot get the perfectly aligned IMU to the original trajectories. The best way is to run an algorithm that estimates the biases to calibrate the IMU, with additional sensor measurements (such as VIO), so that the bias information is included in your dataset.

[dtneverdie]

@smorantg2 Hi, it seems that you are using RONIN dataset to get model, i am trying to do so too at present. Could you give me the synced RoNIN dataset in TLIO format? That would help a lot to my work and i would be very appreciate.

[patrickctrf]

@dtneverdie @nbgit10 @raabuchanan @smorantg2 @Andrew001207 @yonamoto Any of you managed to reproduce the data preparation? I really need to reproduce these results, please

[nbgit10]

Yes, we used our own proprietary data to test this algorithm and got it to work. It was a bit of initial effort to adopt to the data format used by TLIO and to convert our ground-truth to match the VIO format. We used the provided gen_fb_data.py to generate the files needed to train TLIO. If I remember correctly evaluating the filter did not require these files. I also think there might be some minor bugs in the code published here where I needed to adapt the code a little bit. Unfortunately, I cannot share my work, however, with some Python skills and some experience on IMU/VIO data it was less than a day to make the code run (aka train and evaluate TLIO and use the provided filter with TLIO/GT). But the code is quite complex so understanding it in detail requires a bit more time. The readme and the issues should answer everything needed to convert your own data to run the code.

[CathIAS]

@nbgit10 We appreciate your comment for helping the community. It's also a helpful feedback to us. Thank you!

[IannoIITR]

@nbgit10 @CathIAS how many hours of data is required for a basic functioning network? I have about 26+hours of data and the network isn't converging.

[nbgit10]

I don't remember how much data we used but it was less than the paper used. We were able to train the network until convergence. We finetuned the hyperparameters of the network but also the perturbations in the data preprocessing according to our needs as well as the filter. The network's performance depends on the quality of your ground-truth, the quality of the IMU itself, the variety of motions in your dataset and many more aspects.

If your network is not converging you might need to adapt the hyperparameters, modify when to change the loss function or in the end gather additional data. The more complex the motions in your dataset, the more data you will need. I don't know what type of motions you try to cover but in general 26h sounds enough.

[raabuchanan]

@patrickctrf @IannoIITR We were also able to produce similar results for a walking robot and integrated the network into our robot state estimation pipeline. You can read paper here and see the video here

As @nbgit10 says there are some bugs in the code which makes it hard to replicate but it's nothing that can't be figured out with some work. In our research with legged robots (which walk in a much more constrained way than humans) we use an order of magnitude less data (only about 2 hours). We could possibly get better performance from more data but we are less focused on pure IMU-only estimation and instead use this as an extra odometry source to improve robot stat estimation.

[joefscholtz]

@smorantg2 I am also trying to use RoNIN's dataset to train a TLIO model, so it would be reassuring to know if you made it work x). Same as you, _mytimestamps.txt, _imumeasurements.txt and _evolvingstate.txt are somewhat easy to get from synced data (which implies has_vio == 1 as you said). But I've used imu_raw = imu_calib as nbgit10 said. And I'm using imu_freq = imu_base_freq = 200 and sample_freq = 20 for most part of my tries. I think I might doing something wrong with _calibstat.txt, where I doing the following

# calib_state.txt VIO calibration states at image rate (used in data_io.py)
# [t, acc_scale_inv (9), gyr_scale_inv (9), gyro_g_sense (9), b_acc (3), b_gyr (3)]
acc_scale_inv=np.identity(3).reshape([9])
gyr_scale_inv=np.identity(3).reshape([9])
gyro_g_sense=np.zeros(9)

b_acc=np.zeros(3)
b_gyr=np.zeros(3)

@raabuchanan were you able to get any good results when training with Newer College Dataset? I was able to train a RoNIN model with it and got good results, so I'm currently trying to train a TLIO model

Thank you all

[raabuchanan]

@joearrop I have not tried with the Newer College Dataset but I'm excited to hear you trained a RoNIN model in spite of there not actually being that much data.

Also your values for calib_state.txt look sensible to me, I also disabled the scale and g-sense errors. Possibly your problem is using the raw IMU measurements instead of calibrated. For calibrated IMU measurements I estimated the biases for each axis for each IMU measurement then simply subtracted from the raw values.

[Xyanpian]

Hi, it seems that you are using RONIN dataset to get model, i am trying to do so too at present. Could you give me the synced RoNIN dataset in TLIO format? That would help a lot to my work and i would be very appreciate.

[Xyanpian]

非常感谢@CathIAS和@raabuchanan您的解释。我可以成功地训练和运行网络和过滤器。

我仍然想知道正在使用的坐标系。您是否在本地 （IMU） ENU 帧中拥有原始数据，或者是否仍然在 NED 甚至 ECEF 中？ Which dataset do you use for format conversion?

[Xyanpian]

Many thanks @CathIAS and @raabuchanan for your explanations. I can train and run both network and filter successfully.

I am still wondering about the coordinate systems in use. Are you having the raw data in a local (IMU) ENU frame or maybe NED or even ECEF still?

Which dataset do you use for format conversion?