Closed Snowbound9173 closed 4 months ago
Since I've actually kind of been struggling without posting on this for a few weeks, here's a little more information in case it helps.
In my output for the bagfile "hku1.bag" this is the console output:
[ INFO] [1692443700.353026273]: IMU Initializing: 0.5 %
[ WARN] [1692443700.353069093]: Reset ImuProcess
FAST-LIO not ready
[ INFO ]: get point cloud at time: 1577836999.699665.
[ INFO] [1692443700.452178139]: IMU Initializing: 15.0 %
FAST-LIO not ready
[ INFO ]: get point cloud at time: 1577836999.799514.
[ INFO] [1692443700.552403072]: IMU Initializing: 25.0 %
FAST-LIO not ready
[ INFO ]: get point cloud at time: 1577836999.899351.
[ INFO] [1692443700.651165775]: IMU Initializing: 35.0 %
FAST-LIO not ready
[ INFO ]: get point cloud at time: 1577836999.999992.
[ INFO] [1692443700.751892802]: IMU Initializing: 45.0 %
FAST-LIO not ready
I can't tell why the IMU will initialize with the hku1.bag file but mine wouldn't even pick up the topic seemingly..
Here's a comparison of the two side by side. You try to tell me which one is my livox HAP IMU and which is the hku1.bag avia IMU
I suggest you first break down the problem and only open lidar_en
, and confirm if there are any issues with the LIO subsystem in LIVO. Afterward, check the entire system to confirm if there are any issues with the timestamps and extrinsic parameters of the camera and LiDAR.
Hi @xuankuzcr really appreciate the response,
I can't spot a problem, but I'm still not clear about which way the extrinsics are measured. For example, I've used the same extrinsics between FAST-LIVO and r3live. These extrinsics are very accurate in r3 live and give me a very good detailed map. The Livox HAP gives (supposedly) 450,000 points per second (reality: i'm seeing roughly 15k per message, at 30hz, matching my 4k wide angle camera also capturing at 30hz).
The same bag files play just fine in R3live and fast-lio2.
So in summary: to rule out extrinsic issues, if you know, can you confirm which way extinsics are measured? Though I've attmpted to use inverted values (ie: values directly calculated from https://github.com/hku-mars/livox_camera_calib). And i've tried to use values which work great in r3live.
I still don't actually have this working. Like xuankuzcr asked, only running with lidar enabled, no image vio. Unable to even get the application to acknowledge the IMU messages at all. I'm pretty much lost. :(
The transform matrix of fast-livo and fast-lio are different, it is a known situation,please check the first issues after the software release, you will find how the transform works, it is an inverted matrix. There is not many issues with this software, I would check the issues history there is good points there to know
On Fri, 22 Sept 2023, 08:43 Some person learning ROS and trying to automatically map an enviornment to 3d pointcloud., < @.***> wrote:
I still don't actually have this working. Like xuankuzcr asked, only running with lidar enabled, no image vio. Unable to even get the application to acknowledge the IMU messages at all. I'm pretty much lost. :(
— Reply to this email directly, view it on GitHub https://github.com/hku-mars/FAST-LIVO/issues/69#issuecomment-1730957204, or unsubscribe https://github.com/notifications/unsubscribe-auth/AITQOVCW32SRRVCSBISIMBLX3U6SZANCNFSM6AAAAAA3WP3SJU . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Hi FPSychotic! Thank you very much for the reply,
I took your advice and I've done my very best to try and read every single issue all 68 before this one.
https://github.com/hku-mars/FAST-LIVO/issues/15
https://github.com/hku-mars/FAST-LIVO/issues/20
https://github.com/hku-mars/FAST-LIVO/issues/25
https://github.com/hku-mars/FAST-LIVO/issues/35
https://github.com/hku-mars/FAST-LIVO/issues/37
Rcl represents the rotation matrix of LiDAR frame w.r.t. Camera frame.
Pcl represents the translation of LiDAR frame w.r.t. Camera frame.
extrinsic_R represents the rotation matrix of LiDAR frame w.r.t. IMU frame.
extrinsic_T represents the translation of LiDAR frame w.r.t. IMU frame.
If you set them correctly according to the physical meaning, there should be no wrong alignment of the image and LiDAR scan.
With my bag with all three topics fully synchronised i still see no messages in console for the IMU data. All three topics are synchronised.
If my Livox Lidar HAP lidar has embedded IMU, should the configuration be this?
mapping:
acc_cov_scale: 100
gyr_cov_scale: 10000
fov_degree: 120
extrinsic_T: [ 0.0, 0.0, 0.0 ]
extrinsic_R: [ 1, 0, 0, 0, 1, 0, 0, 0, 1]
Should there be no translation between the lidar and IMU? What would I do to check?
As said before, I know the software works, I tested the sample bagfiles and was able to replicate the same behaviour. The expectation isn't that there is a bug, there's just no other way to open a call for help or discussion about a configuration without creating an 'issue'. I do appreciate all the work and I'm impressed by it. I'm especially happy with r3live right now since I was able to create this image from it using my mounted lidar/camera on 3d board: It's truely amazing.
Appreciate the project developers and thank anyone offering help.
Just for quick proof that the project does work here's my screenshot from my machine of running the hku1.bag file from the OneDrive download:
And when hku1.bag runs, it immediately has messages pertaining to IMU Initialization:
One interesting thing to note to compare the configurations between the avia_resize.yaml and my livox_hap.yaml is that if i play back the hku1.bag file while i have used my configuration, i actually will get IMU initialization.
So in this case, my configuration (especially since i've disabled images) works (to a point). You can definitely see that the IMU is initializing.
But if i try and play back my own bag file: I get no mention of IMU anywhere.
So now i'm back to the fact that my rosbag must be missing something. There must be some requirement I haven't met.
Here's the rosbag info from my own recording: Do I need to record more topics?
Here's the hku1 rosbag info:
hku1 basically has the same main 3 topics as my own rosbag though.. The major difference being that the /livox/lidar topic is /livox_ros_driver2/CustomMsg.
Try synchronization computer lidar with ptpd. https://qiita.com/ykatsu111/items/95df91515d7983d87c91 Follow only the part of ptpd. In the camera - lidar matrix In R3live is like 1 2 3 X 4 5 6 Y 7 8 9 Z
In fast livo the calibration matrix you get from Lidar camera package is transposed as 1 4 7 X 2 5 8 Y 3 6 9 Z
Set Blind param to your Lidar minimum distance, 5m is very long, if you make the test in indoor, you will get very few fearures and everything will start to drift.
Probably you should change in every file in the src folder the #include livox_ros_driver to livox_ros_driver2. Same with functions that include livox_ros_driver to livox_ros_driver2 Same with package.xml , later recompile. 128 lines looks excessive, put 64 or less, not sure how that could affect If you made any other change as covariances or any other thing ,restore the default values, only you need is change the matrix and what the doc says
Thanks again very much FPSychotic, really appreciate your thoughts and insight.
I'll have to translate that link, I only speak english. It'll be my bedtime reading.
"Probably you should change in every file in the src folder the #include livox_ros_driver to livox_ros_driver2."
So, the reason I believe r3live, fast-lio and fast-livo use livox_ros_driver at all is for the customMessage data type processing (I'm pretty sure). And in this case, the custom message data is no different for either lidar when saved. The bag file is already recorded on a totally different machine. That machine needs the livox_ros_driver2 for recording the messages. But in this side I'm just processing them with whichever project. Obviously I can process this same bag in fast-lio, r3live but not fast-livo.
If you know the phrase domestic blindness? That's what I feel like. There's a setting i overlook right in front of me over and over without realizing it's wrong.
You raised some questions of the settings in my initial configuration. Thanks for checking those too:
common:
lid_topic: "/livox/lidar"
imu_topic: "/livox/imu"
preprocess:
lidar_type: 1 # Livox LiDAR custommsg
scan_line: 6 # Default 6
blind: 0.5 # blind x m disable | Default 5
I have been updating these with different values. I'm now 100% sure that the correct scan_line is 6 (i get less points with 5, and exactly the same amount of points with 6 as with 7, or 70 or 700). I tested that in R3Live. It wasn't immediately obvious from the livox datasheets and I couldn't find much online because Livox don't explain how they get non-repeating scan lines but they actually explain it really well with the Livox Avia I initially couldn't be sure if the Livox HAP worked the same or not. But it does, it's just a different non-repeating pattern.
I just want to remind you, I'm not worried about the Camera to Lidar matrix, since I've disabled the camera. There's no point putting images when the IMU to Lidar isn't working at all. I have set:
img_enable : 0
If you made any other change as covariances or any other thing ,restore the default values, only you need is change the matrix and what the doc says
Yes I have reset this back a few times and changed them a few times in a few directions. Personally I don't quite understand covariance and that's really something I'll have to look into. In Fast-LIO and R3Live that wasn't really user-changeable through the configuration. But that's OK I've reverted them.
I've tried playing my bagfile while using the avia_resize.yaml configuration just to be completely sure. It does not work on my bagfile even with the camera image processing disabled. You might think "Oh that's because the IMU/Lidar driver is different" but actually, for Fast-LIO and R3Live and another project called pv-lio using the "Avia" configuration works 100% with the Livox HAP. They're actually very similar units. The HAP might have a few additional features and a different lidar projection but it's custom message data structure is the same.
Probably you should change in every file in the src folder the #include livox_ros_driver to livox_ros_driver2.
Yeah I can create a new workspace and try that. It's something new I haven't tried. I will look at doing this soon.
I was curious if covariance would affect the processing of the IMU and whether or not it would change the messages in the console.
mapping:
acc_cov_scale: 100
gyr_cov_scale: 10000
fov_degree: 90
extrinsic_T: [ 0.04165, 0.02326, -0.0284 ]
extrinsic_R: [ 1, 0, 0,
0, 1, 0,
0, 0, 1]
OK then lets totally mess up the IMU orientation with regards to the lidar. So again this is the avia_resize.yaml with using the known-good hku1.bag file:
extrinsic_T: [ 0.04165, 0.02326, -0.0284 ]
extrinsic_R: [ 1, 0, 0,
0, -1, 0,
0, 0, -1]
Then I playback the hku1.bag file:
process[laserMapping-1]: started with pid [1230353]
process[rviz-2]: started with pid [1230354]
process[republish-3]: started with pid [1230355]
Multi thread started
debug:0 MIN_IMG_COUNT: 1000
[ INFO] [1695644309.462244733]: Found parameter: laserMapping/cam_model, value: Pinhole
[ INFO] [1695644309.462650898]: Found parameter: laserMapping/cam_width, value: 1280
[ INFO] [1695644309.462765199]: Found parameter: laserMapping/cam_height, value: 1024
[ INFO] [1695644309.462883841]: Found parameter: laserMapping/cam_fx, value: 863.591
[ INFO] [1695644309.462996402]: Found parameter: laserMapping/cam_fy, value: 863.1
[ INFO] [1695644309.463126104]: Found parameter: laserMapping/cam_cx, value: 621.666
[ INFO] [1695644309.463247165]: Found parameter: laserMapping/cam_cy, value: 533.972
[ INFO] [1695644309.463360267]: Found parameter: laserMapping/cam_d0, value: -0.0944205
[ INFO] [1695644309.463470138]: Found parameter: laserMapping/cam_d1, value: 0.0946728
[ INFO] [1695644309.463580010]: Found parameter: laserMapping/cam_d2, value: -0.00807971
[ INFO] [1695644309.463691891]: Found parameter: laserMapping/cam_d3, value: 8.07461e-05
[ INFO ]: get point cloud at time: 1577836978.699827.
[ INFO ]: get point cloud at time: 1577836978.799653.
[ INFO] [1695644316.720836822]: IMU Initializing: 0.5 %
[ WARN] [1695644316.720859093]: Reset ImuProcess
FAST-LIO not ready
[ INFO ]: get point cloud at time: 1577836978.899480.
[ INFO] [1695644316.820447622]: IMU Initializing: 10.5 %
I increased both numbers by 100x and it had no affect when running the avaya_resize.yaml with hku1.bag. It still attempts to process the IMU messages with respect to the Lidar. In fact I even went further, I inverted the extrinsic for the avaya_resize.yaml and played hku1.bag. This still initialized the IMU. This makes sense. It wouldn't be until you moved that there would be a problem. So i'm sure my extrinsic between IMU and Lidar isn't the issue, since i'd have messages in the console and a bad pointcloud instead.
[ INFO] [1695643794.152553250]: IMU Initializing: 97.5 %
[ INFO] [1695643794.152584311]: IMU Initials: Gravity: -0.2711 -0.4026 -9.7980 0.9940; state.bias_g: 1000.0000 1000.0000 1000.0000; acc covarience: 0.00112183 0.00060384 0.00099876; gry covarience: 0.00000738 0.00000528 0.00000687
[ INFO] [1695643794.152594871]: IMU Initials: Gravity: -0.2711 -0.4026 -9.7980 0.9940; state.bias_g: 0.0000 0.0000 0.0000; acc covarience: 1.12182852 0.60383990 0.99875795; gry covarience: 0.73768713 0.52752602 0.68718525
I figure I need to rule out more things: I'm going to be really sneaky.. I'll play the hku1.bag file but only 1 topic at a time to see what affect this has on console messages:
rosbag play --topics /livox/lidar hku1.bag
process[laserMapping-1]: started with pid [1203781]
process[rviz-2]: started with pid [1203782]
process[republish-3]: started with pid [1203783]
Multi thread started
debug:0 MIN_IMG_COUNT: 1000
[ INFO] [1695643984.163079756]: Found parameter: laserMapping/cam_model, value: Pinhole
[ INFO] [1695643984.163527409]: Found parameter: laserMapping/cam_width, value: 1280
[ INFO] [1695643984.163656070]: Found parameter: laserMapping/cam_height, value: 1024
[ INFO] [1695643984.163781361]: Found parameter: laserMapping/cam_fx, value: 863.591
[ INFO] [1695643984.163973713]: Found parameter: laserMapping/cam_fy, value: 863.1
[ INFO] [1695643984.164092753]: Found parameter: laserMapping/cam_cx, value: 621.666
[ INFO] [1695643984.164206854]: Found parameter: laserMapping/cam_cy, value: 533.972
[ INFO] [1695643984.164323015]: Found parameter: laserMapping/cam_d0, value: -0.0944205
[ INFO] [1695643984.164437826]: Found parameter: laserMapping/cam_d1, value: 0.0946728
[ INFO] [1695643984.164551057]: Found parameter: laserMapping/cam_d2, value: -0.00807971
[ INFO] [1695643984.164666418]: Found parameter: laserMapping/cam_d3, value: 8.07461e-05
[ INFO ]: get point cloud at time: 1577836980.999310.
[ INFO ]: get point cloud at time: 1577836981.099941.
[ INFO ]: get point cloud at time: 1577836981.199794.
[ INFO ]: get point cloud at time: 1577836981.299636.
[ INFO ]: get point cloud at time: 1577836981.399467.
[ INFO ]: get point cloud at time: 1577836981.499281.
[ INFO ]: get point cloud at time: 1577836981.599906.
This gives me messages only about point cloud. (exactly like my own bag file)
OK this time only IMU $ rosbag play hku1.bag --topics /livox/imu
{No messages in console}
Without a lidar message, the IMU seems to be ignored completely.
I'll add both IMU and Lidar (no other topics) $ rosbag play hku1.bag --topics /livox/lidar /livox/imu
process[laserMapping-1]: started with pid [1213151]
process[rviz-2]: started with pid [1213152]
process[republish-3]: started with pid [1213153]
Multi thread started
debug:0 MIN_IMG_COUNT: 1000
[ INFO] [1695644096.731326423]: Found parameter: laserMapping/cam_model, value: Pinhole
[ INFO] [1695644096.731718455]: Found parameter: laserMapping/cam_width, value: 1280
[ INFO] [1695644096.731830256]: Found parameter: laserMapping/cam_height, value: 1024
[ INFO] [1695644096.731953407]: Found parameter: laserMapping/cam_fx, value: 863.591
[ INFO] [1695644096.732068718]: Found parameter: laserMapping/cam_fy, value: 863.1
[ INFO] [1695644096.732182309]: Found parameter: laserMapping/cam_cx, value: 621.666
[ INFO] [1695644096.732291740]: Found parameter: laserMapping/cam_cy, value: 533.972
[ INFO] [1695644096.732405650]: Found parameter: laserMapping/cam_d0, value: -0.0944205
[ INFO] [1695644096.732515751]: Found parameter: laserMapping/cam_d1, value: 0.0946728
[ INFO] [1695644096.732625242]: Found parameter: laserMapping/cam_d2, value: -0.00807971
[ INFO] [1695644096.732736293]: Found parameter: laserMapping/cam_d3, value: 8.07461e-05
[ INFO ]: get point cloud at time: 1577836978.799653.
[ INFO ]: get point cloud at time: 1577836978.899480.
[ INFO] [1695644100.275823435]: IMU Initializing: 0.5 %
[ WARN] [1695644100.275847325]: Reset ImuProcess
FAST-LIO not ready
[ INFO ]: get point cloud at time: 1577836978.999310.
[ INFO] [1695644100.374246996]: IMU Initializing: 13.0 %
So as expected all you should need in the messages is Lidar and IMU.
So therefore the IMU to Lidar doesn't matter for IMU messages showing in the console. Rather the lack of IMU messages in the console must be a factor of the IMU messages not being parsed (and potentially silently dropped) by the process for my bag file.
That suggests actually the configuration file itself isn't the problem, it's my bagfile and recording.
Which i can't fathom but must be true. So what is making my IMU messages impossible to be read? My messgaes should be hard synced because my IMU is not external it's fully internal. And I've corrected the lidar/imu messages. But it doesn't matter if corrected or not. Unless I'm not understanding what that sync really means.
Here's a single message from my own IMU on my bag file which doesn't work
---
header:
seq: 5685
stamp:
secs: 1695452552
nsecs: 758416176
frame_id: "livox_frame"
orientation:
x: 0.0
y: 0.0
z: 0.0
w: 0.0
orientation_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
angular_velocity:
x: 0.0012063446920365095
y: 0.026742398738861084
z: 0.00451506394892931
angular_velocity_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
linear_acceleration:
x: 0.3819573223590851
y: -0.038354191929101944
z: 0.907524049282074
linear_acceleration_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
---
Here's a single message from the hku1.bag which works
---
header:
seq: 33134
stamp:
secs: 1577836986
nsecs: 966875792
frame_id: "livox_frame"
orientation:
x: 0.0
y: 0.0
z: 0.0
w: 0.0
orientation_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
angular_velocity:
x: -0.003557032672688365
y: 0.044258665293455124
z: -0.000774329761043191
angular_velocity_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
linear_acceleration:
x: 0.032630160450935364
y: 0.03728320449590683
z: 0.992362380027771
linear_acceleration_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
---
I don't know how to figure out why my IMU data isn't being processed. It doesn't look like the configuration is to blame in this case because even wrong, that configuration will process IMU with lidar as described above.
I'm going to read your blog post here; https://qiita-com.translate.goog/ykatsu111/items/95df91515d7983d87c91?_x_tr_sl=ja&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp
And I might create a new catkin workspace and retry to install fast-livo again but this time reference livox_ros_driver2 instead just in case.
Thanks again very much FPSychotic, really appreciate your thoughts and insight.
I'll have to translate that link, I only speak english. It'll be my bedtime reading.
"Probably you should change in every file in the src folder the #include livox_ros_driver to livox_ros_driver2."
So, the reason I believe r3live, fast-lio and fast-livo use livox_ros_driver at all is for the customMessage data type processing (I'm pretty sure). And in this case, the custom message data is no different for either lidar when saved. The bag file is already recorded on a totally different machine. That machine needs the livox_ros_driver2 for recording the messages. But in this side I'm just processing them with whichever project. Obviously I can process this same bag in fast-lio, r3live but not fast-livo.
If you know the phrase domestic blindness? That's what I feel like. There's a setting i overlook right in front of me over and over without realizing it's wrong.
You raised some questions of the settings in my initial configuration. Thanks for checking those too:
common: lid_topic: "/livox/lidar" imu_topic: "/livox/imu" preprocess: lidar_type: 1 # Livox LiDAR custommsg scan_line: 6 # Default 6 blind: 0.5 # blind x m disable | Default 5
I have been updating these with different values. I'm now 100% sure that the correct scan_line is 6 (i get less points with 5, and exactly the same amount of points with 6 as with 7, or 70 or 700). I tested that in R3Live. It wasn't immediately obvious from the livox datasheets and I couldn't find much online because Livox don't explain how they get non-repeating scan lines but they actually explain it really well with the Livox Avia I initially couldn't be sure if the Livox HAP worked the same or not. But it does, it's just a different non-repeating pattern.
I just want to remind you, I'm not worried about the Camera to Lidar matrix, since I've disabled the camera. There's no point putting images when the IMU to Lidar isn't working at all. I have set:
img_enable : 0
If you made any other change as covariances or any other thing ,restore the default values, only you need is change the matrix and what the doc says
Yes I have reset this back a few times and changed them a few times in a few directions. Personally I don't quite understand covariance and that's really something I'll have to look into. In Fast-LIO and R3Live that wasn't really user-changeable through the configuration. But that's OK I've reverted them.I've tried playing my bagfile while using the avia_resize.yaml configuration just to be completely sure. It does not work on my bagfile even with the camera image processing disabled. You might think "Oh that's because the IMU/Lidar driver is different" but actually, for Fast-LIO and R3Live and another project called pv-lio using the "Avia" configuration works 100% with the Livox HAP. They're actually very similar units. The HAP might have a few additional features and a different lidar projection but it's custom message data structure is the same.
Probably you should change in every file in the src folder the #include livox_ros_driver to livox_ros_driver2.
Yeah I can create a new workspace and try that. It's something new I haven't tried. I will look at doing this soon.I was curious if covariance would affect the processing of the IMU and whether or not it would change the messages in the console.
mapping: acc_cov_scale: 100 gyr_cov_scale: 10000 fov_degree: 90 extrinsic_T: [ 0.04165, 0.02326, -0.0284 ] extrinsic_R: [ 1, 0, 0, 0, 1, 0, 0, 0, 1]
OK then lets totally mess up the IMU orientation with regards to the lidar. So again this is the avia_resize.yaml with using the known-good hku1.bag file:
extrinsic_T: [ 0.04165, 0.02326, -0.0284 ] extrinsic_R: [ 1, 0, 0, 0, -1, 0, 0, 0, -1]
Then I playback the hku1.bag file:
process[laserMapping-1]: started with pid [1230353] process[rviz-2]: started with pid [1230354] process[republish-3]: started with pid [1230355] Multi thread started debug:0 MIN_IMG_COUNT: 1000 [ INFO] [1695644309.462244733]: Found parameter: laserMapping/cam_model, value: Pinhole [ INFO] [1695644309.462650898]: Found parameter: laserMapping/cam_width, value: 1280 [ INFO] [1695644309.462765199]: Found parameter: laserMapping/cam_height, value: 1024 [ INFO] [1695644309.462883841]: Found parameter: laserMapping/cam_fx, value: 863.591 [ INFO] [1695644309.462996402]: Found parameter: laserMapping/cam_fy, value: 863.1 [ INFO] [1695644309.463126104]: Found parameter: laserMapping/cam_cx, value: 621.666 [ INFO] [1695644309.463247165]: Found parameter: laserMapping/cam_cy, value: 533.972 [ INFO] [1695644309.463360267]: Found parameter: laserMapping/cam_d0, value: -0.0944205 [ INFO] [1695644309.463470138]: Found parameter: laserMapping/cam_d1, value: 0.0946728 [ INFO] [1695644309.463580010]: Found parameter: laserMapping/cam_d2, value: -0.00807971 [ INFO] [1695644309.463691891]: Found parameter: laserMapping/cam_d3, value: 8.07461e-05 [ INFO ]: get point cloud at time: 1577836978.699827. [ INFO ]: get point cloud at time: 1577836978.799653. [ INFO] [1695644316.720836822]: IMU Initializing: 0.5 % [ WARN] [1695644316.720859093]: Reset ImuProcess FAST-LIO not ready [ INFO ]: get point cloud at time: 1577836978.899480. [ INFO] [1695644316.820447622]: IMU Initializing: 10.5 %
I increased both numbers by 100x and it had no affect when running the avaya_resize.yaml with hku1.bag. It still attempts to process the IMU messages with respect to the Lidar. In fact I even went further, I inverted the extrinsic for the avaya_resize.yaml and played hku1.bag. This still initialized the IMU. This makes sense. It wouldn't be until you moved that there would be a problem. So i'm sure my extrinsic between IMU and Lidar isn't the issue, since i'd have messages in the console and a bad pointcloud instead.
[ INFO] [1695643794.152553250]: IMU Initializing: 97.5 % [ INFO] [1695643794.152584311]: IMU Initials: Gravity: -0.2711 -0.4026 -9.7980 0.9940; state.bias_g: 1000.0000 1000.0000 1000.0000; acc covarience: 0.00112183 0.00060384 0.00099876; gry covarience: 0.00000738 0.00000528 0.00000687 [ INFO] [1695643794.152594871]: IMU Initials: Gravity: -0.2711 -0.4026 -9.7980 0.9940; state.bias_g: 0.0000 0.0000 0.0000; acc covarience: 1.12182852 0.60383990 0.99875795; gry covarience: 0.73768713 0.52752602 0.68718525
I figure I need to rule out more things: I'm going to be really sneaky.. I'll play the hku1.bag file but only 1 topic at a time to see what affect this has on console messages:
rosbag play --topics /livox/lidar hku1.bag
process[laserMapping-1]: started with pid [1203781] process[rviz-2]: started with pid [1203782] process[republish-3]: started with pid [1203783] Multi thread started debug:0 MIN_IMG_COUNT: 1000 [ INFO] [1695643984.163079756]: Found parameter: laserMapping/cam_model, value: Pinhole [ INFO] [1695643984.163527409]: Found parameter: laserMapping/cam_width, value: 1280 [ INFO] [1695643984.163656070]: Found parameter: laserMapping/cam_height, value: 1024 [ INFO] [1695643984.163781361]: Found parameter: laserMapping/cam_fx, value: 863.591 [ INFO] [1695643984.163973713]: Found parameter: laserMapping/cam_fy, value: 863.1 [ INFO] [1695643984.164092753]: Found parameter: laserMapping/cam_cx, value: 621.666 [ INFO] [1695643984.164206854]: Found parameter: laserMapping/cam_cy, value: 533.972 [ INFO] [1695643984.164323015]: Found parameter: laserMapping/cam_d0, value: -0.0944205 [ INFO] [1695643984.164437826]: Found parameter: laserMapping/cam_d1, value: 0.0946728 [ INFO] [1695643984.164551057]: Found parameter: laserMapping/cam_d2, value: -0.00807971 [ INFO] [1695643984.164666418]: Found parameter: laserMapping/cam_d3, value: 8.07461e-05 [ INFO ]: get point cloud at time: 1577836980.999310. [ INFO ]: get point cloud at time: 1577836981.099941. [ INFO ]: get point cloud at time: 1577836981.199794. [ INFO ]: get point cloud at time: 1577836981.299636. [ INFO ]: get point cloud at time: 1577836981.399467. [ INFO ]: get point cloud at time: 1577836981.499281. [ INFO ]: get point cloud at time: 1577836981.599906.
This gives me messages only about point cloud. (exactly like my own bag file)
OK this time only IMU $ rosbag play hku1.bag --topics /livox/imu
{No messages in console}
Without a lidar message, the IMU seems to be ignored completely.
I'll add both IMU and Lidar (no other topics) $ rosbag play hku1.bag --topics /livox/lidar /livox/imu
process[laserMapping-1]: started with pid [1213151] process[rviz-2]: started with pid [1213152] process[republish-3]: started with pid [1213153] Multi thread started debug:0 MIN_IMG_COUNT: 1000 [ INFO] [1695644096.731326423]: Found parameter: laserMapping/cam_model, value: Pinhole [ INFO] [1695644096.731718455]: Found parameter: laserMapping/cam_width, value: 1280 [ INFO] [1695644096.731830256]: Found parameter: laserMapping/cam_height, value: 1024 [ INFO] [1695644096.731953407]: Found parameter: laserMapping/cam_fx, value: 863.591 [ INFO] [1695644096.732068718]: Found parameter: laserMapping/cam_fy, value: 863.1 [ INFO] [1695644096.732182309]: Found parameter: laserMapping/cam_cx, value: 621.666 [ INFO] [1695644096.732291740]: Found parameter: laserMapping/cam_cy, value: 533.972 [ INFO] [1695644096.732405650]: Found parameter: laserMapping/cam_d0, value: -0.0944205 [ INFO] [1695644096.732515751]: Found parameter: laserMapping/cam_d1, value: 0.0946728 [ INFO] [1695644096.732625242]: Found parameter: laserMapping/cam_d2, value: -0.00807971 [ INFO] [1695644096.732736293]: Found parameter: laserMapping/cam_d3, value: 8.07461e-05 [ INFO ]: get point cloud at time: 1577836978.799653. [ INFO ]: get point cloud at time: 1577836978.899480. [ INFO] [1695644100.275823435]: IMU Initializing: 0.5 % [ WARN] [1695644100.275847325]: Reset ImuProcess FAST-LIO not ready [ INFO ]: get point cloud at time: 1577836978.999310. [ INFO] [1695644100.374246996]: IMU Initializing: 13.0 %
So as expected all you should need in the messages is Lidar and IMU.
So therefore the IMU to Lidar doesn't matter for IMU messages showing in the console. Rather the lack of IMU messages in the console must be a factor of the IMU messages not being parsed (and potentially silently dropped) by the process for my bag file.
That suggests actually the configuration file itself isn't the problem, it's my bagfile and recording.
Which i can't fathom but must be true. So what is making my IMU messages impossible to be read? My messgaes should be hard synced because my IMU is not external it's fully internal. And I've corrected the lidar/imu messages. But it doesn't matter if corrected or not. Unless I'm not understanding what that sync really means.
Here's a single message from my own IMU on my bag file which doesn't work
--- header: seq: 5685 stamp: secs: 1695452552 nsecs: 758416176 frame_id: "livox_frame" orientation: x: 0.0 y: 0.0 z: 0.0 w: 0.0 orientation_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] angular_velocity: x: 0.0012063446920365095 y: 0.026742398738861084 z: 0.00451506394892931 angular_velocity_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] linear_acceleration: x: 0.3819573223590851 y: -0.038354191929101944 z: 0.907524049282074 linear_acceleration_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] ---
Here's a single message from the hku1.bag which works
--- header: seq: 33134 stamp: secs: 1577836986 nsecs: 966875792 frame_id: "livox_frame" orientation: x: 0.0 y: 0.0 z: 0.0 w: 0.0 orientation_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] angular_velocity: x: -0.003557032672688365 y: 0.044258665293455124 z: -0.000774329761043191 angular_velocity_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] linear_acceleration: x: 0.032630160450935364 y: 0.03728320449590683 z: 0.992362380027771 linear_acceleration_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] ---
I don't know how to figure out why my IMU data isn't being processed. It doesn't look like the configuration is to blame in this case because even wrong, that configuration will process IMU with lidar as described above.
I'm going to read your blog post here; https://qiita-com.translate.goog/ykatsu111/items/95df91515d7983d87c91?_x_tr_sl=ja&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp
And I might create a new catkin workspace and retry to install fast-livo again but this time reference livox_ros_driver2 instead just in case. Just tell that the custom message of in ---livox_Ros_driver2 is different and the main reason it was released, in test units it was using the original ros driver.
Find here https://medium.com/geekculture/livox-ptp-time-sync-on-raspberry-pi-4-bb31b78db838 the configuration of PTPd for synchronisation. Note if you use Jetson probably it wont start the service in boot and you will need input the command in every boot. Without this software you will need synchro with GPS or PPS, synchro is like water for slam woth lidar.
I would forget the bags, you cannot control them, you don't know how they were recorded,my self find bags just won't work. Use the real lidar to can debug properly. For sure you will need the livox ros driver2, I needed or the mid-360 wouldn't work i.e. with point lio. I didn't install fast-livo with mid-360 but have it in the Avia
Doesn't have sense use Fast-livo without camera, is in that case better use Fast-lio with Scancontext/pgo or R3live if you want color.
I would forget the bags, you cannot control them, you don't know how they were recorded,my self find bags just won't work.
Oh. I think you didn't realise. Those are my bag files. I recorded them with my equipment that I purchased. I've taken dozens of recordings and processed them all successfully with r3Live and Fast-LIO and others. They just don't work with this project.
The recordings are just my own apartment and the street around my place.
So if there's a recording setting I need to adjust, then I definitely can change it and process it on a new bag file.
Edit: Oh you probably mean forget using my old bags and re-do the recording after checking sudo ptpd -M -i eth0 -C
and I fully agree. I'm going to attempt to follow that soon.
Thanks very much by the way :)
I would forget the bags, you cannot control them, you don't know how they were recorded,my self find bags just won't work.
Oh. I think you didn't realise. Those are my bag files. I recorded them with my equipment that I purchased. I've taken dozens of recordings and processed them all successfully with r3Live and Fast-LIO and others. They just don't work with this project.The recordings are just my own apartment and the street around my place.
So if there's a recording setting I need to adjust, then I definitely can change it and process it on a new bag file.
Edit: Oh you probably mean forget using my old bags and re-do the recording after checking
sudo ptpd -M -i eth0 -C
and I fully agree. I'm going to attempt to follow that soon.Thanks very much by the way :)
The same others record bags of bad quality, you can do it too, in example as you said already, just your data probably is bad synchronized, but also could be you have a high CPU load while record the bag and data is missing or bad. I would insist you try in realtime, is personal opinion, but setup something for bags, and for live, is work minimum twice, is the best advice I can give you. Also the most convenient would be you share the bags for proper advice and debug. That your bags or from others, work in fast lio or r3live doesn't mean many to be honest, the requirements or tolerance to bad data can be different. Fast livo is more intense in processing due to it gives quite more image quality. I would focus in the aplication and its target use, fast-livo and and real lidar, any other thing is distract you and complicate the thing. For this software and you use, the most important is get the IMU perfectly synced with the computer and ROS, be focus in that, even before of test any other thing. Later the IMU and Camera calibrations.
Sorry I had to spend so long between updates. I'm not a student, nor engineer, nor developer. I'm simply a systems administrator by day. This is just a hobby.
That medium.com article you linked is really good I think. I actually don't have hardware timesync I've got a Beelink i7 for my recording device which was needed because any kind of Pi or other low-end PC would choke doing 4k at 30fps camera. (I power it with some batteries and a modified car-power-charger to get from 12V to 19V).
Back to the topic at hand, here's what outputs when i try and follow that very clear guide.
livox@beelink-record:~$ ethtool -T enp2s0
Time stamping parameters for enp2s0:
Capabilities:
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none
So with the same issue as this author: I followed along and installed ptpd
Configuring with
sudo ptpd -M -i enp2s0 -C
I then launch the livox driver for HAP
$ roslaunch livox_ros_driver2 msg_HAP.launch
So here i then run rostopic echo /livox/lidar/header
rostopic echo /livox/lidar/header
---
seq: 5926
stamp:
secs: 1695711334
nsecs: 549993515
frame_id: "livox_frame"
---
seq: 5927
stamp:
secs: 1695711334
nsecs: 599920988
frame_id: "livox_frame"
---
OK but my old time stamp wasn't that far out was it? I should have compared before and after. Well there's another way to see if this worked. It's to go and play it on the processing computer...
No. Still nothing.
So i check if my livox is time synced a different way by just echoing the topics for the usb_cam which is recorded by the USB on the machine and should have the local machine time stamp, and the lidar which gets a timestamp from ptpd master...
They're offset by 0.06seconds i guess but they're essentially the same time with an offset as the message arrives. Not a new start from 0 epoch.
I assume then ptpd is working... and therefore not the issue here?
Edit: I'm going to sync my bag properly which will take a while.
Is impossible to help you as you still don't share the bags, also no commands seen in the pictures, no idea what camera, and how it synchronized, no thing about errors, topics rviz . The info you give in unstructured and still lacks of any detail. Can you please share the bags to can run them, config and launch files? No idea if developers can help you with the info you given, but in my personal case I'm unable.
On Tue, 26 Sept 2023, 10:37 Some person learning ROS and trying to automatically map an enviornment to 3d pointcloud., < @.***> wrote:
Sorry I had to spend so long between updates. I'm not a student, nor engineer, nor developer. I'm simply a systems administrator by day. This is just a hobby.
That medium.com article you linked is really good I think. I actually don't have hardware timesync I've got a Beelink i7 for my recording device which was needed because any kind of Pi or other low-end PC would choke doing 4k at 30fps camera. (I power it with some batteries and a modified car-power-charger to get from 12V to 19V).
Back to the topic at hand, here's what outputs when i try and follow that very clear guide.
@.***:~$ ethtool -T enp2s0 Time stamping parameters for enp2s0: Capabilities: software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE) software-receive (SOF_TIMESTAMPING_RX_SOFTWARE) software-system-clock (SOF_TIMESTAMPING_SOFTWARE) PTP Hardware Clock: none Hardware Transmit Timestamp Modes: none Hardware Receive Filter Modes: none
So with the same issue as this author: I followed along and installed ptpd [image: image] https://user-images.githubusercontent.com/112289631/270559984-dd56c5fd-8534-4735-adfe-0d1d7e061773.png
Configuring with sudo ptpd -M -i enp2s0 -C [image: image] https://user-images.githubusercontent.com/112289631/270560551-8f979cfc-187e-4a82-9207-d5a0963918d6.png
I then launch the livox driver for HAP $ roslaunch livox_ros_driver2 msg_HAP.launch [image: image] https://user-images.githubusercontent.com/112289631/270560924-fd283b3b-15f1-496b-8944-0e1e982624f2.png
So here i then run rostopic echo /livox/lidar/header
rostopic echo /livox/lidar/header
seq: 5926 stamp: secs: 1695711334 nsecs: 549993515 frame_id: "livox_frame"
seq: 5927 stamp: secs: 1695711334 nsecs: 599920988 frame_id: "livox_frame"
OK but my old time stamp wasn't that far out was it? I should have compared before and after. Well there's another way to see if this worked. It's to go and play it on the processing computer... [image: image] https://user-images.githubusercontent.com/112289631/270601451-a48fb0fb-834c-4dc1-8e62-ac62d289696c.png
No. Still nothing.
So i check if my livox is time synced a different way by just echoing the topics for the usb_cam which is recorded by the USB on the machine and should have the local machine time stamp, and the lidar which gets a timestamp from ptpd master... [image: image] https://user-images.githubusercontent.com/112289631/270609706-40816551-d7ad-4e44-811c-9afec0b7757d.png
They're offset by 0.06seconds i guess but they're essentially the same time with an offset as the message arrives. Not a new start from 0 epoch.
I assume then ptpd is working... and therefore not the issue here?
— Reply to this email directly, view it on GitHub https://github.com/hku-mars/FAST-LIVO/issues/69#issuecomment-1735181934, or unsubscribe https://github.com/notifications/unsubscribe-auth/AITQOVAKAPCBFIWJVO57J2LX4KO5FANCNFSM6AAAAAA3WP3SJU . You are receiving this because you commented.Message ID: @.***>
The problem is the bag files are over 90GB but you're right I can take new bag files and share them something shorter, maybe a few seconds of footage maybe 15seconds or something may be doable.
Thanks for your patience, a lot of 12 hour work days this week, haven't been able to focus on this at all.
Appreciate the help.
Hi FPSychotic,
I was able to upload as requested: https://drive.google.com/drive/folders/10m3wC94n6CB3IpbWwT5wb0zE_4Yxu0Jl There should be 6 files:
Again thanks for your help, and I do not expect you to thanklessly stress or keep giving your expertise and knowledge on demand. At any time if you do not have the patience or time to offer help, then it's understandable and simply thank you for helping so far.
There is no problem I will help while I can. I will come back to you after test, i will test anytime in the next six days, the reply will be next weekend as latest.
On Sat, 30 Sept 2023, 00:19 Some person learning ROS and trying to automatically map an enviornment to 3d pointcloud., < @.***> wrote:
Hi FPSychotic,
I was able to upload as requested: https://drive.google.com/drive/folders/10m3wC94n6CB3IpbWwT5wb0zE_4Yxu0Jl There should be 6 files:
- Camera Configuration 'imx477_4k.yaml' which is the configuration for the usb_cam which is a Arducam IMX477
- livox.yaml which is the configuration for the livox HAP
- livox.launch which is the launch file for the arducam imx477 configuration and the livox HAP
- full-4k_0000_orig.bag the original recording unaltered by any post-processing
- full_4k_0000_msg_synchro.bag which is the same file but now with the topics synchronised
- full_4k_0000_cam_dup_removed.bag which is the same file but the camera (arducam imx477 with usb board) duplicates i've learned repeats the same image sometimes. So i pass it through a script that creates an MD5sum of every camera message, and then remembers the last unique MD5 and if the next message matches the last unique, then it removes that duplicate image.
Again thanks for your help, and I do not expect you to thanklessly stress or keep giving your expertise and knowledge on demand. At any time if you do not have the patience or time to offer help, then it's understandable and simply thank you for helping so far.
— Reply to this email directly, view it on GitHub https://github.com/hku-mars/FAST-LIVO/issues/69#issuecomment-1741575445, or unsubscribe https://github.com/notifications/unsubscribe-auth/AITQOVHGEYEUBOX3GMP54X3X45JQDANCNFSM6AAAAAA3WP3SJU . You are receiving this because you commented.Message ID: @.***>
Hi,
I've got some lidar/camera synchronised rosbag files that I've recorded using a Livox HAP which uses an embedded IMU and a different driver using both Livox Driver 2 and SDK2.
I'm able to build, compile and run the rosbags shared in this project, but i'm unable to get my own rosbags to view. I get lots of messages for my set camera topics and lidar topics, but I'm concerned that there's an IMU problem.
When i read through your notes it mentions the /Handheld_ws project workspace you use, https://github.com/xuankuzcr/Handheld_ws and within it it contains a custom livox_driver built with a modified IMU.
The ros driver of Livox LiDAR (fixed IMU timestamp, latest modified on January 10th, 2022).
However this won't help me, as that project does not work with Livox HAP which requires the https://github.com/Livox-SDK/livox_ros_driver2 driver.
So my question in troubleshooting my lack of IMU being processed in the project, is this custom IMU timestamp modification required? If so is there advice you could add here?
Or am I way off in troubleshooting and this isn't my issue at all?
My setup uses:
Note: I'm able to process R3Live with this setup, but perhaps FAST-LIVO requires additional changes to IMU?
BTW some relevant configuration data for my fast-livo yaml relevant to the IMU:
Note2: Also i've tried it with and without img_enable since i can't even get lidar+imu working together. Note3: Livox HAP Lidar was set to 10hz at the time.