search

ethz-asl / rovio

Other
1.14k stars 507 forks source link

real time super slow #192

Open tiralonghipol opened 6 years ago

tiralonghipol commented 6 years ago

Hi. I calibrated my camera with Kalibr, and setup everything. I was wondering which parameters can I act on in order to speed up the performance for real time.. Right now, seems like rovio keeps doing calculations on the first frame on startup and if I move the camera the frame still fixed. I'm using an i7 pc, and the cpu in not saturated.. any hint? Thank you!

helenol commented 6 years ago

Rovio should run real-time at about 20-30 Hz using about ~20% CPU on a standard laptop CPU with default parameters out of the box. Make sure you compile in Release mode (and not debug) and that the data is piped in correctly, and that you're feeding in IMU data that's time synchronized to the camera. Rovio will not work without IMU data.

tiralonghipol commented 6 years ago

Ok then I'm assuming some bad camera/imu sync.. how to check that? any suggestion? thank you for your reply, btw

helenol commented 6 years ago

Maybe the easiest thing to do is to first check that it's running correctly on the euroc dataset, and in real-time. This will rule out that it's anything other than your data. Probably the easiest way to check your camera-IMU time sync is to run kalibr ( https://github.com/ethz-asl/kalibr ) for cam-IMU calibration. If you're getting huge reprojection errors, or the motions aren't matching, then this is your issue.

tiralonghipol commented 6 years ago

ok I solved the problem of the 'fixed initial frame' (really thank you!!) I've repeated the calibration with Kalibr many many times, and I think this is the best result i can obtain with such a cheap imu like the mpu6050.. what do you think?

Calibration results

Normalized Residuals

Reprojection error (cam0): mean 0.588927292038, median 0.488037950524, std: 0.426175123678 Reprojection error (cam1): mean 0.592739630203, median 0.479429252212, std: 0.469319461226 Gyroscope error (imu0): mean 1.17970688177, median 0.910501933594, std: 1.10498299154 Accelerometer error (imu0): mean 0.552302048441, median 0.481720104149, std: 0.334006283476

Residuals

Reprojection error (cam0) [px]: mean 0.588927292038, median 0.488037950524, std: 0.426175123678 Reprojection error (cam1) [px]: mean 0.592739630203, median 0.479429252212, std: 0.469319461226 Gyroscope error (imu0) [rad/s]: mean 0.080011613555, median 0.0617532456391, std: 0.0749435927439 Accelerometer error (imu0) [m/s^2]: mean 0.0374589474289, median 0.0326718470585, std: 0.0226534083097

Transformation (cam0):

T_ci: (imu0 to cam0): [[-0.99996808 -0.00446951 -0.00662233 0.08393679] [ 0.00445056 -0.99998597 0.00287312 -0.01480612] [-0.00663508 0.00284356 0.99997394 -0.06664928] [ 0. 0. 0. 1. ]]

T_ic: (cam0 to imu0): [[-0.99996808 0.00445056 -0.00663508 0.08355779] [-0.00446951 -0.99998597 0.00284356 -0.01424124] [-0.00662233 0.00287312 0.99997394 0.06724594] [ 0. 0. 0. 1. ]]

timeshift cam0 to imu0: [s] (t_imu = t_cam + shift) 0.0

Transformation (cam1):

T_ci: (imu0 to cam1): [[-0.99978718 -0.00505324 -0.02000146 -0.0151341 ] [ 0.00495019 -0.99997424 0.00519807 -0.01584605] [-0.02002721 0.00509795 0.99978644 -0.06499639] [ 0. 0. 0. 1. ]]

T_ic: (cam1 to imu0): [[-0.99978718 0.00495019 -0.02002721 -0.01635413] [-0.00505324 -0.99997424 0.00509795 -0.01559077] [-0.02000146 0.00519807 0.99978644 0.06476218] [ 0. 0. 0. 1. ]]

timeshift cam1 to imu0: [s] (t_imu = t_cam + shift) 0.0

Baselines:

Baseline (cam0 to cam1): [[ 0.99991031 0.00054608 -0.01338164 -0.09994716] [-0.00051506 0.99999717 0.00232161 -0.000842 ] [ 0.01338287 -0.0023145 0.99990777 0.00048916] [ 0. 0. 0. 1. ]] baseline norm: 0.0999518992169 [m]

Gravity vector in target coords: [m/s^2] [-0.09245357 -9.80600617 0.04602388]

Calibration configuration

cam0

Camera model: pinhole Focal length: [460.8359042600893, 460.59651287135466] Principal point: [354.44182102814153, 202.07767726099678] Distortion model: equidistant Distortion coefficients: [-0.10387145094103571, -0.03249349371414373, 0.0522382910564682, -0.030513973256192786] Type: aprilgrid Tags: Rows: 6 Cols: 6 Size: 0.088 [m] Spacing 0.0264 [m]

cam1

Camera model: pinhole Focal length: [458.97504833502177, 458.4133972042093] Principal point: [366.09274574886456, 235.02411089106826] Distortion model: equidistant Distortion coefficients: [-0.11659497785718727, 0.03624322804994296, -0.08002202623988071, 0.05188458182376746] Type: aprilgrid Tags: Rows: 6 Cols: 6 Size: 0.088 [m] Spacing 0.0264 [m]

IMU configuration

IMU0:

Model: calibrated Update rate: 184.0 Accelerometer: Noise density: 0.005 Noise density (discrete): 0.0678232998313 Random walk: 0.00433 Gyroscope: Noise density: 0.005 Noise density (discrete): 0.0678232998313 Random walk: 0.00266 T_i_b [[ 1. 0. 0. 0.] [ 0. 1. 0. 0.] [ 0. 0. 1. 0.] [ 0. 0. 0. 1.]] time offset with respect to IMU0: 0.0 [s] <

ZacharyTaylor commented 6 years ago

The output timeshift from kalibr is currently 0.0s. This indicates that it was run without time calibration. Could you rerun it with the --time-calibration flag as this will let us know if there is any offset in timestamping between your camera and IMU

tiralonghipol commented 6 years ago

I feel really stupid.. thank you! What if the seller declared "hardware camera-imu sync" ? Does the --time-calibration flag still valid ?

Adding that flag, btw, leads to a new error: Exception in thread block: [aslam::Exception] /home/pol/kalibr_workspace/src/Kalibr/aslam_nonparametric_estimation/aslam_splines/src/BSplineExpressions.cpp:447: toTransformationMatrixImplementation() assert(_bufferTmin <= _time.toScalar() < _bufferTmax) failed [1.53259e+09 <= 1.53259e+09 < 1.53259e+09]: Spline Coefficient Buffer Exceeded. Set larger buffer margins!

Before you ask, my imu.yaml file that I used for the second step in the calibration (kalibr) is:


#Accelerometers
accelerometer_noise_density: 0.005    #Noise density (continuous-time)
accelerometer_random_walk:   4.33e-03   #Bias random walk

#Gyroscopes
gyroscope_noise_density:     0.005   #Noise density (continuous-time)
gyroscope_random_walk:       2.66e-03   #Bias random walk

rostopic:                    /imu0      #the IMU ROS topic
# update_rate:                 200.0      #Hz (for discretization of the values above)
update_rate:                 184.0      #Hz (for discretization of the values above)
ZacharyTaylor commented 6 years ago

If the system has accurate hardware timestamping than --time-calibration will output something like 1ms or less of offset verifying that the timestamping is accurate.

That error occurs as the system has a larger time offset than kalibr is expecting (default max of 0.01s). Try setting --timeoffset-padding 0.1 and rerunning. However this already suggest the hardware timestamping has some offset in it. Out of curiosity which sensor are you using?

tiralonghipol commented 6 years ago

Here is the link to the sensor: https://www.robotshop.com/en/optor-visual-inertial-camera.html

This is really strange:

After Optimization (Results)
==================
Normalized Residuals
----------------------------
Reprojection error (cam0):     mean 0.339600946353, median 0.260765657464, std: 0.289844298467
Reprojection error (cam1):     mean 0.333649057074, median 0.232928501278, std: 0.358334196077
Gyroscope error (imu0):        mean 1.6110380717, median 1.12279369489, std: 1.55530907923
Accelerometer error (imu0):    mean 0.702226789553, median 0.632598330057, std: 0.401069750157

Residuals
----------------------------
Reprojection error (cam0) [px]:     mean 0.339600946353, median 0.260765657464, std: 0.289844298467
Reprojection error (cam1) [px]:     mean 0.333649057074, median 0.232928501278, std: 0.358334196077
Gyroscope error (imu0) [rad/s]:     mean 0.109265918176, median 0.076151573417, std: 0.105486194011
Accelerometer error (imu0) [m/s^2]: mean 0.0476273380974, median 0.0429049062122, std: 0.0272018739181

Transformation T_cam0_imu0 (imu0 to cam0, T_ci): 
[[-0.99928476  0.0338582   0.01683991  0.06379936]
 [-0.03217929 -0.99527841  0.09157172 -0.01407266]
 [ 0.01986085  0.09096432  0.99565608  0.02065697]
 [ 0.          0.          0.          1.        ]]

cam0 to imu0 time: [s] (t_imu = t_cam + shift)
-0.0167532691232

Transformation T_cam1_imu0 (imu0 to cam1, T_ci): 
[[-0.99947848  0.03209441  0.00356489 -0.03643763]
 [-0.0316184  -0.99508185  0.0938743  -0.01489952]
 [ 0.0065602   0.09371263  0.99557767  0.02203061]
 [ 0.          0.          0.          1.        ]]

cam1 to imu0 time: [s] (t_imu = t_cam + shift)
-0.00588901836098

IMU0:
----------------------------
  Model: calibrated
  Update rate: 184.0
  Accelerometer:
    Noise density: 0.005 
    Noise density (discrete): 0.0678232998313 
    Random walk: 0.00433
  Gyroscope:
    Noise density: 0.005
    Noise density (discrete): 0.0678232998313 
    Random walk: 0.00266
  T_i_b
    [[ 1.  0.  0.  0.]
     [ 0.  1.  0.  0.]
     [ 0.  0.  1.  0.]
     [ 0.  0.  0.  1.]]
  time offset with respect to IMU0: 0.0 [s]
ZacharyTaylor commented 6 years ago

So I have bad news, if this sensor is doing hardware triggering / timestamping than the driver is throwing this information away and just stamping things on arrival in software.

Here is the line in the driver it happens at https://github.com/optor-vis/vi-stereo-v1/blob/master/optor_VI_Sensor_SDK_V1.0/SDK/src/optorcam.cpp#L954

The two cameras have different timing offsets because they are sent one at a time and so have different arrival times. While an offset of 1 or 2ms probably wouldn't be a deal breaker with 16ms of offset things will probably diverge pretty much immediately

tiralonghipol commented 6 years ago

Thanks again for your support and your patience! So now what do you suggest? I think the options are 2: modify the driver o use other sensors, am I right?

ZacharyTaylor commented 6 years ago

If no hardware timestamps exist you don't have that many good options. You can take the values kalibr gives you and code these into the driver to get things a bit more consistent. However, these offsets will change with factors such as your CPU load and the arrival times will usually have a spread of a few milliseconds.

time

Since you know the images and IMU are running at a fixed rate you can implement a simple kalman filter to smooth out this ripple and try hardcode the offset, though I am unsure how accurate this will get you.

Most people using VI sensing have their own custom solution. Of the commercial solutions I know of that we have tested: The S360 sensor - https://zurichsense.com/ expensive and made in low numbers, but we have had good results with it The DUO - https://duo3d.com/ a little pricey and from memory the IMU is heavily filtered. We had one working but not reliably. Realsense ZR300 - out of production, but maybe you can find one. A little sensitive to shocks and we had to modify the provided driver, but it does work. ZED Mini - the maplab guys just got one of these. So far it seems like it works though we are still testing

tiralonghipol commented 6 years ago

many thanks again! Another question (a little off topic): have you ever tried rovio in dark or completely dark environments? or what could be the best camera/sensor for low light condition?

ZacharyTaylor commented 6 years ago

One with a torch attached. Though seriously we have managed to run rovio on images that are so dark a person can barely see whats going on. As a rule of thumb the bigger the sensor and the larger the lens the more light will be captured, however you hit diminishing returns pretty quickly making active illumination a good option for dark enviroments.

tiralonghipol commented 6 years ago

Do you have any rosbag to share? or related video/paper about these experiments?

ZacharyTaylor commented 6 years ago

The darkest flights we have done were probably in the back of the building shown here https://www.youtube.com/watch?v=49WiPHkkhBw

Here is a short section of rovio running on part of one of these flights

peek 2018-08-02 16-54

tiralonghipol commented 6 years ago

This is reeeeally impressive! and also what I was looking for! Thanks again

Razbotics commented 6 years ago

What is the status of rovio running with optor camera? I have a optor camera and i have finished with calibration... but i get slow odometry results

ZacharyTaylor commented 6 years ago

If you are getting slow odometry this usually means that rovio has diverged and it is falling behind real time while it tries to find new features in a large region.

Unfortunately the cause of this will be that tho optor camera relies on poor time synchronization through arrival time timestamping. This combined with the fairly low quality IMU means it is unlikely to ever work well with rovio.

Razbotics commented 6 years ago

Hey thanks for the info...though its a bad news for me...well is there any workarounds for better timestamping?....i found that VINS mono is working quite well with optor cam and i heard that rovio works more robust than VINS mono using optor cam..

Razbotics commented 6 years ago

Is there a way to hack trigger pins of optor camera? I want them for hardware synchronization

ZacharyTaylor commented 6 years ago

I believe VINS has online time offset estimation, which allows it to compensate for these sort of issues. I am unsure about what is possible with the optor. I have never used one, just looked through its driver.

Razbotics commented 6 years ago

Yep ViNS mono can compensate that issue and maybe thats the reason VINS works quite well. Do you have knowledge of hardware pins of a camera module? I can see some accessible pins HS, VS , LED out and PCLK on the back of optor cam. I was hoping if i can use one of them for hardware timestamping and synchronisation.

chenwinki commented 5 years ago

@tiralonghipol Hello, could you give me a hint how to get the param of imu like this ,thank you very much: Accelerometer: Noise density: 0.005 Noise density (discrete): 0.0678232998313 Random walk: 0.00433 Gyroscope: Noise density: 0.005 Noise density (discrete): 0.0678232998313 Random walk: 0.00266

tiralonghipol commented 5 years ago

@chenwinki can you be more specific on your question please? what do you mean by 'get the param of imu' If you are talking about the configuration file (-yaml), you do not get anything, you set it as you like (check che datasheet of your IMU, of course) hope it helps

chenwinki commented 5 years ago

@tiralonghipol Thank you for you advice! I calibrate imu using imu_utils, and calibrate imu and camera using kalibr. but I got bad result, even when I put my VI sensor static, the result shows the VI sensor are moving(about ten meter in 100s), could you give me a hint about the result?

tiralonghipol commented 5 years ago

The Optor / Loitor VI sensor has many problems, being a cheap sensor. The worst one is that the sync between images and imu is not really hardware After many tries with different algorithms I decided to switch to the MyntEye camera and I immediately got good results (after proper calibration, of course) For more details see the discussion above https://github.com/ethz-asl/rovio/issues/192#issuecomment-408401898

mechaliomar commented 5 years ago

@tiralonghipol So, you have been using the mynteye camera successfully! -I am also planning to use it for 6 DOF pose estimation using ROVIO. -Can you suggest something before I start ?

BEST REGARDS UESTC, CHINA

tiralonghipol commented 5 years ago

Hi @mechaliomar I ended up in building my own visual inertial sensor. Unfortunately the commercial sensors I tested during this year were not as good as declared Currently i'm using rovio onboard an autonomous drone and the results are really good https://youtu.be/N7SK1ekzhe0