Troubleshooting Reproduction of Paper Results on UrbanNav Dataset

[TanguyHardelin]

Hello, I hope this message finds you well. We are currently working on reproducing the results presented in your paper on the UrbanNav dataset and have encountered some difficulties. In your paper, you reported the following impressive results:

We are eager to reproduce these results; however, our own experiments have yielded significantly different and unsatisfactory outcomes. For instance, we obtained results that are far from our expectations:

We have come across the open issues regarding the incorrect extrinsic Camera/IMU calibration for the UrbanNav dataset, specifically Issue#27 and Issue#17. Considering this, we are interested in understanding the configuration you utilized in your paper to compute the reported results. In your paper, you mentioned using the left image from ZED camera. Could you please share the camera-to-IMU transformation that you employed?

In order to remove source of errors, we proceeded the data with the INS + GNSS configuration, which was the default setting in your repository. Here is the content of our configuration file:

# GICI offline pseudo-real-time RTK/IMU TC estimation with ROS topics as input and output
stream:
  streamers:
    - streamer:
        tag: str_ros_gnss_rov
        type: ros
        topic_name: /gici/gnss_rover
        io: input
        format: gnss_raw
        enable_observation: true
    - streamer:
        tag: str_ros_gnss_ref
        type: ros
        topic_name: /gici/gnss_reference
        io: input
        format: gnss_raw
        enable_observation: true
        enable_antenna_position: true
    - streamer:
        tag: str_ros_gnss_eph
        type: ros
        topic_name: /gici/gnss_ephemeris
        io: input
        format: gnss_raw
        enable_ephemeris: true
        enable_ionosphere_parameter: true
    - streamer:
        tag: str_dcb_file
        output_tags: [fmt_dcb_file]
        type: file
        path: /gici/COD0MGXFIN_20211370000_01D_01D_OSB.BIA
        enable_time_tag: false
    - streamer:
        tag: str_ros_imu
        type: ros
        topic_name: /imu/data
        io: input
        format: imu
    - streamer:
        tag: str_tc_solution_file
        input_tags: [fmt_tc_solution_file]
        type: file
        path: /gici/rtk_tc_solution.txt
        enable_time_tag: false
    - streamer:
        tag: str_solution
        type: ros
        topic_name: solution
        queue_size: 5
        io: output
        format: pose_stamped
    - streamer:
        tag: str_solution_odometry
        type: ros
        topic_name: solution_odometry
        queue_size: 10
        io: output
        format: odometry
        subframe_id: Body
    - streamer:
        tag: str_solution_path
        type: ros
        topic_name: solution_path
        queue_size: 100
        io: output
        format: path

  formators:
    - formator:
        io: input
        tag: fmt_dcb_file
        type: dcb-file
    - formator:
        io: output
        tag: fmt_tc_solution_file
        type: nmea 
        use_esa: true
        talker_id: GP

estimate:
- estimator:
    tag: est_rtk_imu_tc
    type: rtk_imu_tc
    input_tags: [str_ros_gnss_rov, str_ros_gnss_ref, str_ros_gnss_eph, fmt_dcb_file, str_ros_imu]
    str_ros_gnss_rov_roles: [rover]
    str_ros_gnss_ref_roles: [reference]
    str_ros_gnss_eph_roles: [ephemeris]
    fmt_dcb_file_roles: [code_bias]

    str_ros_imu_roles: [major]
    output_tags: [fmt_tc_solution_file, str_solution, str_solution_odometry, str_solution_path]
    output_align_tag: str_ros_imu
    output_downsample_rate: 40
    compute_covariance: true
    rtk_imu_tc_options:
      max_window_length: 3
    rtk_options:
      use_ambiguity_resolution: true
    gnss_estimator_base_options:
      use_outlier_rejection: true
      reject_one_outlier_once: false
      max_pesudorange_error: 4.0      
      max_phaserange_error: 0.06      
      max_doppler_error: 0.5
      gnss_common:
        min_elevation: -20.0           
        min_SNR: [35.0, 30.0]
    imu_estimator_base_options:
      imu_parameters:
        sigma_bg: 1.0e-3
        sigma_ba: 1.0e-2
        sigma_g_c: 1.0270904839480961e-02
        sigma_a_c: 1.1197412605492375e-02
        sigma_gw_c: 9.1355383994881894e-05
        sigma_aw_c: 1.1751767903346351e-04
      body_to_imu_rotation: [90.0, 0.0, 0.0]
      body_to_imu_rotation_std: 5.0
      car_motion: true
      car_motion_min_velocity: 3.0
      car_motion_max_anguler_velocity: 5.0
    estimator_base_options:
      max_iteration: 5
      num_threads: 4
      max_solver_time: 0.05
      verbose_output: true
      solver_type: dense_schur
      trust_region_strategy_type: dogleg
    ambiguity_resolution_options:
      system_exclude: [R]
      min_percentage_fixation_uwl: 1.0
      min_percentage_fixation_wl: 0.9
      min_percentage_fixation_nl: 0.9
      ratio: 2.0
    gnss_imu_initializer_options:
      max_iteration: 30
      num_threads: 4
      max_solver_time: 0.5
      gnss_extrinsics: [0, 0.86, -0.31]
      gnss_extrinsics_initial_std: [0.0, 0.0, 0.0]
      time_window_length_slow_motion: 0.05
      time_window_length_dynamic_motion: 0.5
      min_acceleration: 0.5

logging:
  enable: true
  min_log_level: 0
  log_to_stderr: true
  file_directory: /logs

We used the /imu/data topic provided by the dataset's ROSBag, while all GNSS-related topics were converted from RINEX to ROSBag using your tool on the unstable branch. To ensure compatibility, we made minor modifications to integrate the QZSS service. We have uploaded these changes in the following merge request: https://github.com/chichengcn/gici-open/pull/48

Following your instructions, we executed the replay as follows:

rosrun gici_ros gici_ros_main <path to our configuration file>

Followed by ROSBag replay with all generated ROSBag.

rosbag play *.bag

The ROSBag files containing the ephemeris data and gnss_reference data were generated from RINEX files downloaded from the website mentioned in the dataset. We also generated the ephemeris messages for all configurations. Despite these efforts, we have been unable to reproduce the expected results. We believe there may be something obvious that we are missing on our end. Is there anything specific that could explain our poor results? Additionally, we would greatly appreciate it if you could provide us with the necessary steps to reproduce the results presented in your paper using this dataset.

Thank you for your attention to this matter, and we look forward to your response. Best regards, Tanguy

[chichengcn]

Hi,

I found several improper parts from your descriptions.

First, I'm not sure if your reference stream (/gici/gnss_reference) and ephemerides (/gici/gnss_ephemeris) have been generated correctly. In our experiment, we downloaded these data from the Hong Kong Satellite Positioning Reference Station Network (SatRef) and converted them into rosbags. We uploaded the rosbag files, as while as the raw files we have downloaded, onto OneDrive. You can replay our rosbags together the UrbanNav rosbags by rosbag play *.bag.

We have uploaded our configuration file running the rosbags above, see config. This file also contains the extrinsics parameters you asked for. However, the parameters are still too imprecise, so we enabled real-time extrinsics estimation to overcome this.

Given the very bad results use have provided, I think there should be many warning and error logs outputted by GICI. If your progress still do not work properly using the above ways, you can check them out.

Besides, one of the reasons for naming the "unstable" branch is that the rosbag converted from rinex is not stable. Since the timestamps of the rinex ephemerides is always ahead of the observations, we piled all the leading ephemerides at the begining of the rosbags, which may cause rostopic bursting when start playing the rosbag. During this short period, some of the ephemerides data may lost and hence the corresponding satellite cannot be used in a long period, causing performance downgrading. This issue happens on some of our test computers while the others can run properly. So you should notice this issue when replaying UrbanNav.

There is another issue I should mention. The rosbag provided by UrbanNav is too big, so that you should use a very fast storage device to replay them in real-time even for 1x speed. My computer (top level gaming computer) suffered from occasional lagging when replaying the raw rosbags, To overcome this, I filtered the useless data out by rosbag filter to make the rosbag just remaining the ZED left camera and IMU data.

By the way, our algorithms do not support QZSS currently, so the decoders do not need to support it either. So I'm sorry, I will not accept your pull request.

Hope that my response is helpful to you.