Closed qiaozhijian closed 1 year ago
Code is as follows.
import os
import json
import tensorflow._api.v2.compat.v1 as tf
import numpy as np
import open3d as o3d
from waymo_open_dataset import dataset_pb2 as open_dataset
import cv2
tf.enable_eager_execution()
class OpenLane(object):
def __init__(self, dataset_dir):
self.dataset_dir = dataset_dir
def fetch_data(self, segment, timestamp):
gt_json = os.path.join(self.dataset_dir, 'lane3d_1000/training', segment, '{}.json'.format(timestamp))
with open(gt_json, 'r') as fp:
gt_dict = json.load(fp)
image_path = os.path.join(self.dataset_dir, 'images', gt_dict['file_path'])
img = cv2.imread(image_path)
lane_all = []
for lane in gt_dict['lane_lines']:
xyz = np.asarray(lane['xyz']).reshape(3, -1).T
lane_all.append(xyz)
lane_points = np.vstack(lane_all)
return img, lane_points
def visualize_pcd(points):
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(points)
o3d.visualization.draw_geometries([pcd])
def lanemap(filename, openLane_dir):
dataset = tf.data.TFRecordDataset(filename, compression_type='')
segment = filename.split('/')[-1].replace('.tfrecord', '')
openLane = OpenLane(openLane_dir)
world_frame = None
lane_map = []
for cnt, data in enumerate(dataset):
frame = open_dataset.Frame()
frame.ParseFromString(bytearray(data.numpy()))
timestamp = "{:<018}".format(frame.timestamp_micros) # openlane seems to use 18 digit
image0 = next(c for c in frame.images if c.name == open_dataset.CameraName.FRONT)
calibration = next(cc for cc in frame.context.camera_calibrations if cc.name == image0.name)
ex0 = np.array(calibration.extrinsic.transform, dtype=np.float32).reshape(4, 4)
vechile_pose = np.array(image0.pose.transform, dtype=np.float32).reshape(4, 4)
cam0_pose = vechile_pose @ ex0
img, lane_points_c = openLane.fetch_data(segment, timestamp)
# lane_points_c = lane_points_c[lane_points_c[:,0]<20]
if world_frame is None:
world_frame = cam0_pose
T_wc = np.linalg.inv(world_frame) @ cam0_pose
lane_points_w = lane_points_c @ T_wc[:3, :3] + T_wc[:3, 3].reshape(1, 3)
# visualize_pcd(lane_points_c)
lane_map.append(lane_points_w)
lane_map = np.vstack(lane_map)
visualize_pcd(lane_map)
if __name__ == '__main__':
# Replace this path with your own tfrecords.
filename = '/media/qzj/Extreme SSD/datasets/waymo/raw_data/segment-9568394837328971633_466_365_486_365_with_camera_labels.tfrecord'
openLane_dir = '/media/qzj/Extreme SSD/datasets/OpenLane/'
lanemap(filename, openLane_dir)
@qiaozhijian thanks for your effort and advice on this dataset. We are trying to reproduce the problem and fix it. In fact, 3D lanes filter by projection, consistent with the your second way, have been applied. Due to the interpolation operation, some outlier points error may be caused.
Refer to the issue 31, it may causes this problem. In new openlane, this problem will be effectively improved.
Refer to the issue 31, it may causes this problem. In new openlane, this problem will be effectively improved.
Thanks, Ricard Lee. I agree with you that coordinate system may cause this problem. But better yet, check whether this problem is solved using new openlane referring to my code. I know producing GT, especially for far lane points, is very difficult and of a lot work. So, feel free!
@qiaozhijian Thansk , before release, we will check this problem again.
I'm sorry that I had a mistake in my code.
lane_points_w = lane_points_c @ T_wc[:3, :3] + T_wc[:3, 3].reshape(1, 3)
should be
lane_points_w = lane_points_c @ T_wc[:3, :3].T + T_wc[:3, 3].reshape(1, 3)
I accumulate the 3D lane GT using pose provided by origin waymo dataset. I find there is a serious blur of accumulated 3D submap in the segment(segment-9568394837328971633_466_365_486_365_with_camera_labels).
Let's look at the first small section of this submap. There are even some outliers such as red points.
For easy understanding, the corresponding images are as follows
![Screenshot from 2022-09-08 14-44-05](https://user-images.githubusercontent.com/21232185/189054589-199bc1be-24a0-4db6-8069-c4689e558367.png)
I think there are two ways to improve the quality of map: First, use only the closer lane points. For example, 20m