Issues with Monocular 3D Object Detection Leaderboard Submission

[towardsautonomy]

Hello,

I am trying to submit results to the monocular 3d object detection leaderboard test server, however, I received all zeros within the metrics table. The exact same method worked with validation submission and I received reasonable performance metrics.

The only changes between the two submissions were:

1. For validation submission, I set object.num_lidar_points_in_box=100, because I wanted to be able to do local evaluation as well. I do not think it is necessary for test submission?
2. For test submission, I set metrics_pb2.Object().camera_name to correspond to the camera sensor that the image came from.

Could the above two have caused the said issue? I do not want to lose another opportunity (we only get 3 attempts) trying out these changes, so if someone could help me out here it would be really appreciated.

Thanks, Shubham

[wangbo-zhao]

Hello. When did you submit and get the results? I submitted it yesterday but I still do not get any results.

[towardsautonomy]

You should get the results back within minutes. Unfortunately they don’t show you the error messages (at least I have not seen one) if the submission file has issues.

[hfslyc]

Hi,

Could you share the code on how to generate your submission? In the mean time, I'm checking your submission. The points you mentioned shouldn't affect the evaluation.

[Cc-Hy]

Hi,

Could you share the code on how to generate your submission? In the mean time, I'm checking your submission. The points you mentioned shouldn't affect the evaluation.

@hfslyc

Hello, for the camera-only 3d task, it doesn't matter if I don't modify the "id" field in the submission file because I don't do the tracking task, is that right?

[towardsautonomy]

Hi,

Could you share the code on how to generate your submission? In the mean time, I'm checking your submission. The points you mentioned shouldn't affect the evaluation.

Hi @hfslyc, here is the snippet I am using to save predictions to a .bin file. Following which, I use bazel-bin/waymo_open_dataset/metrics/tools/create_submission to create the submission and submit the gunzipped version to Waymo server.

import os

import glob
from PIL import Image
from io import BytesIO
from src.models import *
from src.utils import *

try:
    from waymo_open_dataset import dataset_pb2, label_pb2
    from waymo_open_dataset.protos import metrics_pb2

except ImportError:
    raise ImportError(
        'Please run "pip install waymo-open-dataset-tf-2-6-0 --user" '
        'to install the official devkit first.')

from glob import glob
import numpy as np
import tensorflow as tf

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # FATAL

class WaymoDataset(object):
    def __init__(self,
                 load_dir,
                 workers=4,):
        self.load_dir = load_dir
        self.workers = workers
        self.filter_empty_3dboxes = True
        self.filter_no_label_zone_points = True

        self.selected_waymo_classes = ['VEHICLE', 'PEDESTRIAN', 'CYCLIST']
        self.selected_waymo_locations = None
        self.save_track_id = False

        # turn on eager execution for older tensorflow versions
        if int(tf.__version__.split('.')[0]) < 2:
            tf.enable_eager_execution()

        self.lidar_list = [
            '_FRONT', '_FRONT_RIGHT', '_FRONT_LEFT', '_SIDE_RIGHT',
            '_SIDE_LEFT'
        ]
        self.type_list = [
            'UNKNOWN', 'VEHICLE', 'PEDESTRIAN', 'SIGN', 'CYCLIST'
        ]
        self.camera_id_to_name = {
            1: 'FRONT',
            2: 'FRONT_LEFT',
            3: 'FRONT_RIGHT',
            4: 'SIDE_LEFT',
            5: 'SIDE_RIGHT'
        }
        self.camera_name_to_id = {
            'FRONT': 1,
            'FRONT_LEFT': 2,
            'FRONT_RIGHT': 3,
            'SIDE_LEFT': 4,
            'SIDE_RIGHT': 5
        }

        self.k2w_cls_map = {
            'Car': label_pb2.Label.TYPE_VEHICLE,
            'Pedestrian': label_pb2.Label.TYPE_PEDESTRIAN,
            'Sign': label_pb2.Label.TYPE_SIGN,
            'Cyclist': label_pb2.Label.TYPE_CYCLIST,
        }

        self.load_dir = load_dir

        self.tfrecord_pathnames = sorted(
            glob(os.path.join(self.load_dir, '*.tfrecord')))

        self.cur_dataset_idx = 0
        self.cur_frame_index = 0
        self.n_frames_this_dataset = -1

    def get_dataset(self, dataset_idx):
        pathname = self.tfrecord_pathnames[dataset_idx]
        dataset = tf.data.TFRecordDataset(pathname, compression_type='', num_parallel_reads=self.workers)
        return dataset

    def get_next_frame(self,):
        """Get frame from dataset."""
        n_datasets = len(self.tfrecord_pathnames)
        if self.n_frames_this_dataset != -1 and \
           self.cur_frame_index >= self.n_frames_this_dataset:
            self.cur_dataset_idx += 1
            self.cur_frame_index = 0
        if self.cur_dataset_idx >= n_datasets:
            print('No more data in the dataset.')
            return None
        if self.cur_frame_index == 0:
            print('Loading Dataset: {}/{}'.format(self.cur_dataset_idx+1, n_datasets))
            self.dataset = self.get_dataset(self.cur_dataset_idx)
            self.n_frames_this_dataset = len(list(self.dataset.as_numpy_iterator()))

        for i, data in enumerate(self.dataset):
            if i == self.cur_frame_index:
                frame = dataset_pb2.Frame()
                frame.ParseFromString(bytearray(data.numpy()))
                self.cur_frame_index += 1
                return frame

# main function
if __name__ == "__main__":

    # parse arguments                
    args = parse_args()

    # define model
    model = ObjectDetector( args )

    # load weights
    model = load_pretrained_weights(model, args)

    # load dataset
    dataset = WaymoDataset(load_dir=args.dataroot)
    out_dir = os.path.join('results', args.exp_id)
    if not os.path.exists(out_dir):
        os.makedirs(out_dir)

    print('Number of tfrecords: {}'.format(len(dataset.tfrecord_pathnames)))

    global_frame_counter = 1
    gt_objects = metrics_pb2.Objects()
    prediction_objects = metrics_pb2.Objects()
    frame = dataset.get_next_frame()
    while frame is not None:
        print('Dataset: {} | Frame: {} | Global Frame: {}'.format(
            dataset.cur_dataset_idx+1, dataset.cur_frame_index, global_frame_counter))

        context_name = frame.context.name
        frame_timestamp_micros = frame.timestamp_micros

        segment_name = dataset.tfrecord_pathnames[dataset.cur_dataset_idx].split('/')[-1].split('.')[0]
        objects = metrics_pb2.Objects()
        for i, image in enumerate(frame.images):
            camera_id = image.name
            # Fetch matching camera calibration.
            calibration = next(cc for cc in frame.context.camera_calibrations
                                if cc.name == camera_id)
            K = np.zeros((3, 4))
            K[0, 0] = calibration.intrinsic[0]
            K[1, 1] = calibration.intrinsic[1]
            K[0, 2] = calibration.intrinsic[2]
            K[1, 2] = calibration.intrinsic[3]
            K[2, 2] = 1
            img = np.array(Image.open(BytesIO(image.image)))

            pred_label_dict = model.predict(img, K, conf_thres=0.7, nms=True)

            # add to waymo metrics object
            for obj in pred_label_dict:
                box = label_pb2.Label.Box()
                obj_pose = SomeTransformation(obj)
                box.center_x = obj_pose['x']
                box.center_y = obj_pose['y']
                box.center_z = obj_pose['z']
                box.length = obj_pose['l']
                box.width = obj_pose['w']
                box.height = obj_pose['h']
                box.heading = obj_pose['heading']

                o = metrics_pb2.Object()
                o.object.box.CopyFrom(box)
                o.object.type = dataset.k2w_cls_map[obj['class']]
                o.score = obj_pose['conf']
                o.context_name = context_name
                o.frame_timestamp_micros = frame_timestamp_micros
                o.camera_name = camera_id
                prediction_objects.objects.append(o)

        frame = dataset.get_next_frame()
        global_frame_counter += 1

    # save to file
    predictions_filename = os.path.join(out_dir, 'predictions.bin')
    with tf.io.gfile.GFile( predictions_filename, 'wb' ) as f:
        f.write(prediction_objects.SerializeToString())
    print('Predictions saved to {}'.format(predictions_filename))

[wangbo-zhao]

@yinzp-simple Hello. Can I ask you how do you generate the submission file? I follow the official tutortial but I still can not get any results from the website.

[SimpleFlyingBird]

@yinzp-simple Hello. Can I ask you how do you generate the submission file? I follow the official tutortial but I still can not get any results from the website.

Just follow the official tutorial. Maybe you can visualize your predictions to have a check~?

[hfslyc]

@towardsautonomy

The bug is actually because of setting camera_name. Please leave them empty when submitting. I've cleared your submissions and you still have 3 quota. Please submit again and let me know it works for you.

Best, Wayne

[wangbo-zhao]

I use the official tool to eval my validation submission and I get reasonable results. But when I submit it to the website, I can not get any results from the website.

yinzp-simple @.***> 于2022年5月20日周五 15:40写道：

@yinzp-simple https://github.com/yinzp-simple Hello. Can I ask you how do you generate the submission file? I follow the official tutortial but I still can not get any results from the website.

Just follow the official tutorial. Maybe you can visualize your predictions to have a check?

— Reply to this email directly, view it on GitHub https://github.com/waymo-research/waymo-open-dataset/issues/491#issuecomment-1132583108, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANR3QJWVS4YOQM2OLNRA43TVK46WLANCNFSM5WKMO6LA . You are receiving this because you commented.Message ID: @.***>

[wangbo-zhao]

@hfslyc Hello. I have tried several times to submit my val predictions to the website but it can not return any results. Everything goes well when I eval it on my computer with your evaluation tool.

[hfslyc]

@yinzp-simple you are submitting the results for the testing set. We have a new test set in testing_3d_camera_only_detection.

Best, Wayne

[hfslyc]

@wangbo-zhao what's your submission email?

[hfslyc]

Hi, Could you share the code on how to generate your submission? In the mean time, I'm checking your submission. The points you mentioned shouldn't affect the evaluation.

@hfslyc

Hello, for the camera-only 3d task, it doesn't matter if I don't modify the "id" field in the submission file because I don't do the tracking task, is that right?

Yes, the id field does not matter.

[wangbo-zhao]

Thanks for your reply. I use @.*** to submit to the validation sever to learn how to submit.

Wayne Hung @.***> 于2022年5月20日周五 15:57写道：

@wangbo-zhao https://github.com/wangbo-zhao what's your submission email?

— Reply to this email directly, view it on GitHub https://github.com/waymo-research/waymo-open-dataset/issues/491#issuecomment-1132597796, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANR3QJSTLHETIZHV4C6CGWTVK5AWNANCNFSM5WKMO6LA . You are receiving this because you were mentioned.Message ID: @.***>

[wangbo-zhao]

@hfslyc caixvkun999@gmail.com

[hfslyc]

@wangbo-zhao "error_message": "The account name acc@domain.com is different from the registered email caixvkun999@gmail.com."

The front end currently has a problem to display error messages. Therefore, if you see a submission which doesn't have results out in 30 mins, please let me know and I can check the error message for you.

Best, Wayne

[wangbo-zhao]

Thank you very much. I will try it later..

Wayne Hung @.***> 于2022年5月20日周五 16:06写道：

@wangbo-zhao https://github.com/wangbo-zhao "error_message": "The account name @. is different from the registered email @."

The front end currently has a problem to display error messages. Therefore, if you see a submission which doesn't have results out in 30 mins, please let me know and I can check the error message for you.

Best, Wayne

— Reply to this email directly, view it on GitHub https://github.com/waymo-research/waymo-open-dataset/issues/491#issuecomment-1132605670, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANR3QJTPJLHCMK6FRQJAUQLVK5BXNANCNFSM5WKMO6LA . You are receiving this because you were mentioned.Message ID: @.***>

[SimpleFlyingBird]

@yinzp-simple you are submitting the results for the testing set. We have a new test set in testing_3d_camera_only_detection.

Best, Wayne

Thank you for your reply! I did use the wrong testing set : ( Could you please help me to clear my two wrong submission(submitted by yinzphnu@gmail.com at 5/20/22 11:36 AM and 5/21/22 5:26 PM). It would be so hard for me to have only one chance last! @hfslyc Thanks a lot!

[hfslyc]

@yinzp-simple We've answered you question through gmail. Thanks!

[towardsautonomy]

Thanks for all the help. I have resolved the issues following @hfslyc's comments.