Feature: guidance for converting custom 3D tracking data to nuscenes format

michael-hoss commented 3 months ago

I have the following functionality available in a private repository that wraps my fork of nuscenes-devkit:

convert my own non-nuScenes 3D object tracking data into nuscenes classes
convert these classes to dicts, then dump them to the nuscenes json files
call the tracking eval directly from python (not from shell) to compute the metrics

If you guys are interested in having this functionality (docs and helper classes for conversion) also here in the main repo, let me know (cc @whyekit-motional). I could then move this stuff into my nuscenes-devkit fork and open a PR whenever I find time.

whyekit-motional commented 2 months ago

@michael-hoss thanks for these suggestions!

Could you elaborate more on these functonalities?:

What do you mean by your "own non-nuScenes 3D object tracking data"?
Which nuScenes classes are you referring to? Is it TrackingBox?
How do you envision "calling the tracking eval directly from python"? I think users can already run the tracking eval via python by doing something like this?:
```
from nuscenes.eval.tracking.evaluate import TrackingEval

nusc_eval = TrackingEval(...)
nusc_eval.main(render_curves=True)
```

michael-hoss commented 2 months ago

What do you mean by your "own non-nuScenes 3D object tracking data"?

I mean completely custom data, in my particular case for example simulated collaborative object perception data from the artery simulator https://github.com/riebl/artery, which yields me both ground truth and predictions.

Instead of writing my own devkit for visualization, metrics evaluation etc. I just convert it to nuscenes and use nuscenes-devkit.

Which nuScenes classes are you referring to? Is it TrackingBox?

Pretty much like TrackingBox (with default values and file output), but the classes I refer to exactly describe the schema of the json files that need to be present on disk for the evaluation to work:

for ground truth, the nuscenes schema, like Attribute, CalibratedSensor etc.
for data under test, the classes for the submission and tracking results (see https://www.nuscenes.org/tracking -> "Results format"), which is almost equal to TrackingBox already.

When I convert my own custom data to the nuscenes json format, I find it easier and less error-prone (wrt. dict key spelling) if I can first convert my data into such Python classes rather than dicts.

Or do such classes already exist in the codebase and I have overlooked them (besides TrackingBox)? Could they be generated by the object-relational mapping that you use?

see code example

```python @dataclass class Attribute(NuScenesWritable): """Properties for different types of objects/road users""" token: Guid name: str = "vehicle.moving" description: str = "Vehicle is moving." json_filename: str = field(default="attribute.json") def to_dict(self): dict_representation = asdict(self) dict_representation.pop("json_filename") return dict_representation ``` where `NuScenesWritable` and `Guid` are helper classes to make things work more smoothly ```python @dataclass class NuScenesWritable: """Base class for all nuScenes classes that correspond to a json file""" @property def json_filename(self) -> str: raise NotImplementedError("This field must be implemented by the subclass.") def to_dict(self) -> dict: # like the `serialize` function in `TrackingBox` raise NotImplementedError("This field must be implemented by the subclass.") ``` ```python class Guid(str): """Custom string class for GUIDs""" def __new__(cls, value=None): if value is None: value = str(uuid4().hex) elif not cls.is_valid_guid(value): raise ValueError("Invalid GUID format") return super().__new__(cls, value) @classmethod def is_valid_guid(cls, value: str) -> bool: # Empty strings are allowed upon creation. # Convention: they will be replaced by an actual guid later on. # Some may stay empty if e.g. a sample has no predecessor. if (isinstance(value, str) and len(value) == 32) or value == "": return True return False ```

How do you envision "calling the tracking eval directly from python"? I think users can already run the tracking eval via python by doing something like this?:

Oh true, sorry for the confusion. That works well and I also use it just like this. Just had to look into the code because as far as I remember, the docs were more about the CLI.

whyekit-motional commented 2 months ago

@michael-hoss thanks for the detailed explanation! :100:

For the conversion of custom tracking data into nuScenes classes, we will leave it to individual repos to implement this in their respective codebases (otherwise, nuscenes-devkit will become bloated with multiple converters)
For conversion of classes to dicts / json, TrackingBox (in addition to other classes like DetectionBox) already supports this via the serialize method: https://github.com/nutonomy/nuscenes-devkit/blob/4df2701feb3436ae49edaf70128488865a3f6ff9/python-sdk/nuscenes/eval/tracking/data_classes.py#L308-L310

michael-hoss commented 2 months ago

Thanks for the quick response!

nutonomy / nuscenes-devkit