Qiang-Xu
commented
1 year ago Hey @nnop, this is a great question. We started with a image dataset internally, it was heavily inspired by COCO dataset, basically using JSON to represent a relational schema. Then we began to build fusion datasets and we basically took the same method since our developers were already familiar with the setup and not to mention there was code there which could be reused.
In general, I think JSON isn't a bad choice because
- it is portable. You only need to download the file(s), that's it. MySQL/PostgreSQL/MongoDB, there is an extra dependency, we'll need to either release a DB snapshot or use containers.
- it is easy to use/understand. Anyone can open it with his/her favorite editor within seconds and start something within minutes.
There are developers from various background: robotic, ML, engineering, etc, I do think this serves everyone well with very little learning curve.
There are downsides as you may know too:
- It is slow and memory consuming. We'll have to load the whole JSON into memory before doing anything.
- We'll have to write lots of tedious code to query the data, basically simulate what SQL does.
Internally, we've switched to SQLite(which is also portable) late in 2018 and there is an ORM layer(SQLAlchemy) on top it so that our developers don't bother with SQL most of the time. Unfortunately, nuScenes started before that, we didn't get the time/bandwidth to make it to nuScenes.(nuPlan is a modified version of what we have internally today but relational schema isn't very scalable for huge datasets)
nnop
Thanks for the work on providing such an awesome AD dataset. The schema is just a database design. But, why don't use MongoDB or SQL to manage the data? The data query world be quite easy.