Extend the XVIZPrimitiveBuilder API to link more information (like velocity, acceleration, orientation) into /tracklets objects

[tkt028]

Motivation Context: In my project, we have rich data (like velocity, acceleration, orientation) associated with each tracklet object (non-Ego vehicle). PROBLEM: Currently, XVIZPrimitiveBuilder API [1] just support: style(), polygon(), polyline(), points(), etc, but not support a way to associate extra information (as mentioned above) with each tracklet object. This fact creates a blocker for features like: we want to have special processing / display when users select a tracklet object in streetscapes.gl. Note that I am aware of the Data Stream labels which is a data stream independent from the data stream /primitives, this solution is not good enough because for a given tracklet object, there is no way to look up associated extra information in Data Stream labels.

Wish XVIZPrimitiveBuilder API allows to associate more information to tracklet objects so that we can add more features which are specific to each tracklet object.

NOTE This feature would support ticket in ref[2] because we have more information (such as orientation) of each tracklet object. If a tracklet object is selected, we want the Camera to follow that object and the Camera orientation should respect the orientation of the vehicle.

[1] https://github.com/uber/xviz/blob/master/docs/api-reference/xviz-builder.md [2] https://github.com/uber/streetscape.gl/issues/256

[tkt028]

I make the code changes for this request, but I have no write access. Please help consider this change and merge into the master. Thank you very much!

// Write permission error in my machine
ERROR: Permission to uber/xviz.git denied to tkt028.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I attach here the patch. 0001-xviz-417-Extend-the-XVIZPrimitiveBuilder-API-to-add-.patch.txt

How it works? I simply add a method "data()" to XVIZPrimitiveBuilder API to contain the extra data associated with the primitive objects in the property base so that this information is available for application-level features.

Sample code to use this method

                xvizBuilder
                    .primitive('/tracklets/objects')
                    .polygon(points)
                    .data({
                        "velocity": 123.5,
                        "acceleration": 30.3,
                        "orientation": orientationQuarternion
                    })
                    .id(objectId);

The sample output of XVIZ frame

{
  "type": "xviz/state_update",
  "data": {
    "update_type": "snapshot",
    "updates": [
      {
        "timestamp": 1534883787.068,
        "poses": {
          "/vehicle_pose": {
            "timestamp": 1534883787.068,
            "mapOrigin": {
              "longitude": -121.9318621943,
              "latitude": 37.3146625269,
              "altitude": 12.1044184975
            },
            "position": [
              0,
              0,
              0
            ],
            "orientation": [
              0.786244846,
              -0.5200232285,
              -0.0794160647
            ]
          }
        },
        "primitives": {
          "/tracklets/objects": {
            "polygons": [
              {
                "vertices": [
                  [
                    -121.9318762094,
                    37.3146411597,
                    11.3361635925
                  ],
                  [
                    -121.9318911258,
                    37.3146555697,
                    11.3402691875
                  ],
                  [
                    -121.9318484293,
                    37.3146837662,
                    11.2606567824
                  ],
                  [
                    -121.9318335129,
                    37.3146693562,
                    11.2565511838
                  ]
                ],
                "base": {
                  "object_id": "1",
                  "style": {
                    "height": 1.6122326993,
                    "fill_color": "#FF0000",
                    "stroke_color": "#50B3FF",
                    "stroke_width": 10
                  },
                  "classes": [
                    "Car"
                  ],
                  "data": {                 // NOTE: the extra data are here.
                    "velocity": 17.0942451587,
                    "velocityVector": {
                      "x": 6.5738332458,
                      "y": -0.4282775614,
                      "z": 3.8728041155
                    },
                    "acceleration": 9.9139034013,
                    "accelerationVector": {
                      "x": 0.0414319802,
                      "y": -2.412983682,
                      "z": -1.8184643239
                    },
                    "orientation": {
                      "w": 0.8958744974,
                      "x": 0.3604790233,
                      "y": -0.2519889424,
                      "z": 0.0629708786
                    }
                  }
                }
              }
            ]
          }
        }
      }
    ]
  }
}

Sample screenshot in Streetscapes.gl of how frame data look like

[twojtasz]

So we actually do support some associated data, but not in the way you are thinking.

The way we support this is with the id method when you build the streams. https://github.com/uber/xviz/blob/master/examples/converters/kitti/src/converters/tracklets-converter.js#L80

As you can see in this code, we create separate streams for each datum velocity, etc Then the front-end application will merge attributes for streams with a matching id.

This is how we show this information when you click on an object.

Now, this works but the data you are passing may run into some issue. The main stream in XVIZ for such data is our time_series. The scalar values are fine as they map to just a number, but your vector and mtx values won't fit nicely in our current data model (we don't provide an array type for time_series)

A short-term hack would be to turn it into a string :(

I will add an issue for adding array valued time_series data

[twojtasz]

So there are ways to track an object today (I'll be adding to some FAQ/examples in the next few weeks). However it doesn't track the orientation of that object. That would be cool and will add that to the issue.

[twojtasz]

Huge thanks for taking the time to work this out! Will work to make sure our use-case are covered.

[twojtasz]

See https://github.com/uber/xviz/issues/422

[tkt028]

Hi @twojtasz, Thank you very much for your below explain about how XVIZ organizes the data structure.

So we actually do support some associated data, but not in the way you are thinking.

The way we support this is with the id method when you build the streams.
https://github.com/uber/xviz/blob/master/examples/converters/kitti/src/converters/tracklets-converter.js#L80

As you can see in this code, we create separate streams for each datum velocity, etc
Then the front-end application will merge attributes for streams with a matching id.

I think this approach is OK for simple feature like rendering a Text Label when users click a vehicle object. For feature like [1] which make the Camera to follow a selected vehicle object, it's important to be able to look up the orientation information of the vehicle so that we can adjust the Camera View parameters. Since I didn't find a way to look up the orientation information if I have another data stream, so I think it makes sense to attach the extra information to a vehicle object in a tightly-coupled way. Currently, this solution works very well in my project for the feature [1]. But if you have a better way, please feel free to suggest it because I don't want my codes diverge from the AVS code base. Thank you very much!

[1] https://github.com/uber/streetscape.gl/issues/256