Persistent elements

[twojtasz]

Some data that we want to represent does not have a temporal relationship to the scene. It is persistent and fixed.

Another consideration will be management of this data over long periods of time and or distance. What control mechanisms are available to ensure that this data does not accumulate such that it creates a burden on the runtime.

Examples of such data would be map information, signage, or geo-spatial markers.

For the management issue, we could consider long data sets over large distance with mapping data that accumulates well beyond it's usefulness to the current area of focus. How can this data be safely purged?

Some systems, ROS, have multiple ways to specify such information. Such as a flag stating persistence or a pre-specified lifetime on the object.

There are different approaches to achieve this functionality, each with their own trade offs.

Options to specify this data

1. A separate message type A separate message type could be used (instead of state_updates it might be world_updates) where the pose timestamp is not used.

2. Stream metadata attributes We could add an attribute to stream metadata that marks the stream as perpetual, or with a duration.

3. Per-object attributes Allow per object attributes irrespective of the stream

Option 1 feels the best, as it is clear this data is different and distinct from normal state_updates.

Option 2 This would be a second choice, but the burden of labeling every stream as special feels a bit more overhead than I would prefer (very minor issue), but also the processing of validating streams against a list of stream metadata feels less clean than having a separate entity at a higher level (another minor issue maybe)

Option 3 Just increases the processing at run-time, included mainly to address that it is an option that was considered and capture the reasons why it is a bad fit.

Options to manage this data

1. No management for v2.0
2. Manage by bounding box center distance to current pose. Basically anything beyond a fixed distance is free to be purged.

A major problem is is that we have no robust mechanism to request the data that has been purged.

For the code generating the XVIZ data, if tagged with a known distance or duration the XVIZ conversion could re-emit by managing some state on the dynamic conversion side. Or alternatively, the data could simply be periodically sent based on some criteria or state during the conversion process. But w/o robust server-client state management (which is really not a core focus of XVIZ i would say) maybe any data that needs purged would have to use custom injection of data into the seen (as supported by streetscape.gl) rather than be part of the XVIZ pipeline?

It may be nice to allow XVIZ to be used to produce those layers.

[ibgreen]

Just a thought: If writing this much text around proposals, we might want to consider adopting something like the deck/luma micro-RFC system:

https://github.com/uber/deck.gl/tree/master/dev-docs/RFCs

It gives a bit more permanence to these writeups than having them in a soon-to-be-closed ticket.

[jlisee]

I am with Ib we should follow the light weight RFC format used by deck.gl, but until then here are my thoughts on this:

We have another some other use case(s) for this I hit as well:

• Static objects from a simulation world, like parked vehicles or generic obstacles. You might have a lot of these and only want to send them once.

One idea for a development track

1) Static streams & server side state management

This is option 2 combine with leaving it to the server to know what part of the world is local and send it up the client. And when the local world changes enough send up a new model.

Overview

• User marks /map/data and /sim/objects/static as static streams
• The robotics system starts and sends a status_update of type incremental containing and an initial version of /map/data and /sim/objects/static (snapshot streams would no overwrite streams marked static)
• The client renders the data on screen and continues to display over the next 15 seconds, even though other non static data might of declared stale and hidden
• The robotic system advances through the world spatially and sends new version of /map/data relative to that space
• The client replaces /map/data with this latest copy

Pros

• Client side implementation is simple, just modify data lookup rules
• Underlying robotic system most likely maintains a slowly, changing every few seconds vs. at 10 Hz, model like this already
• No need to invent a data fetch protocol

Cons

• Increases complexity a bit on the server side
• No way to see static data at more than local scale
• Does not deal with lifetime needed for ???
• Does not deal with level of detail needed for large scale

2) Next steps

These are some thoughts on where we could go from here. Need more thought on making sure this is the simplest solution we can come up with.

Persistence

So the idea here is what the server is really doing us is showing us windows into a larger world. Only sending us the parts of objects relevant at the current spatial location. If we work under that assumption the front end can simple hold onto these objects. Then do it's own LOD/paging to show those objects as it's rendering budget allows.

If we keep adding features we can provide help on the LOD front.

Object metadata streams

The idea here is we have a stream(s) explicitly that just provides information about an object, and that itself could be static and dynamic. In this case the possible bits of metadata:

• lod/min_distance - distance(m) - show if the viewer is farther away than this
• lod/next_objects - array<object_id> - what objects should be hidden when this is shown
• lifetime/end - timestamp only show before this time, optional
• lifetime/start - timestamp only show after this time, optional
• parent - (FUTURE) use to define dynamic transform network for articulated systems

What this allows is for something like map data to be sent at multiple scales, and to define objects that only live for a certain time period.

Data Fetching

We can have a data_request message that is something like:

• bounding_box - pair of lat/long objects - lat/long coordinates
• time_range - pair of timestamps - timestamp range
• view_distance - distance(m)` - how close are you to the ground plane, helps with maps and LOD data

Then the client can respond with a static_data message that contains a list of stream_sets of static streams outlining this data.

[twojtasz]

https://github.com/uber/xviz/pull/403 Addresses a basic persistent property, not covering the dynamic nature of LOD or data management.