search

immersive-web / depth-sensing

Specification: https://immersive-web.github.io/depth-sensing/ Explainer: https://github.com/immersive-web/depth-sensing/blob/main/explainer.md
Other
52 stars 15 forks source link

Revise the depth sensing API #11

Closed bialpio closed 3 years ago

bialpio commented 3 years ago

Notable changes:

Maksims commented 3 years ago

I might have missed something, but how to configure API to use both CPU and GPU paths?

bialpio commented 3 years ago

I initially did not plan on allowing that. We could introduce a "mixed" XRDepthSensingUsage where both paths are available if that's something we think is needed, but in this case I think the UA could do no better than what the application could do anyway (so for "cpu-optimized", the app can call texImage2D() w/ the data it got, and for "gpu-optimized", the app can call readPixels() w/ the texture it got).

Related question: do we think it is acceptable for the UA to support a subset of usage modes?

Maksims commented 3 years ago

I believe having two explicit paths in API, is limiting to the developer. Holy grail would be having single API, that works CPU and GPU paths, but providing hint to API, we can improve performance of a specific path.

So instead of single hint to API, we would have two, providing them while creating XRSession. Something like (not the best to name them): depthSensingRawData and depthSensingGpuTexture. By default they could be both enabled, providing full API coverage. If developer wants to optimize, and not using one of the paths, can provide false flags to XRSession or use updateDepthSensingState.

bialpio commented 3 years ago

I believe having two explicit paths in API, is limiting to the developer. Holy grail would be having single API, that works CPU and GPU paths, but providing hint to API, we can improve performance of a specific path.

The restriction is not really a limitation since the app could still perform the upload to/read from the GPU, in a way that should be no worse than what the UA would do.

To me the problem is that from UA's perspective, it is not free to provide access to both paths. I'd like the apps to make a conscious decision when providing the usage hint to the API (I don't want the apps to use "mixed" usage "just in case") - if it is slightly less ergonomic to use but makes it clear that it's potentially costly to access the data differently than the preferred usage says, I think it is a fair trade-off. Maybe instead of "mixed", we should call it "unoptimized" to make the consumers more careful.

Note that it is way easier to remove restrictions from already shipped API than to add them when it turns out they are needed - we should keep that in mind here. :)

Additional note: in my current proposal, the UA can fail the configuration if the app passed in something else than the preferred settings (hence the question in my earlier comment). This is also something that does not reduce the expressiveness of the API, but it does place additional burden on the app writers. OTOH, it means that UA not providing helpers (smaller implementation burden and less code to maintain).

So instead of single hint to API, we would have two, providing them while creating XRSession. Something like (not the best to name them): depthSensingRawData and depthSensingGpuTexture. By default they could be both enabled, providing full API coverage. If developer wants to optimize, and not using one of the paths, can provide false flags to XRSession or use updateDepthSensingState.

That's an interesting idea. I'm not a huge fan of defaults that we know may be wrong from the start (an app may pay for something it didn't actually need if it only uses one path) - I think the approach with the UA reporting what it prefers and then the app attempting to negotiate the exact configuration is better here.

Maksims commented 3 years ago

It feels like this will enforce developers to develop two paths of getting depth information, and compensate for lack of gpu/cpu functions by implementing extra functionality to extract it from what's available. This will lead to a lot of overhead for developer, while it could be enforced by the API, so browsers would have both paths available, with additional hint/toggles (as mentioned before) to optimize for specific use cases.

cabanier commented 3 years ago

Sorry, I had forgotten to publish my feedback :-\