search

BuildingVolumes / Unity_Geometry_Sequence_Streaming

A package and tools for Unity to stream large Pointcloud and Mesh frame sequences from disk
https://buildingvolumes.github.io/Unity_Geometry_Sequence_Streaming/
MIT License
15 stars 2 forks source link

visionOS Support #7

Open ddhvr opened 3 months ago

ddhvr commented 3 months ago

Will it be supported on this platform? It works fine in the editor but it always shows the errors as you can see in the attached files. Tryna to build a demo in the AVP(Apple Vision Pro), but nothing showed up in the headset. It seems like it cannot load data from StreamingAssets path.

Screenshot 2024-05-14 at 6 05 57 PM
ChristopherRemde commented 3 months ago

Hey! Yeah the file reading, as well as the Pointcloud shader support both are problems that we need to address. But luckily, we're trying to build an apple vision pro app with this package as well at the moment, so support for the AVP should come in the next weeks!

ddhvr commented 3 months ago

I'm excited that you're porting it to visionOS. If you need to test anything on the AVP, just let me know. I have the device and am ready to help.

ChristopherRemde commented 3 months ago

Hey thank you very much for the offer! Luckily, we also have a device available.

Just one more information that would be helpful for me: Was the sequence that you tried to load a Pointcloud, mesh, or mesh & texture sequence?

ddhvr commented 3 months ago

I tried the mesh and texture one, but they were not synced when I used "play to device" on the simulator. You might encounter the same issue, so I hope there's a good solution.

The attached file should help you see the difference more clearly.

https://github.com/BuildingVolumes/Unity_Geometry_Sequence_Streaming/assets/104874300/6b9bcb11-4fc0-4d43-86ea-d2d36a78331f

ChristopherRemde commented 3 months ago

Wow nice volumetric video quality! I can't quite tell, but the mesh/geometry seems to be playing fine, right? It's just an issue with the textures?

deephog commented 3 months ago

Hi Chrostopher,

I'm DD's colleague Yao. Thank you for sharing your amazing project and being so helpful on solving this AVP issue.

It doesn't seem obvious in this sequence since the actor is not moving much, but geometry-wise, the sequence playback is normal. The problem is, because it is a per-frame reconstructed sequence, each frame has its own set of uv and texture map. When mesh and texture are temporally misaligned, textures may get mapped onto wrong portion of the mesh, which is our primary suspicion of what is happening. Actually we managed to make frames temporally coherent as much as possible, the mapping only drastically change every about 10 frames, which is why you see the change of texture pattern happens less frequently than the actual frame rate. But I guess the temporal misalignment is larger than 10 frames that we cannot even see one frame with proper texture mapping.

ChristopherRemde commented 3 months ago

Hey Yao, thank you very much! Great to see the package being used :)

I designed the playback in such a way that this de-synchronizing should never be occuring, unless something goes really wrong. In this case it seems like some textures can't be loaded, which throws an exception and disturbs the loading order. This is probably a fault with the metal API. I'll update and test the package with the Metal API in mind and will implement support for the AVP in the next weeks. Please bear with me, for some time, but I'll let you know as soon as possible when this is done!

If you encounter any other issue or have any other feedback or critique please let me know!

deephog commented 3 months ago

Thank you for the insights! Looking forward to your implementation, and if you need us to do more tests, please let us know.

ChristopherRemde commented 3 months ago

Thanks, this is great! :) Will test your sequence on the AVP when everything is ready!

Just to be sure, this is a public forum and anybody can download your sequence. My email is in the bio if you want to take it private!

ddhvr commented 3 months ago

Thanks, I've emailed it to you.

ddhvr commented 3 months ago

Hi Chrostopher,

How's the process going?

Best, DD

ChristopherRemde commented 3 months ago

Hey DD

I'm working on it! But it'll take some more time to complete it, as I can only work on it for a small portion of my time. I think it should be ready in around 3-4 weeks!

ddhvr commented 2 months ago

Hi Chrostopher,

Any updates?

Best, DD

ChristopherRemde commented 2 months ago

Hey DD,

This update has unfortunately taken a lot more effort than I thought. I needed to rewrite a larger part of the plugin, but in return the performance has also increased quite a bit! (10-20x times in my test cases), which will also help a lot to run this on mobile devices, like the AVP.

I've finished about 80% of the features so far, so should be up and running soon!

ChristopherRemde commented 1 month ago

Finally got to test the new version and your sequence running on the AVP and I think I could pin down the main issue with your sequence.

The plugin currently uses .DDS textures, which are not supported on mobile Chipsets/devices. This is the reason why you could see the geometry, but not the textures.

I'll implement support for ATSC texture compression for mobile devices now, which should enable the plugin to run textured sequences on iOS and Android!