Request: Point Clouds with an option to fix the point size, instead that the point size get smaller when the user get closer.

[MartnDe]

Hello Neos Team.

I am experimenting in Neos with real volumetric video recordings. At the moment I use 2 Azure Kinect locally matched and recorded(Frontand Back) (soon 4 sensors from all sides). I export then PLY Streams, this stream I import to Neos as they are PLY Point Cloud they have the typical behaving of an Point Cloud. I mean that the point size gets smaller when the user get closer. I would love to have the chance to set the point size to a fixed size not only to keep the solid look of an point cloud as well to reduce the points and rise the point size accordingly to the imported file that I can maybe even reduce the amount of points. I experimented with 300 frames and of an 30fps recording each frame is around 10mb, this is already a 60% compression of the original size and I could compress to 85 or 90% (5mb or 3,5mb) if I would be able to fix or adjust the point size ...

Short form: request of adjustable point size for PLY for example as an float or an int input option.

... I hope this will be possible ...

Thx and greatings Martn D. https://www.martnd.com https://vimeo.com/martnd https://sketchfab.com/MartnDe

Somehow by this message it is now public ... that Neos is one of the first Social VR platforms which can handle real Volumetric Video. (Not yet kinect live stream, but you can walk around the recording and see the recorded character really from all side so it is real volumetric and not "fake volumetric" with 2d video and green-screen).

I would be more then happy to show my volumetric recording result inside of Neos to the Neos team, I plan as celebration of volumetric video (fluent 30fps) inside of NEOS a open house concert with recorded artists inside of my private "digital twin" flat which I imported in NEOS, but this i will announce on discord when I am ready.

[H3BO3]

Have you tried adjusting the settings on an Unlit material for your point cloud? Specifically the ones pointed out.

[H3BO3]

The UsePerBillBoardScale setting is optional, that depends on if you have size data in your cloud or not.

[MartnDe]

Have you tried adjusting the settings on an Unlit material for your point cloud? Specifically the ones pointed out.

I never have seen this in my inspector "UnlitMaterial" ... I have to test this directly ... how do i get this attached ?

Maybe this is the solution ? .... i thought about during import to decide either "normal" point cloud or "optional fixed size" Point Cloud .... but maybe your mentioned solution is already the solution i have to simple test it ...

[H3BO3]

I visited with Martn for a while and was able to narrow down his issue to a few points (lol), with possible solutions.

He is rendering a 6.2M-vertex point cloud mesh and the points are becoming too small when close up.

We tried to switch it to use billboard geometry, but that rendered 6.2 million quads, one for every point, and also pointed every single quad at the viewer synchronously, causing massive frame drops. This solution will not be feasible.

Is it possible to render the points larger or set a relative/absolute scale for them? Points not disappearing after sub-pixel size would be the most desirable result.

In one iteration of his mesh, there was a high external update time while he was looking in the direction of the mesh that wasn't present before. Is that related to the GPU? Is there anything that could have caused a new-ish issue with that, like drivers, changes on neos, etc. He said it started occurring around the 21st to the 23rd of May.

[Frooxius]

You need to use billboard geometry if you want to control the point size, that's done by rendering a quad, so unfortunately there's no way getting around that.

However all the pointing of the quads is done in geometry shader on the GPU, so it should be pretty fast. It's quite a lot of points, so it'll be quite heavy on the GPU still. However it's hard to say anything without knowing specifics, notably:

• What GPU are you using?
• What drivers are you using?
• What's the GPU load when looking at the mesh? (ideally measured by GPU-Z)
• What was the actual framerate you experienced with this?
• Do you perhaps have a mesh collider on the object?

[H3BO3]

Here is his hardware

Here is mine

We were both getting very consistent frame drops, only when looking in the direction of the mesh, causing a lot of quads to be rendered. I don't believe I saw a mesh collider while configuring the item, that would have resulted in much larger problems in my experience, and I don't believe it would have been affected by frustum culling.

I wasn't able to get actual GPU load numbers unfortunately. I'm sure Martn might be able to grab those, or myself, if I have access to the meshes again.

[MartnDe]

What ever is needed I will go now inside of neos and will grab and document what ever is needed to find a solution.

With the issue of the dots in compare billboard squares ... the issue is that the squares need to be directed into the viewing direction (needs render power) and the aesthetic changes drastically so for moving volumetric film recordings squares are destroying the fine detailed look of the volumetric. -> A Dot a round object dont needs to be redirected it can be loaded and thats it ...

("ongoing extra thoughts"= as well we could think about LOD or like a fixed points size for import each PLY fixed during import but that the render of the dots only happens when you look in this direction and only for a certain amount of distance to the user then the PLY handling would be even better for large scale worlds and better then any other engine ;) )

That is why I asked for a different up-loader in which the user maybe give the PLY (point cloud) a point size for initialization ... or an option to change the points size at inspector ....

for performance of many points I have much more then 6 million .... in other worlds ....

I have one world in which ... i had before the change (weekend 21.05.2021) Virtual Germany Exhibition Room - Volumetric Showcase by Martin Demmer PLY1 = 4,6 Million Points - A photogrammetry of a sound recording Studio from Gdansk PLY2 = 440.000 Points - A volumetric capture single frame of an human PLY3 = 1,7 Million Points - A photogrammetry of an forest area with a bridge PLY4 = 1 Million Points - A house with trees and park behind PLY5 = 5 Million Points - A staircase photogrammetry plus a GLB model trasformed (by CloudCompare) into PLY then melt PLY6 = 5 Million Points - A photogrammetry PLY from a Flat plus one 5fps 30frames asset of PLY dancers

and all was fluent before the update 21.05.2021

-> then from the named weekend in May 2021 I have to switch off the PLY5 & PLY6 to get the world fluent ...

Yesterday we tested as well second world "Home Concert World" PLY5 = 5 Million Points - A staircase photogrammetry plus a Model as GLB trasformed (by CloudCompare) into PLY PLY6 = 5 Million Points - A photogrammetry PLY from a Flat PLY7 = 6 Million Points - A photogrammetry Street with several houses and backyard plus nature around Plus 30fps Ply Recording 300 Frames for 10seconds one running at 30fps each frame 6MB up to 10MB (10mb file = 300.00 Points)

This "Home Concert World" runs fine to the moment we switch to Billboard squares

Graphic Card is a NVIDIA 1660 Super 6GB VRam CPU = AMD Ryzen 7 5800X 8-Core Processor, 4200 Mhz, 8 Core(s), 16 Logical Processor(s) Main RAM = 32GB internet connection = 1gigabit down / 50mbit up - RJ45 connection Link Quest1 128GB version from USBC 3.1 Gen2 Port - Cable USB-c to USB-c NeosVR started with Neos Luncher and Rift Settings

Thx and Greetings Martn D.

[Frooxius]

I'm not sure I fully understand the problem. The Unlit material in billboard mode already lets you adjust the point size through the inspector, it's a functionality we already have.

Is performance the only issue with this? I'm not too sure what could be done there, since it sounds like you're rendering very detailed geometry, usually I'd recommend keeping the total under 2 million points/triangles total for smooth frames, but many of these seem to be already over that.

We haven't changed anything with rendering that would affect performance of this. Perhaps driver update could've affected it somehow?

However with that amount of geometry I think you're simply running into the limits of computational power of your GPU, especially since you get performance problems when you look in the direction of the mesh.

Checking the GPU usage in GPU-Z can help there. I'd also potentially check VRAM usage, it's possible you're getting over the limit.

[H3BO3]

@Frooxius Having the billboard as a quad changes the style he's going for, but he can't adjust the point size without it. That's the problem the first paragraph is talking about

[Frooxius]

I'm having trouble understanding what exactly is the desired style though? If you want to change the size, it needs to be a quad so it can cover multiple pixels. If you want it to be a single point (pixel), then you can't change the size.

You can assign a texture to the Unlit material and it will use it for each of the quads, so that gives you control over how they look.

[H3BO3]

A simple texture could be looked into, true. But I am wondering why the point just disappears once it gets too close/small. I've worked with GPU particles in unity and they didn't exhibit the same effect from what I remember, they just stayed a constant pixel size on the display.

I think the desired style is how it looks now, without the billboard, but without the particles becoming completely invisible as the user approaches.

[Frooxius]

It shouldn't disappear at all, but stay a single pixel large. Usually that's very tiny in comparison when you get really close, so you get smalls screen coverage.

[MartnDe]

I'm not sure I fully understand the problem. The Unlit material in billboard mode already lets you adjust the point size through the inspector, it's a functionality we already have.

But, sadly the billboard mode needs much more power, as the squares or even if they would been disks need to be directed into the view of the user ... and a ball or a 3 dimensional point which would be scale-able would not be need to be directed ...

Is performance the only issue with this?

for the volumetric it is an optimization handling request ... as with scale able points i could reduce points ...

I'm not too sure what could be done there, since it sounds like you're rendering very detailed geometry, usually I'd recommend >keeping the total under 2 million points/triangles total for smooth frames, but many of these seem to be already over that.

as I try to explain there was a moment the same PLY all at that Showcase room didn't create an issue since around then they do ?

We haven't changed anything with rendering that would affect performance of this. Perhaps driver update could've affected it somehow?

at the same day when it started to be different a MultiTool Switcher "3x1t5tyl3" which i had at my inventory and which was several hundred MB big shrink-ed down in size to 21MB and the functions where lost ??? just as an help to fix the day or update when it happens ....

However with that amount of geometry I think you're simply running into the limits of computational power of your GPU, especially since you get performance problems when you look in the direction of the mesh.

yes i agree as unity 2020.3.1 dont handles more then around 10million points when i experiment in unity ... but really i had it all at the same room before day X and all was fine it was one of the reasons why i thought WOW neos is the plattform for PLYs :)

Checking the GPU usage in GPU-Z can help there. I'd also potentially check VRAM usage, it's possible you're getting over the limit.

I downloaded and checked lick you requested and YES ... the VRAM get full and performance goes up when i look into the direction of serval or super big point clouds ...

still it wasn't like that before day X ;)

Anyway ...

here a clip which shows the working Volumetric ply streams ... maybe we find a way without billboard to change the point size it would be amazing ...

i will add one link to a clip I did inside of neos NEOS with the volumetrics out of fun ... in which the double bass player gets more invisble as i approach ...

Neos Clip 1 = https://vimeo.com/560845184

... and two to clips of an different projects which I do in unity with fixed point size ... here i can go and move through the PLYs and they stay how they are ....

Unity clip example 1 - fixed point size = https://vimeo.com/528441357/7d9571be62

Unity clip example 2 - fixed point size = https://vimeo.com/523885289

as well the GPU-Z log and screenshot

GPU-Z Sensor Log.txt

greetings Martn D.

i hope we can find a way ...

[Frooxius]

I wouldn't recommend using spheres for the points, as that would perform even worse. With a quad, you have 4 vertices and 2 triangles per point that need to be rendered. With a ball you need significantly more, which will drive up the cost even more.

If you want to control the size, you need to use billboard rendering, there's no other way around it. Rendering a single point is faster, because it doesn't need to do all the extra work for rendering larger size.

From the performance statistics it just looks like you're hitting your GPU hardware limits. This is a bit tricky thing, because there's not that much that can be done on our end, you'll need to optimize the content itself.

One thing we could try is have a lower quality, but a bit more efficient way of positioning the quad. That could reduce some of the GPU upload with such large meshes, but I'm not sure by how much.

[MartnDe]

Frooxius i found an super old code with doesn't need graphic power at all at least it looks like and it has I build splitting of too big Point-clouds in sub instances and it takes the points and does triangles out of it ... then you can choose and apply different materials or forms to map on this triangles and then for example the 13million point cloud is splittend and not all loaded at same time and these disks (that's how they appear) are not redirecting like billboards ... the visual result is quite beautiful and safe extrem performance and is then an obj after the import ... with texture file what makes the handling super easier ... the code is around 6 years old and has some holes and not needed functions but it is a beautiful shader or points to geometry importer ... this splitting and later loading for big clouds and this option of transform into triangles is outstanding for creative usage ... I recorded you a video from in engine of my unity project ... https://vimeo.com/564114157/c8912e95bd ... I guess the video explain it easier ... I had already send you that link over discord in a DM but here again ... maybe it inspires you or other coders how to do a step in this direction ... thx and looking forward to your reaction ... (this message is written from phone sorry for not better formatting) greetings Martn