Implement virtualized geometry rendering

novemberist commented 3 years ago

Describe the project you are working on

none specific, but would be relevant for any 3d project

Describe the problem or limitation you are having in your project

Producing high quality game-ready 3D assets is generally a very labour-intensive task for any engine, requiring many specialized steps (sculpting/high-poly modeling/photogrammetry, retopology, baking/creating texture maps, optimizing/tweaking poly count etc.)

Describe the feature / enhancement and how it helps to overcome the problem or limitation

So I guess most people have already taken a look at UE5's impressive new features by now. Now, I know that UE has a very different focus (and a amount of resources) than Godot.

Nevertheless, it is never a bad idea to draw some inspiration from other technology for Godot's long-term future. So maybe this is a good place for discussion, even if implementing similar features in Godot is not realistic at this point.

One of the big new advertised features of UE5 is Virtualized Geometry (called Nanite)

https://docs.unrealengine.com/5.0/en-US/RenderingFeatures/Nanite/

These are the advertised benefits of this new technology:

Multiple orders of magnitude increase in geometry complexity, higher triangle and objects counts than has been possible before in real-time
Frame budgets are no longer constrained by polycounts, draw calls, and mesh memory usage
Now possible to directly import film-quality source arts, such as ZBrush sculpts and photogrammetry scans
Use high-poly detailing rather than baking detail into normal map textures
Level of Detail (LOD) is automatically handled and no longer requires manual setup for individual mesh's LODs
Loss of quality is rare or non-existent, especially with LOD transitions

While the increase in visual quality of games using this technology in the future is immediately obvious, the more interesting fact is, that developement time for 3d assets might be significantly decreased, since the usual tedious process of optimizing and tweaking models will mostly disappear (at least this is what this technology promises)

This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Godot's target audience.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I have no idea, but this is how Nanite in UE5 works according to Epic:

Nanite integrates as seamlessly as possible into existing engine workflows, while using a novel approach to storing and rendering mesh data.

-During import: meshes are analyzed and broken down into hierarchical clusters of triangle groups.

-During rendering: clusters are swapped on the fly at varying levels of detail based on the camera view, and connect perfectly without cracks to neighboring clusters within the same object. Data is streamed in on demand so that only visible detail needs to reside in memory. Nanite runs in its own rendering pass that completely bypasses traditional draw calls. Visualization modes can be used to inspect the Nanite pipeline.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, as this is rendering technology.

Is there a reason why this should be core and not an add-on in the asset library?

This would have to be a core engine feature

2plus2makes5 commented 3 years ago

Yeah it's huge for sure, but i think we should first see if it really delivers what it promises and what are the dowsides.

One last thing:

"This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Godot's target audience."

Realistic assets need realistic animations, sounds, voices etc, all things that a small team can't afford.

novemberist commented 3 years ago

Yeah it's huge for sure, but i think we should first see if it really delivers what it promises and what are the dowsides.

One last thing:

"This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Godot's target audience."

Realistic assets need realistic animations, sounds, voices etc, all things that a small team can't afford.

One of the caveats seems to be that, since mesh data is streamed from disk, users will probably be required to have a fast SSD to make the most of this technology. However, SSDs are of course relatively cheap compared to GPUs.

2plus2makes5 commented 3 years ago

One of the caveats seams to be that, since mesh data is streamed from disk, users will probably be required to have a fast SSD to make the most of this technology. However, SSDs are of course relatively cheap compared to GPUs.

It makes me think 2 things: 1)it was tailored for the PS5 and XBS/DX12. 2)I can see this being used to push some sort of partly installed/partly streamed(under subscription obviously) hybrid future for games. :(

mrjustaguy commented 3 years ago

Partly installed/Partly streamed future ain't coming, it's going to be one or the other, however Streaming has the huge downside of latency and need for good infrastructure so a local pc market will always to some extent exist.. What is actually going on is there's a move to use huge amounts of highly compressed data, not to mention, SSDs are getting faster, and the capacity is going up.

Honestly atm, I don't see a point in Godot Trying to support it unless it isn't too much trouble to do so, which I doubt, also I think Nanite is Proprietary, so we'd need to make our own implementation.. And really like why? Godot isn't a AAA game engine, it's not aiming to be... Not even larger AA teams would don't need That sort of stuff.

novemberist commented 3 years ago

Honestly atm, I don't see a point in Godot Trying to support it unless it isn't too much trouble to do so, which I doubt, also I think Nanite is Proprietary, so we'd need to make our own implementation.. And really like why? Godot isn't a AAA game engine, it's not aiming to be... Not even larger AA teams would don't need That sort of stuff.

Of course I wasn't talking about implementing UE's Nanite in Godot, but rather discussing the feasibility of similar technology in Godot at some point. Why? Because, as I mentioned above, this is not just about AAA-quality graphics, but much more about the workflow when creating assets.

If this technology manages to establishes itstelf, in the future traditional asset creation pipelines (texture maps etc.) might become obsolete quickly, not only for big studios.

Things like photogrammetry (and directly using the resulting models in-engine) are especially interesting for developers that don't have a lot of dedicated 3d artists.

mrjustaguy commented 3 years ago

I'm sorry to drop the ball but, Photogrammetry in the way you're thinking about it is dead on arrival for multiple reasons...

High poly counts
Hard to Properly Use
Is already becoming obsolete (AI)
If you can't photo it, you're toast.

all 4 of those points are being addressed with AI/Machine-Learning which in turn negates the argument for supporting Nanite-like Tech. How is this possible?

AI can generate less points to get the same visual quality and triangulate that
AI can guess most of the Occluded parts, and has a better understanding of Camera Angles, Lighting and such, meaning Less Input is needed for the Output
Photogrammetry is Computationally VERY expensive, AIs doing this stuff, Ain't nearly as computationally expensive.. Here's one example (there are many more different ones too) https://www.youtube.com/watch?v=548sCh0mMRc
Describe what you want - AI will generate. This is already possible for images, and code (search for GPT3)

While yes, these aren't all available yet and there is still a ton of work to do, they're getting better faster and faster, in both Results and Efficiency.

Also one thing to Note, Godot 4 comes with an Auto LOD system, so worrying about Polycounts isn't as big of a deal as it used to be, unless you're real close to the Mesh, and eventually we'll get Mesh Shaders to further drop the polycount rendering issue, as things that ain't on screen, won't be rendered, or well, to a much less extent compared to now as it allows for much much better culling, and instead of having a single mesh that is entirely drawn even if it is only contributing a pixel (if even that) to the picture, it'll draw the meshlets (a part of a mesh) that are contributing to the pixel, that are on screen, instead of the WHOLE mesh.

2plus2makes5 commented 3 years ago

Partly installed/Partly streamed future ain't coming, it's going to be one or the other, however Streaming has the huge downside of latency and need for good infrastructure so a local pc market will always to some extent exist.. What is actually going on is there's a move to use huge amounts of highly compressed data, not to mention, SSDs are getting faster, and the capacity is going up.

Right now a Netflix-like game service either: -completely streams the game with all the problems we know. or -makes you download dozen or hundreds of Gbs.

Instead they could make you download only the base files and then stream all the rest, unlike frames assets could be donwloaded before they are actually needed and once dowloaded they can be used as many times as needed.

I'm ok with this being used only for subscription services, but the many examples of big game companies' greed make me think that it won't stop there.

mrjustaguy commented 3 years ago

I've never seen a netflix-like game service that makes you download dozens or hundreds of GBs, that only sends assets as needed. I've only seen something similar when downloading games to allow you to play while downloading, but it is a full game download still, just allows you to play things as they have downloaded.

Could you please give examples of the 2nd type of game services, that makes you download dozens or hundreds of GBs the way that isn't actually downloading the Game?

Also: I did look up at Nanite, and according to them, the size increase isn't that big, because it does save room on Textures and LODs, and it does seem to be compressed, so on that end it isn't actually as terrifying as it sounds in terms of the actual size added, and for some assets it might in fact decrease the size, as 4k normal maps are HUGE in terms of space usage, and doesn't have the fidelity of an Actual Mesh, and Nanite offers the fidelity, and removes the need for such huge textures, and the actual size is in the ball-park of the current stuff, going both above and below the current equivalent-result technique file sizes depending on the scenario.

aaronfranke commented 3 years ago

Could you please give examples of the 2nd type of game services, that makes you download dozens or hundreds of GBs the way that isn't actually downloading the Game?

Sure, two examples:

Microsoft Flight Simulator 2020
Outerra Anteworld

Note that neither of these games are feasible to make in Godot for other reasons.

2plus2makes5 commented 3 years ago

@mrjustaguy -i guess i wasn't clear, when i was talking about downloading hundreds og GBs i was referring to the download of the entire game, but think how incovenient it is, people has to wait to play, has to have lots of space on the hard disk and once downloaded a game could be pirated...

On the other hand as you and @aaronfranke pointed there are already examples of partial stream of a game, so it's not a new thing. With partial installed partial streamed there would be less wait and space for the user and since the data that's not useful anymore could be removed the game could be never completely on the hard disk so it can't be pirated(well i guess it could still be but it would be harder).

For streaming services that's absolutely a good thing, but my fear is that this could lead to a future where instead of buying a full game with DRM we will buy only the base files and the rest being streamed.

mrjustaguy commented 3 years ago

My only issue with Games as a Service are the fact that they in no way guarantee that I'll be able to play the game in the future, and could die at any moment... I mean when you've got the whole game locally, it's only about your PC being able to run it, and dealing with any DRM or Emulation issues that come up.

Your fear that Games will be insanely large because of Nanite however, are unfounded as Instead of using Normal Maps etc to store data, you store data in actual polygons... and Trust me, Normal Maps take Exponentially more Space AND lose some fidelity.. and what I mean by that is, from 1k x 1k, to 2k x 2k, to 4k x 4k, to 8k x 8k and beyond, the number of pixels to store goes up really fast, and isn't efficiently used in terms of data storage, compared to a well designed Mesh storage system (like Nanite)

The only reason why we use Normal Maps and such is because it's cheaper for the GPU to texture stuff, compared to pushing tons of triangles, and not because Textures are more Space friendly, because spoiler alert, they're often not. It's simply Cheaper to have a Million triangles, compared to 1/10th as many triangles plus LODs and a 4k normal map (equal-ish results) in terms of data storage.. I mean it's like 3 million Vector3 points, versus 300k Vector3 Points, plus like half as many points in LODs, and 4k x 4k pixels which is some 16 Million Pixels, and it goes the same for larger meshes.. 10m triangles vs 8k or higher res normal map required, which would be 64m+ pixels for equivalent output.. Also Let's not forget, that instead of Triangles, you can do some impressive compression of the mesh by instead of having multiple copies of the same vertex, one could store all the Verts, and the data to connect them, separately and gain quite a bit of extra space if there's a ton of vertex sharing among triangles, which means that the actual size of storing the Triangles is even smaller.

But still, atm, I don't see the point for Godot to go down that Route, as the work to make something like this is just too large and not that worth the effort for a small benefit (if any) for the types of projects most are using Godot for and it's not like traditional methods are Horrible on the small scale, it's just that they Poorly scale up with the Increase in Visual Fidelity of modern AAA games.

Really You can think of it as simple efficiency calculation. Current methods are like 40% efficient, new method is 50% efficient, but who cares if you're 10% more efficient if it already takes you practically no effort at 40% efficiency to pull something off? Where it starts to make a difference is when it's at a larger scale, where the effort lost to lower efficiency actually starts to matter.

2plus2makes5 commented 3 years ago

@mrjustaguy the tecnique in itself seems fantastic and i don't have anything against it, but i fear how some greedy companies could use it, the step from buying games that stream from the hard disk to buying games that stream from internet seem way too easy and obvious,

mrjustaguy commented 3 years ago

They don't need the technique to do such a greedy move. they really don't, they could do so easily with current stuff if they wanted to (and much MUCH older stuff), so really no need to worry about the technique incentivizing that move especially as the space required is on par with current techniques, so it really doesn't change anything in that respect.

Are Companies more likely to go down that Greedy route today? Yes, But not because of the technique, it's because They are getting Greedier, because why make a good product and sell it for a reasonable price if you can make a horrible product and sell it for an absurd price? This is something that the Internet allows, especially Digital goods, and they've been inching to the Horrible Product for an Absurd Price for quite a while, it's just that at the pace at which they're doing it they ain't spawning enough outrage to force them to back off, and people start normalizing the new, worse products instead.

Can this be changed? With Capitalism, a hard No, but does it matter? again, no because within a few decades Capitalism will Be dead because it simply won't be able to function because it revolves around the Idea of People Having work to do for which they in return get money, something that won't be true when AI & Robots are able to do everything humans can, and far better than humans can..

However I think this whole segment is Off topic, with the "Partly Installed-Partly Stream" topic in mind that is.

Wolfe2x7 commented 3 years ago

@2plus2makes5 -- I wouldn't worry about gaming turning to streaming. Maybe the likes of Microsoft or Sony would try to go all-in, but any suggestion that Nintendo ever would is hilarious to anyone familiar with Nintendo's ethos and stubbornness, and some genre communities (like the fighting game community) are never going to adopt streaming -- and you can bet they will make a lot of noise about trying to force streaming upon them.

I'm convinced there will always be developers who perceive the negatives of streaming games, continuing to make games for those of us who intend to keep playing games "locally".

On topic, I had one comment:

"This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Godot's target audience."

Realistic assets need realistic animations, sounds, voices etc, all things that a small team can't afford.

Not necessarily always. You can simulate a vehicle or make a super-pretty modernized Marble Madness or Bust-a-Move clone (or more ideas) without necessarily animating or voicing a single character, and there are free resources for sounds, much like textures. Some individuals are very talented with animation; I've been seeing that in the Godot community.

mrjustaguy commented 3 years ago

For creating games (but in general too), the quality of the game will primarily be the result of the following, roughly in this order:

Team Talent (Natural Competence with Computers and such)
Team Problem Solving Capabilities (Intelligence & Creativity)
Team's Dedication (How much they care and put in to the Project)
Tools (But only to a degree, there's a point of diminishing returns with High number of tools & tool features, and too much can turn into negative benefits)

So yeah, a Solo talented, intelligent, dedicated person with decent tools -CAN- beat a group of not so talented, not creative, not too dedicated people with lots of specialized tools... So it's really Mostly about The person doing the stuff, with the tools being complementary to the person, and not being The thing that makes them automatically capable of doing something, they just help

mrjustaguy commented 3 years ago

Ok, so it's possible that I was wrong with the way they're implementing nanite, as they Might not even be Storing Geometry in a normal sense (https://github.com/godotengine/godot-proposals/issues/2793#issuecomment-850319791), as Juan pointed out in his Speculation on it's implementation on Twitter - https://twitter.com/reduzio/status/1398467670056570880

However Considering the Paper that's posted in the Speculations, I Don't think they're using that approach, as Geometry Images are actually Significantly Smaller, compared to Nanite, as 70k faces taking up ~25kb, assuming somewhat linear Face-Size ratio scaling, would mean ~500kb (20x, as 70k x 20 is 1.4m triangles which is close to 1.6m used in UE example that takes ~19mb)

I mean 19mb is a 38x size increase over 0.5mb, so unless UE is using the method, just with much Higher Res Geometry Images, It's probably Not the same method.

UPDATE: Yes, Nanite doesn't use Geometry Images

Ofc, this data is According to The Paper and UE for their respective methods, and who knows how they Actually work & scale irl...

2plus2makes5 commented 3 years ago

I'm happy to see you are all optmist so i hope you are right, so let's return to the topic.

I saw just a pair of videos comparing ue4 and ue5 and well the difference in quantity of stuff on screen is immense.

Unless there's a huge catch or drawback Nanite is clearly the future of 3d graphics, Unity and other 3d engines won't lag behind and will implement their own version, if that will be the case like it or not Godot will have to do the same, not immediately because we still know almost nothing about Nanite, especially the drawbacks, but in a not too distant future.

anunknowperson commented 3 years ago

Essentially, Nanite is just an automatic LOD system. I think it's not that hard to implement. But you need to create some analogy of the Decimate modifier from Blender. When importing meshes, automatically create a certain number of LODs for them. While the project is running, you need to load these LODs from the disk, depending on the distance. Of course, this will be a useful technology for the engine.

mrjustaguy commented 3 years ago

That is not how Nanite works, it's a bit more complicated.

Also Godot already has an Automatic LOD in 4.0, and it's quite visible that it's not the same as Nanite, both visually and performance wise.

TechnoPorg commented 3 years ago

There's going to be a talk at SIGGRAPH 2021 by Epic explaining in detail how Nanite works: http://advances.realtimerendering.com/s2021/index.html

TechnoPorg commented 3 years ago

The slides from the talk have been posted, which include the script, the illustrations, and links to other work that was referenced during the talk, as well as some extra stuff that didn’t make it in due to time constraints. Kudos to Epic for freely sharing this amazing technology, I probably wouldn’t have if I’d come up with it 🙂. http://advances.realtimerendering.com/s2021/Karis_Nanite_SIGGRAPH_Advances_2021_final.pdf

atirut-w commented 3 years ago

I hope Godot will get its own implementation of Nanite, seeing that the graphics in Godot 4.x is already creeping up to UE level(VL lighting, SDFGI, voxel GI, realistic lighting, etc.).

Calinou commented 3 years ago

@atirut-w Please don't bump issues without contributing significant new information. Use the :+1: reaction button on the first post instead.

generrosity commented 3 years ago

There's going to be a talk at SIGGRAPH 2021 by Epic explaining in detail how Nanite works: http://advances.realtimerendering.com/s2021/index.html

I hope this is of interest to this subject. Appologies if not. New SIGGRAPH video on Youtube by a Epic engineers - A Deep Dive into Nanite Virtualized Geometry, 1hr.

https://www.youtube.com/watch?v=eviSykqSUUw

Mikeysax commented 1 year ago

I just saw this and could be useful for a future implementation in Godot given this Nanite-like implementation is made with Vulkan and C++:

https://github.com/milkru/vulkanizer

From the repo:

"Rendering engine prototype made with Vulkan 1.3. Code is written using a simpler C++ coding style. Project is written for the C++20 standard and x64 system. Currently the code is tested only on Windows, using MSVC (Visual Studio) and MINGW (Visual Studio Code) compilers. Linux is not completely supported at the moment, but it should be easy to port, since all third party libraries are cross platform."

Zireael07 commented 1 year ago

@Mikeysax What's "Nanite-like" about this? Looks like yet another Vulkan renderer...

mrjustaguy commented 1 year ago

Meshlet culling, which is basically the same principle as Nanite, just different and simpler implementation that also requires Turing/RDNA2 GPUs because of mesh shaders unlike Nanite which has a Compute Fallback that allows GCN & Maxwell to use it.

CK85 commented 1 year ago

Hi, I'm currently looking into Godot, with the intention to port NanoTech to Godot. NanoTech is similar to Nanite and is mainly based on C# and compute shaders, so it should not be too complicated to port it. For more informations you can join our discord server: https://discord.gg/kKA9UyR4qg

Broken1334 commented 8 months ago

WebGL2 Virtual Geometry (some info about implementation) https://twitter.com/twodelab/status/1732212197894603186

Cluster-based rendering, Nanite-style (200M triangles in 1ms), based on "Multiresolution structures for interactive visualization of very large 3D datasets" paper, author wrote "Most of UE5's Nanite pipeline pretty much exactly follows that paper." https://www.youtube.com/watch?v=7JEHPvSGaX8

VoidEssy commented 7 months ago

Nanite like technology would be a very welcome core addition to the engine allowing high-detail assets to be considered when developing the product. Even from indie perspective if you want to attempt something of higher visual fidelity UE5 allows you to save time and resources offering nanite meaning you don't have to spend time and resources working on and implementing LODs.

I'm glad to see @CK85 in this conversation having tracked his Unity NanoTech work for quite a while before Unity decided to throw everyone under the bus including themselves. PS. Unity China supposedly packs Nanite in itself which is off-topic comment on the matter

JMS55 commented 7 months ago

I've been working on an open source implementation for Bevy. Feel free to copy it. It doesn't require any recent (More than WebGPU baseline) hardware features at the moment, although I will eventually add faster paths for when mesh shaders, atomic u64 texture min/max, subgroup ops, etc are supported. I'd be happy to know that my work helped out Godot. If you have any questions, feel free to reach out (the Bevy discord is the easiest way to get in contact with me).

There's also Carrot and a couple of other projects (both open and closed source) recreating Nanite as well.

The siggraph presentation Unreal gave really explained all the tricks very well. The only secret magic is their simplification and error metrics, and all the various tweaking and heuristic tuning they've done to get it to perform well in real-world scenes.

Broken1334 commented 4 months ago

Detailed article about Virtual Geometry implementation for Bevy by @JMS55 https://jms55.github.io/posts/2024-06-09-virtual-geometry-bevy-0-14

GDC 2024 presentation: Maximizing Graphics Performance with Flexible Virtualized Geometry (by VIRTUOS) https://gdcvault.com/play/1034621/Maximizing-Graphics-Performance-with-Flexible

GDC 2024 presentation: Nanite GPU Driven Materials (by Epic Games) https://epicgames.ent.box.com/s/uepi1joznb0a3mhw7hu5a4m2d6vm61pi/file/1495820437550

luxun0517 commented 3 months ago

I've been working on an open source implementation for Bevy. Feel free to copy it. It doesn't require any recent (More than WebGPU baseline) hardware features at the moment, although I will eventually add faster paths for when mesh shaders, atomic u64 texture min/max, subgroup ops, etc are supported. I'd be happy to know that my work helped out Godot. If you have any questions, feel free to reach out (the Bevy discord is the easiest way to get in contact with me).

There's also Carrot and a couple of other projects (both open and closed source) recreating Nanite as well.

The siggraph presentation Unreal gave really explained all the tricks very well. The only secret magic is their simplification and error metrics, and all the various tweaking and heuristic tuning they've done to get it to perform well in real-world scenes.

Great Work!

VeinSyct commented 1 month ago

Similar techniques is implemented in threejs, see topic https://discourse.threejs.org/t/virtually-geometric/28420/58

godotengine / godot-proposals