search

godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
91.51k stars 21.26k forks source link

FBX import takes forever (more than 2 minutes for single object) #85074

Open ShlomiRex opened 1 year ago

ShlomiRex commented 1 year ago

Godot version

v4.1.3 stable official and v4.2.rc.custom_build [80de898d7]

System information

Godot v4.2.rc (80de898d7) - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3070 (NVIDIA; 31.0.15.3742) - AMD Ryzen 5 3600 6-Core Processor (12 Threads)

Issue description

When importing an FBX asset (56MB) it takes way too long (2 minutes and 17 seconds, yes I measured it).

On the other hand, blender takes 1-2 seconds.

Steps to reproduce

  1. Download any FBS asset (I used: https://free3d.com/3d-model/nathan-animated-003-walking-644277.html)
  2. Unzip it
  3. Drag & Drop rp_nathan_animated_003_walking.fbx to the scene / FileSystem to import it

Minimal reproduction project

I can't upload it. For some reason, a single node3d and FBX import, the size of the ZIP is 250MB. I'm not joking, the .godot folder size is 211MB (again this is brand new project that I only imported FBX) image

So I can't upload demo project

Possible solution is discussed here: https://github.com/godotengine/godot-proposals/discussions/8476

ShlomiRex commented 1 year ago

Here is breakdown of timings of imports:

Importing:  res:///rp_nathan_animated_003_walking-3646db1cded66d4d1c459753f4ec3434_rp_nathan_animated_003_dif.jpg
Importing: res:///rp_nathan_animated_003_walking-3646db1cded66d4d1c459753f4ec3434_rp_nathan_animated_003_dif.jpg took 56432488 microseconds
Importing: res://rp_nathan_animated_003_walking.fbx took 68747184 microseconds
Importing:  res://rp_nathan_animated_003_walking.fbm/rp_nathan_animated_003_dif.jpg
Importing:  res://rp_nathan_animated_003_walking-3646db1cded66d4d1c459753f4ec3434_rp_nathan_animated_003_dif.jpg
Importing: res://rp_nathan_animated_003_walking.fbm/rp_nathan_animated_003_dif.jpg took 41480208 microseconds
Importing: res://rp_nathan_animated_003_walking-3646db1cded66d4d1c459753f4ec3434_rp_nathan_animated_003_dif.jpg took 58914194 microseconds
res://rp_nathan_animated_003_walking-3646db1cded66d4d1c459753f4ec3434_rp_nathan_animated_003_dif.jpg: Texture detected as used in 3D. Enabling mipmap generation and setting the texture compression mode to VRAM Compressed (S3TC/ETC/BPTC).
Importing:  res://rp_nathan_animated_003_walking-3646db1cded66d4d1c459753f4ec3434_rp_nathan_animated_003_dif.jpg
Importing: res://rp_nathan_animated_003_walking-3646db1cded66d4d1c459753f4ec3434_rp_nathan_animated_003_dif.jpg took 6797221 microseconds

By adding this code to the engine: image

EDIT: After further investigation, this bit of code takes the majority of the time (40 second import for single jpg texture):

image

It takes 37 seconds in this loop.

EDIT 2:

The main issue is the webpacker (Inside webp_common.cpp we have _webp_packer function): image

For each mipmap (there are 14 mipmaps), we call WebPEncode which takes enormous amount of time (47 seconds, 12 seconds, and so on)

The problem is that only the first few mipmaps take extremly long to process:

Mipmap # Seconds it took to process
0 36
1 12
2 3
3 < 0
4 < 0
...
13 < 0
ShlomiRex commented 1 year ago

Like I said the main problem is the mipmaps. The first mipmap size is 36MB which is significantly more than the original texture (5MB). In addition to size inflation, the mipmap output is worse than the original texture:

image

(Left - original texture , Right - mipmap 0)

I extracted all mipmaps into files so I can view them, and we can see the 5MB texture inflated to 52MB mipmaps:

image

EDIT:

I tried importing only the texture (JPG, 5MB, 8k) and it took 41 seconds! The problem is with WebPEncode which takes 39 seconds:

Importing:  res://rp_nathan_animated_003_dif.jpg
Loading image:  res://rp_nathan_animated_003_dif.jpg
Loading image: res://rp_nathan_animated_003_dif.jpg took 1.000000 seconds
_save_ctex:  res://.godot/imported/rp_nathan_animated_003_dif.jpg-98faee6078c63e2d77287f1507154459.ctex
save_to_ctex_format:  res://.godot/imported/rp_nathan_animated_003_dif.jpg-98faee6078c63e2d77287f1507154459.ctex
Mipmaps count:  1
Processing mipmap:  0
WebPEncode start
WebPEncode end: took 39.000000 seconds
save_to_ctex_format: took 39.000000 seconds
_save_ctex: res://.godot/imported/rp_nathan_animated_003_dif.jpg-98faee6078c63e2d77287f1507154459.ctex took 40.000000 seconds
Importing: res://rp_nathan_animated_003_dif.jpg took 41.000000 seconds

So the main problem is importing high resolution textures. Now, WebPEncode calls third-party library: libwebp which we can't modify. I'll try different encoding options to see what works best.

EDIT:

I found temporary solution - using compression mode COMPRESS_VRAM_COMPRESSED and mipmaps = false (do not generate mipmaps) now the FBX import takes 25 seconds (instead of 137 seconds) and single texture takes 4 seconds instead of 41 seconds. If we only use COMPRESS_VRAM_COMPRESSED and mipmaps = true then it will take 3 more seconds for the FBX model to load.

Calinou commented 1 year ago

This is very likely a bug because the Detect 3D mechanism (which works based on textures being assigned to BaseMaterial3D properties) will automatically use VRAM compression for textures, regardless of their size. It doesn't appear to kick in with FBX import in your case, but it should.

It seems FBX import saves its own textures instead of relying on Godot's import system, and that's likely the root cause. It should probably always use VRAM compression by default, with an import option to disable it if needed. We have similar options available for embedded textures in the glTF importer.

ShlomiRex commented 1 year ago

@Calinou I also noticed the texture is inflated in size: image

Left: original FBX files, totaling 60MB Right: inflated import of FBX (totaling 208MB) And the memory of the texture on the right says 192MB.

Calinou commented 1 year ago

@Calinou I also noticed the texture is inflated in size:

Lossless compression generally results in smaller files than VRAM compression, which has a fixed 1:4 compression ratio (1:6 for S3TC DXT1). This is especially the case with textures that have lots of flat color areas, which compress very well with lossless compression but won't compress any better than an highly detailed texture when using VRAM compression.

If you want smaller file size at the cost of import times and quality, use the Basis Universal compress mode. (We could support using Zstandard compression on VRAM-compressed textures at the same time, but this loses the universal aspect of Basis Universal and will result in slower loading times.)

PS: You may not need a 8192×8192 texture for a character, especially if they're not the main character you constantly view from behind in a third-person game. 4096×4096 or even 2048×2048 should be plenty.

Calinou commented 6 months ago

Can you still reproduce this on 4.3.dev6? FBX import is now handled by ufbx, which should be faster than fbx2glTF.

On my system, it takes about 30 seconds to import the 3D scene from scratch, with most time spent in the VRAM compression process:

EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_dif.jpg
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_gloss.jpg
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_mask01.jpg
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_A.jpg
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_mask02.jpg
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_norm.jpg
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_0.png
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_A.jpg" import took 63 ms.
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_mask02.jpg" import took 84 ms.
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_mask01.jpg" import took 903 ms.
etcpak: Encoding image size 8192x8192 to format DXT1 RGB8, with mipmaps.
etcpak: Encoding took 375 ms.
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_0.png" import took 1453 ms.
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_gloss.jpg" import took 5208 ms.
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_norm.jpg" import took 8780 ms.
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/tex/rp_nathan_animated_003_dif.jpg" import took 9236 ms.
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking.fbx
Using present mode: Enabled
FBX path: res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking.fbx
FBX: Total images: 1
Total materials: 1
FBX: Total skins: 1
FBX: Parsing mesh: 0
FBX: Total meshes: 1
FBX: Total lights: 0
FBX: Total cameras: 0
FBX: Total animations '1'.
FBX: Converting spatial: rp_nathan_animated_003_walking
FBX: Converting spatial: rp_nathan_animated_003_walking_CTRL
FBX: Converting spatial: rp_nathan_animated_003_walking_geoGRP
FBX: Creating mesh for: rp_nathan_animated_003_walking_geo
WARNING: Adding 'rp_nathan_animated_003_walking_geo' as child to 'Skeleton3D' will make owner '' inconsistent. Consider unsetting the owner beforehand.
     at: add_child (scene/main/node.cpp:1573)
Saving scene to: res://.godot/imported/rp_nathan_animated_003_walking.fbx-70ede9ed2be310c66cd60a4b5c5f9274.scn
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking.fbx" import took 1272 ms.
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_u3d.fbx
FBX path: res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_u3d.fbx
FBX: Total images: 0
Total materials: 0
FBX: Total skins: 1
FBX: Total meshes: 0
FBX: Total lights: 0
FBX: Total cameras: 0
FBX: Total animations '1'.
FBX: Converting spatial: rp_nathan_animated_003_walking
Saving scene to: res://.godot/imported/rp_nathan_animated_003_walking_u3d.fbx-7a3205819f580c588109854c6e77dc56.scn
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_u3d.fbx" import took 33 ms.
EditorFileSystem: Importing file: res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_ue4.fbx
FBX path: res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_ue4.fbx
FBX: Total images: 0
Total materials: 0
FBX: Total skins: 1
FBX: Total meshes: 0
FBX: Total lights: 0
FBX: Total cameras: 0
FBX: Total animations '1'.
Saving scene to: res://.godot/imported/rp_nathan_animated_003_walking_ue4.fbx-58af6c6dd031502eb3949053efefc224.scn
EditorFileSystem: "res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_ue4.fbx" import took 31 ms.
Generated 'res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking.fbx' preview in 32 usec
Generated 'res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_u3d.fbx' preview in 21 usec
Generated 'res://55-rp_nathan_animated_003_walking_fbx/rp_nathan_animated_003_walking_ue4.fbx' preview in 20 usec

Make sure to test this on a fully optimized editor build (production=yes SCons option).

https://github.com/godotengine/godot/pull/91535 should make it possible to achieve much faster VRAM compression speeds in the future, although that PR is only focusing on HDR formats for now, not SDR (most game textures are SDR).

PC specifications - **CPU:** Intel Core i9-13900K - **GPU:** NVIDIA GeForce RTX 4090 - **RAM:** 64 GB (2×32 GB DDR5-5800 C30) - **SSD:** Solidigm P44 Pro 2 TB - **OS:** Linux (Fedora 39)