search

FabianFG / CUE4Parse

Apache License 2.0
282 stars 116 forks source link

[FR] Support for deserializing 5.1+ maps with WorldPartition #95

Open Kein opened 1 year ago

Kein commented 1 year ago

UE version: 5.1.1 Example map: https://pomf2.lain.la/f/ddhmwfxw.7z

Currently it fails at IOEntry reader:

OverflowException: Array dimensions exceeded supported range.
This exception was originally thrown at this call stack:
    CUE4Parse.UE4.IO.IoStoreReader.Read(long, long) in IoStoreReader.cs
    CUE4Parse.UE4.IO.IoStoreReader.Extract(CUE4Parse.UE4.VirtualFileSystem.VfsEntry) in IoStoreReader.cs
    CUE4Parse.UE4.IO.Objects.FIoStoreEntry.Read() in FIoStoreEntry.cs
    CUE4Parse.UE4.IO.Objects.FIoStoreEntry.CreateReader() in FIoStoreEntry.cs
Kein commented 1 year ago

Small update: the generated tiles-maps are deserialized sucessfully. But the root persistent map/WP map itself fails. Case of commercial games: Talos Principle 2

cozzbp commented 10 months ago

This is also happening with Palworld.

yretenai commented 9 months ago

The reason for this is because the world ubulk file is more than 2 GB. The asset and export data is usually around 300 MB. The ubulk file is responsible for a lot of the lighting and terrain data.

There needs to be significant refactors to how deserialization is done for large files. Unreal loads these on a chunk-by-chunk basis as they are accessed, CUE4Parse (and by extension every unreal asset processor) loads the entire file into memory.

A quick workaround is to lazy-load the ubulk until needed, or just not load large ubulks at all and silently fail things that need it.