Open mikhail-dcl opened 1 year ago
I have a plan in my head regarding textures: Integrate a library for decoding (FreeImage) Integrate a library for runtime compression (similar to https://docs.unity3d.com/ScriptReference/Texture2D.Compress.html) that can run on a worker thread (I'm talking to Unity to find one) For textures web requests: use a preallocated buffer, just like we discussed before On a background thread decode and compress Take a texture from the pool (in case it is similarly sized we can set data to the internal array directly, otherwise reload the internal array) This technique can be applied to both single textures and gifs
Early research showed the following conclusions:
I integrated A .NET wrapper of FreeImage with ease and was able to run the image decoding on the background thread and then write raw data back to Texture2D
:
This wrapper has Microsoft dependencies that we don't need, we must clean it up before we can use it: consider forking off, providing an API for Texture2D
, and then integrating.
Libraries for Win32/64, MacOS, and Linux were included. FreeImage
itself provides build scripts for many platforms, it should be easy to rebuild in case of a need
Reading pixels looks like this:
var texture = new Texture2D(width, height, format, false);
// TODO add mip-maps with Rescale
NativeArray
void* unmanagedPointer = rawBits.GetUnsafePtr(); // GetBits() is a native function IntPtr unmanagedBits = GetBits(dib);
Unsafe.CopyBlock(unmanagedPointer, unmanagedBits.ToPointer(), (uint)rawBits.Length);
texture.Apply();
I could not find a way to support 8bit textures, I tried using GraphicsFormat
but Unity asserts it (probably a Unity bug). This should not be a problem as we can store such textures as is, they occupy less memory than fully-colored ones
For some reason, the internal representation is BGRA
(Blue and Red channels are swapped) so I had to swap them manually like this. it is happening via the native API on the background thread so can be considered as neglectable
FIBITMAP redChannel = GetChannel(dib, FREE_IMAGE_COLOR_CHANNEL.FICC_RED); FIBITMAP blueChannel = GetChannel(dib, FREE_IMAGE_COLOR_CHANNEL.FICC_BLUE);
SetChannel(dib, redChannel, FREE_IMAGE_COLOR_CHANNEL.FICC_BLUE); SetChannel(dib, blueChannel, FREE_IMAGE_COLOR_CHANNEL.FICC_RED);
UnloadEx(ref redChannel); UnloadEx(ref blueChannel);
It is possible to rescale textures on a background thread and then upload them as mipmaps
It does allocate unmanaged memory that we are responsible to release, it's fine as it does not involve GC
NativeArray
of the Texture: it supports compressed formats on creation.Kudos to you, based on your notes your approach looks promising!
You probably already saw this but just in case it's useful for you in any way, FYI for the WebGL client, the kernel processes GIFs in a webworker/thread, stores the texture and sends a pointer to the texture to unity and in unity the image is loaded from that pointer.
Output should be a texture in a GPU-friendly format (that can be uploaded to GPU without additional conversions, e.g. by
SetRawData
)It's possible to retrieve textures from an external source; in this case it can't be preprocessed
Unity does not expose API for decoding textures on a worker thread. Unity support recommended using an external library, e.g. FreeImage
I tried to integrate it in the past without any success, and it was not straightforward at all. I found a topic where a guy tried to do the same without success either.
Just like for Meshes we should introduce different levels of LODs, LODs can be used as mip-maps