Compress and Resize Textures

[mikhail-dcl]

Output should be a texture in a GPU-friendly format (that can be uploaded to GPU without additional conversions, e.g. by SetRawData)

It's possible to retrieve textures from an external source; in this case it can't be preprocessed

Unity does not expose API for decoding textures on a worker thread. Unity support recommended using an external library, e.g. FreeImage

I tried to integrate it in the past without any success, and it was not straightforward at all. I found a topic where a guy tried to do the same without success either.

Just like for Meshes we should introduce different levels of LODs, LODs can be used as mip-maps

[mikhail-dcl]

I have a plan in my head regarding textures: Integrate a library for decoding (FreeImage) Integrate a library for runtime compression (similar to https://docs.unity3d.com/ScriptReference/Texture2D.Compress.html) that can run on a worker thread (I'm talking to Unity to find one) For textures web requests: use a preallocated buffer, just like we discussed before On a background thread decode and compress Take a texture from the pool (in case it is similarly sized we can set data to the internal array directly, otherwise reload the internal array) This technique can be applied to both single textures and gifs

[mikhail-dcl]

Early research showed the following conclusions:

• I integrated A .NET wrapper of FreeImage with ease and was able to run the image decoding on the background thread and then write raw data back to Texture2D:

  • This wrapper has Microsoft dependencies that we don't need, we must clean it up before we can use it: consider forking off, providing an API for Texture2D, and then integrating.

    • Libraries for Win32/64, MacOS, and Linux were included. FreeImage itself provides build scripts for many platforms, it should be easy to rebuild in case of a need

    • Reading pixels looks like this:

      var texture = new Texture2D(width, height, format, false);

      // TODO add mip-maps with Rescale NativeArray rawBits = texture.GetPixelData(0);

      void* unmanagedPointer = rawBits.GetUnsafePtr(); // GetBits() is a native function IntPtr unmanagedBits = GetBits(dib);

      Unsafe.CopyBlock(unmanagedPointer, unmanagedBits.ToPointer(), (uint)rawBits.Length);

      texture.Apply();

    • I could not find a way to support 8bit textures, I tried using GraphicsFormat but Unity asserts it (probably a Unity bug). This should not be a problem as we can store such textures as is, they occupy less memory than fully-colored ones

    • For some reason, the internal representation is BGRA (Blue and Red channels are swapped) so I had to swap them manually like this. it is happening via the native API on the background thread so can be considered as neglectable

      FIBITMAP redChannel = GetChannel(dib, FREE_IMAGE_COLOR_CHANNEL.FICC_RED); FIBITMAP blueChannel = GetChannel(dib, FREE_IMAGE_COLOR_CHANNEL.FICC_BLUE);

      SetChannel(dib, redChannel, FREE_IMAGE_COLOR_CHANNEL.FICC_BLUE); SetChannel(dib, blueChannel, FREE_IMAGE_COLOR_CHANNEL.FICC_RED);

      UnloadEx(ref redChannel); UnloadEx(ref blueChannel);

    • It is possible to rescale textures on a background thread and then upload them as mipmaps

    • It does allocate unmanaged memory that we are responsible to release, it's fine as it does not involve GC

• I tried to integrate Compressonator to compress into Desktop format at runtime
  • This project is super mature and feature-reach, supporting the most modern formats such BC7
  • It provides scripts for every standalone platform so it can be built with ease
  • I didn't find any wrappers for C#
  • So I started to write them on my own, but I didn't finish. I don't see any limitations why it should not work: it's a pretty standard flow: just write interop bindings, include native libraries for each platform, and do whatever you want. Then take a pointer to the natively allocated array of colors, and write it to the NativeArray of the Texture: it supports compressed formats on creation.

[pravusjif]

Kudos to you, based on your notes your approach looks promising!

You probably already saw this but just in case it's useful for you in any way, FYI for the WebGL client, the kernel processes GIFs in a webworker/thread, stores the texture and sends a pointer to the texture to unity and in unity the image is loaded from that pointer.