Remove pillow dependency and find material images by traversing node tree

[Mysteryem]

This is a big PR focussed mainly on removing the dependency on Pillow and rewriting how images are found from materials. A few fixes and other implementation changes are also included.

I've squashed as many commits as I could that were simply fixing bugs in my own code and wouldn't result in messy history rewrites, roughly halving the number of commits compared to my development branch.

There's a lot of changes here and the ctypes code in particular took me plenty of time to learn, if you've got any questions, please ask.

Removing the Pillow dependency

The Pillow dependency can cause headaches for some users, particularly Mac or Windows Store users who have troubles installing it. Additionally, its limitation of having to read images from disk has caused issues with .blend files that contain packed images, such as .vrm imports by default. Fairly recent changes to Cats' development branch can also result in all mmd base textures being replaced with new images that are not backed by a file when doing Cats' Fix Model.

The main work with removing the Pillow dependency was not having it be really slow on Blender 2.82 and older, because the fast access methods for image pixels were only added in 2.83. The naïve solutions of iterating through every pixel run 10s of times slower than the fast methods added in 2.83.

Generally I find that this new implementation runs about a second or two faster than the implementation with Pillow (even on Blender versions older than 2.83), though the difference will depend on the number and size of the images, as well as any images that have to be tiled to account for uvs that go outside of the [0,1] bounds and any images that need to be resized. Additionally, Blender doesn't load images from disk until it needs them, so the times can change depending on how many of the images have already been loaded from disk.

• numpy is used to replace most of Pillow's image operations
  • Pasting images into the atlas has been implemented according to Pillow's paste function and coordinate system in order to keep the keep the code as similar as possible. This was the main function from Pillow that I had to implement myself.
  • Resizing images has been implemented by making a temporary copy of the image to be resized and using the Image's scale function that is native to Blender
• Each image's pixels are retrieved from the image as it is loaded into Blender
  • Any image type Blender supports can therefore be used
  • Generated and packed images are now supported
  • Any unsaved changes to images are also used when creating an atlas
  • Image colorspaces are now used according to how they are set in Blender, so long as they are sRGB, Raw, Linear or Non-Color (images are almost always sRGB or are specifically set by the user as Non-Color after being opened in Blender).
  • Attempting to atlas images with other colorspaces is not supported and will raise an exception. This is because their conversions are more complicated and hidden away as Blender internally uses OpenColorIO for them. While Blender 3.2 has added these OpenColorIO color conversions to the Python API, they only operate on a single Color object at a time, which would be far too slow to use on entire images
  • If an image's colorspace does not match the colorspace of the atlas (hardcoded as sRGB for now) it is converted to match
• Blender 2.82 and older does not have the fast access methods for image pixels meaning that typically, each pixel RGBA component would have to be iterated one at a time, which is about 30-40 times slower when reading pixels and about 10-14 times slower when writing pixels.
  • For those versions of Blender there are different solutions to maintain similar speed to the fast access methods depending on the Blender version. Each different solution has its own file and some are extremely complicated and hacky due to the use of ctypes, I have tried to comment them as best as possible and there are a couple of links at the top of ctypes_base.py that were very helpful for me to understand how to set it all up.
  • Reading pixels on 2.82 and older
  • 2.80 to 2.82 reads image pixels using OpenGL by using an undocumented feature added in 2.80 that allows a bgl.Buffer to share the same memory as an existing numpy buffer.
  • 2.79 reads image pixels using OpenGL and uses ctypes to hack into a bgl.Buffer and temporarily set its data pointer to point to the data of an existing numpy buffer.
  • Writing pixels on 2.82 and older
  • 2.79 to 2.82 writes images by using the fact that Blender has specific, and more importantly, fast code that handles setting bpy_prop_array types (like Image pixels) to a Matrix. Using ctypes, the Matrix's data pointer is changed to point to the existing numpy buffer of pixels and the Matrix and image's .pixels have their number of dimensions and length of each dimension temporarily changed to match in order to write the correct number of pixels and pass checks.
  • The only case where there is no solution (and therefore slow iteration is used) is in Blender 2.79 when an image internally uses a float buffer, e.g. images of the type PNG16 or OpenEXR. During testing I found that these images could not be loaded into OpenGL in Blender 2.79. This issue is fixed in 2.80 so it is not a problem in newer versions.
  • In cases where there is no solution, an image fails to load into OpenGL for an unknown reason or one of the solutions using ctypes fails to pass its checks when loading, the fastest iterative methods I could find for accessing image pixels are used. The one exception is on 2.79 where a slightly slower method may be used, but at 1/4 of the memory usage, which makes it significantly faster when low on free memory.

    Finding images (and colors) from materials

    Custom materials or edited materials rarely work correctly currently due to very specifically named nodes needing to be used, often resulting in no image being found for a material or the wrong image.

I have rewritten how images are found from Materials. In this new implementation, Material Combiner will find the active Material Output nodes of each Material in 2.80+ that uses nodes and traverse the node tree backwards from those Material Output nodes, attempting to find supported nodes by following the links between nodes.

For Blender 2.79, the implementation is effectively unchanged.

• Specific support is included for the VRM and MMD group nodes
• Specific support is included for the Specular, Emission, and BSDF shaders, though the Glass, Transparent and Refraction BSDF shaders may produce unexpected results due to their inputs other than color being ignored.
• Mix Shader nodes with Fac input set to exactly 0 or 1 will traverse the correct input according to the Fac value. In the future, it might be possible to modify the code to mix together both inputs according to other Fac values, but I've made no attempt at implementing this.
• In addition to Image Texture nodes, RGB nodes are also supported
• If any node is given the name (not label) "MaterialCombinerOverride" it will be used as the starting point of node traversal. This is useful when using custom material setups that cannot be traversed from the Material Output node to find the correct image/color/shader/group, e.g. the node connected to the material output node is an Add Shader or Mix Shader node (with Fac not being 0 or 1).
• If this system is unable to find any image or color from a material and there is exactly one Image Texture node in the node tree, it will use that one Image Texture as a last resort
• I've attempted to design it in such a way that it should be possible to modify the implementation, in the future, such that when it finds a Principled BSDF shader, VRM/MMD group node or other node with multiple inputs that it also finds the color/image for the metallic/roughness/normals, allowing for similar behaviour to the old multi-combining. Currently, each MaterialSource has only an image and a color attribute, but this could be extended to include metallic, roughness, normal_map etc.

  Other implementation changes

• Image sorting Previously, images would be sorted only by the minimum of their width and height, which could result in unstable sorting and suboptimal sorting when two images had one dimension the same, but one image was much larger in the other dimension. Image sorting in the atlas has been changed to sort according to this order:
  1. maximum of width and height
  2. area
  3. width
  4. image name ('' if no image)
  5. color (white if no color)
• Removed CombineList's 2 type (0 was the Object type and 1 was the Material type). As far as I could tell, 2 wasn't used for anything.
• Disabled the combine operator from supporting the "redo toolbar panel" (the popup in the bottom left corner of the 3D view after saving an atlas that lets you tick on/off only the Cats property of the operator since the other properties are hidden). This is done because the combine operator does not work with this panel, raising a ReferenceError whenever the Cats property is toggled, presumably because it first performs an undo, breaking the references of all the Materials in the Material List and then tries to run the operator again with the changed property, but now with all broken Material references.

  Other new features

• Blank (single color) generated images that have no unsaved changes are treated as their generated_color property instead of as an image.
• The cog/gear UI in the Material List now shows the name, size and icon of the image found from the Material in question so that users are able to check the images before making the atlas

  Fixes

• Properly fixed the issues with missing material previews on Blender 3.0+ (the previews will now be generated when they don't exist)
• Fixed max output image size check always returning False even when a dimension was above 20,000 pixels
• Fixed Custom atlas size not affecting UVs

  Other changes

• Dictionaries that were being used like classes have been replaced with classes
• Type hints for many of the combiner_ops functions and new classes have been added (I couldn't add any variable type hint because they don't exist in Blender 2.79's Python version)
• Many variables in combiner_ops have been renamed
• Timing info is now printed to the console when the combine operator is run
• Added constants for CombineList type values to globs.py
• Replaced globs.version with specifically named constants such as globs.is_blender_2_79_or_older

[Mysteryem]

I should add, that I've only been able to test on Windows. There is a possibility of some of the ctypes code not working on other operating systems, though I would hope that the vast majority of people are using Blender 2.83 or newer anyway where that code isn't used.

[Kosyne]

Does this fix or lay the framework for fixing #15 by any chance?

[Mysteryem]

Does this fix or lay the framework for fixing #15 by any chance?

It's only laid a framework. Currently when it finds a Principled BSDF node, for example, it will only attempt to find an image/color from the Base Color input.

The extension I have in mind is to additionally attempt to find an image/color for each of the other inputs (or only specific, hardcoded inputs to start with for simplicity) and then let the user pick which of the found extra inputs they want to include as additional output atlases.

---

I already built in support for exporting atlases with a Linear colorspace instead of sRGB. It's currently hardcoded to always be sRGB, so this would need to be selectable per atlas output.

Some extra code would be needed to allow for traversing the Normal Map node and something would probably need to be done to support the Invert node in cases of smoothness textures being used since Blender tends to use roughness.

There are extra complications when you consider that different materials could be using different shaders/group nodes with differently named inputs. Some inputs with different names might be the same sort of texture and there could even be cases where inputs with the same names are actually different textures. It would probably require users to assign the inputs themselves or for the code to be hardcoded to only handle specific shader nodes and group nodes with known inputs. Uniqueness of input types is fairly easy since you can do something like <node type>_<input name> or GROUP_<group name>_<input name> for group nodes.

Some image size options would probably be useful too, to allow users to pick between the size of the Base Color image or the maximum of all found images for that material, in cases where different inputs have differently sized images or if there's only a color for the Base Color input, but full textures for other inputs.

Additionally, a default color would need to be able to be set by the user for when one material has certain inputs but another material doesn't.

[Mysteryem]

I've pushed a couple of additional commits. The first is a just small fix for objects with empty material slots.

The second commit replaces the use of a hardcoded node name MaterialCombinerOverride for use as overriding the start of node traversal with a StringProperty attached to the Material that contains the override node name. This way, if any other addon requires specifically named nodes for its functions, Material Combiner won't conflict with its functions.

As the override is now a custom StringProperty, it needs some UI to be able to set/clear it. I have added a "Search Start Override" panel to the Shader Editor with this function, with additional buttons for selecting and refocusing the view on the override node.

In addition to the UI for the override, there is a second panel that shows the material sources that are found from the current material. The magnifying glass button functions the same as the View button for the override node, focusing the view on the node in question.

When editing a Group node, all the buttons except the Clear override are disabled.

If a material isn't using nodes, the Material Sources panel will show the Material name instead of a Node and the diffuse_color property that is used in such cases. The Search Start Override panel will contain a message and button to use nodes

As shader nodes are not used by Material Combiner in Blender 2.79, neither of these panels, nor the operators they use get registered there.