Feature Request: Import .Mat texture alongside .Obj file for Multi-Material Painting

[wenbilliams]

Is this a new feature request? Yes

It would be incredible to be able to load a model with an albedo texture into PrusaSlicer and have the slicer automagically paint each face of the model accordingly.

This could cause issues with the conversion of full-scale colors to the finite (5) colors present in the MMU2S attachment. I propose a solution similar to how Canvas3d converts stamp images to the available colorset.

I'm happy to work on coding this feature if any experienced developers could point me at a few spots to look at. I have plenty of C++ experience. I'm currently trying to dig up the painting tool PR to get at the bits and pieces relevant to applying material to a surface.

Object/Material import reference (shapeways article): https://support.shapeways.com/hc/en-us/articles/360023735394-Full-color-3D-printing-with-3DS-Max Image import / Color filtering reference: https://www.youtube.com/watch?v=rrb-Zn0iLdQ

[wenbilliams]

@hejllukas I see you've done a bunch of PR around multi material painting. Can you point me at any spots in the code that could help me get started writing this?

[hejllukas]

It sounds to me like a cool feature that could simplify printing multi-material objects. Sorry for the later reply.

The multi-material painting consists mainly of three separate parts:

• slic3r/GUI/Gizmos/GLGizmoMmuSegmentation.{hpp, cpp} - This mainly provides the UI for painting and rendering all painted triangles through the method TriangleSelectorMmGui::render().
• libslic3r/TriangleSelector.{hpp, cpp} - Common to all painting gizmos (supports, seams, and multi-material). This provides a backend for triangle painting and triangle splitting (if it is enabled).
• libslic3r/MultiMaterialSegmentation.{hpp, cpp} - This provides segmentation of a model according to the painted triangles. This part will probably not need to be changed for your purpose.

Each model is represented as a ModelObject that contains at least one ModelVolume. Each ModelVolume keeps a TriangleMesh with a representation of that model. The ModelVolume class contains the variable _FacetsAnnotation mmu_segmentationfacets with serialized painting and splitting of triangles from a TriangleMesh. The encoding of stored data is described more in TriangleSelector::serialize(). In short, each triangle is encoded using either 4 or 8 bits (This was chosen because of backward compatibility with support and seams, which used 4 bits and supports up to 4 different states/colors). These bits encode if a triangle is split or not, and it is color (stored as EnforcerBlockerType). Currently, the number of different colors is limited to 16 (default color/extruder and 16 other colors).

In MultiMateraiSegmentation.cpp, all ModelVolume are iterated, and method _FacetsAnnotation::getfacets(), is called, which according to the passed EnforcerBlockerType will only return triangles of the selected color from the data stored in _mmu_segmentationfacets. To assign a color to a triangle, you can call _TriangleSelector::setfacet(), or _TriangleSelector::Triangle::setstate(). Both these methods must always be called on triangles that aren't split/divided. The painted trinagles from the TriangleSelector are then serialized into previously mentioned variable _ModelVolume::mmu_segmentationfacets by calling FacetsAnnotation::set().

EnforcerBlockerType can have values from 0 to 16. 0 is the default color of the model (default extruder). This means that the triangle isn't painted, and there is no need to assign this default color explicitly. Values from 1 to 16 are individual colors/extruders.

If I understand it correctly, .obj files use usemtl to assign the material to faces below this command, then at least in the beginning, there probably would be no need to parse the .mat/.mtl. Also there is also no need to further split/subdivide the triangles into finer ones for this feature. So you wouldn't need to manipulate the triangle splitting/subdivision in TriangleSelector in any way. Nevertheless, for completeness, I will describe it below.

I think you will need to look at libslic3r/Format/Obj.{hpp, cpp} and libslic3r/Format/objparser.{hpp, cpp} where the .obj files are parsed. I saw that there is some support for usemtl, but the read data isn't processed anymore. So I guess you will need to extend _loadobj somehow and figure out how to store the materials.

I think the methods _TriangleSelector::seed_fill_selecttriangles() and _TriangleSelector::seed_fill_apply_ontriangles() could be used as inspiration and to get familiar with TriangleSelector. _TriangleSelector::seed_fill_selecttriangles() is called with starting facet, from its go through all adjacent facets (based on the angle between them) using a queue and flagged that were selected by seed fill (in UI is called Smart fill). Then, in _TriangleSelector::seed_fill_apply_ontriangles(), all go through all triangles and assign the selected color/EnforcerBlockerType to all flagged triangles.

When TriangleSelector is created, it copies vertices to _mvertices and triangles to _mtriangles. After initialization, all triangles are always unsplit/undivided, and _mvertices and _mtriangles have the same data as they were in TriangleMesh. New vertices and triangles are created only when a user paints with brushes with triangle splitting enabled (_TriangleSelector::selectpatch()).

Triangle splitting I don't think this will be needed to implement this feature, but I will still quickly describe how the triangle splitting works. When a user paints with brushes (in the code called Cursors), it is called the method _TriangleSelector::selectpatch(). That assigns a color to a triangle if it is entirely inside a brush. If the triangle is only partially inside a brush, it can be split (unless a resulting edge will be smaller than a given limit determined from the brush diameter). Splitting of a triangle is done through calling methods _TriangleSelector::splittriangle() and _TriangleSelector::performsplit(). Each triangle is always split into two, three, or four triangles. Triangles are split recursively, and their edges are always split in the middle. New vertices created by edge splitting are shared between adjacent triangles of the corresponding edge, so there should be no duplicate vertices in _mvertices.

One drawback to the above-described method of triangle splitting is that it results in painted areas on the brush border always being jagged. This causes issues on flat top and bottom layers, where larger holes in objects can appear at the border of two different colors. We invested some time to find a better solution, but unfortunately, the right solution to this problem isn't quite easy to do it right and stable.

I hope I've mentioned everything that might be useful about the multi-material painting/segmentation background and that it helps you.

I personally think that the most work will be with creating some kind of dialog or gizmo to assign colors/extruders to parsed materials and some user-friendly integration into the slicer.

[wenbilliams]

@hejllukas Thanks so much for the in-depth description. This really helps me get started!

[wenbilliams]

@hejllukas Where can I find sample code to read in .jpeg and .png files in this repo? I have parsed the mtl, and I'm trying to use Boost::GIL but struggling. If this repo has OpenCV I know how to use that to easily parse any image file.

[wenbilliams]

And one last question - how do I publish a branch of this? Do I need to get a permission for this repo, or do I need to fork and then pull request from the other?

[hejllukas]

@hejllukas Where can I find sample code to read in .jpeg and .png files in this repo? I have parsed the mtl, and I'm trying to use Boost::GIL but struggling. If this repo has OpenCV I know how to use that to easily parse any image file.

We don't use OpenCV. We load .png using wxImage::LoadFile from the WxWidgets library. It should support .jpeg/jpg as well. Parts of the GLTexture::load_from_png() function could serve as an example of how to load a .png file using wxImage::LoadFile and get its RGBA representation. But wxImage::LoadFile is also used in other places, or more information about wxImage::LoadFile is in the documentation for WxWidgets.

And one last question - how do I publish a branch of this? Do I need to get a permission for this repo, or do I need to fork and then pull request from the other?

Yeah, you'll have to make a fork and then make a pull request from it (Described, for example, there: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request)

[PaulKC]

That sounds really interesting. Have you found the time to implement something along these lines?

[bubnikv]

@wenbilliams Multi-material segmentation is on our radar, however the issue we are facing is that we want to combine various sources of color segmentation (multiple volumes, modifiers, upcoming text) with the texturing, which is highly non trivial and we did not decide yet how to proceed.

[TylonHH]

Hi I have an complex model (architecture) which is painted and exported to 3MF or 3DS. It’s one STL with five colours. How can I print this multimaterial? Painting again within prusa slicer is no option. Could this FR help? Cheers

[nhnifong]

@hejllukas You mentioned that you do not believe triangle splitting would be necessary relevant to this feature. maybe because @wenbilliams only intended to "paint each face"? in any case, textures would usually have a much higher resolution than the triangle mesh, so if triangle splitting were not used, some complex alternative would be required in order to not lose much of the texture detail. And that alternative would have to be understood by many downstream modules.

In response to a similar issue in BambuStudio, I've been researching how a TriangleSelector method might be created that samples a UV mapped texture in order to fill regions based on that texture. I assumed that triangle splitting would an unavoidable piece of infrastructure in that kind of solution.

Is triangle splitting basically here to stay? or is an entirely new method of storing color information in the works?

https://github.com/bambulab/BambuStudio/issues/1892