Project Grease Pencil into mesh

[Dereuz24]

Hi there!

I was wondering if maybe you have thought on a way to draw straight into mesh from a camera projection without masking by transforming a mesh into grease pencil?

This kind of stuff would allow the built interpolation or extrapolation functionality to follow a mesh across the timeline in a more convenient manner. Currently the solutions are a bit cumbersome having to convert the mesh to GP before masking and using the placing stroke on surface afterward. Basically everything is already built into blender but there's a lot of steps and adjustments in between that could otherwise be very straightforward

If this could be added as a more user friendly function it would make things a lot more easier and satisfying to work with

EDIT: A script like solution I thought about for an animated object is something along the lines of

1. User draws stroke
2. Click on button that basically runs: Clean-up project into surface from camera view
3. User draws next keyframe
4. Click on project into surface
5. Then click on complete and this runs a series of steps like:
6. Going to Frame in the middle of these two strokes.
7. Interpolate with selected interpolation method
8. Clean-up project into surface
9. Repeat until desired number of frames is completed

As for the masking automatation, maybe the selected mesh is selected before the last operation and converted into a clean closed gpen sequence with a solid color (ignoring all materials), preserving the original mesh and automatically placed in a hidden collection, and this new Gpen sequence is placed in the masks menu.

Something along those lines more or less.

[chsh2]

Hi. This seems to be a workflow that I am not quite familiar with, since this add-on mainly focuses on drawing on a plane. I occasionally draw on the surface of meshes, but I do not know what the masking refers to in your description. Are there any tutorials or demos showing how the masking works?

The procedure you mentioned also reminds me of some Blender functions, but I am not sure if they are related:

• There is an option Stroke Placement to snap newly drawn Grease Pencil strokes to the mesh surface.
• There is an operator Bake Mesh to Grease Pencil that works for multiple frames.

[Dereuz24]

Hi there. Sorry for the delay

Yes absolutely. So I just meant a typical clipping mask to paint on an area of a defined layer like in 2D drawing software but with the grease pencil:

• https://www.youtube.com/watch?v=cw5C7AjvB1U

That's a good example of it. He's just painting a shadow only in the area defined as skin.

And absolutely! What I mentioned I'm currently doing it with some macros and scripts because blender does has those operators individually.

So the idea with this is that because NPR work in Blender is so limited, even with custom render engines like malt, sometimes the only solution you have at your disposal is to hand drawn the shadows to not have to deal with problematic lighting, normals or meshes. Sometimes is just faster to use a broken poorly planned topology and just paint or correct shadows on top of it.

I've done that plenty on 2D drawing software but blender does offer decent interpolation for the grease pencil so the idea in general is to use the stroke placement on surface from camera position, and use the selected mesh itself as the mask like in that 2D example.

The stroke placement works very well. Let's say you have a character on frame, and you want to draw an slit on their cheek, you can do it with this method, project the grease pencil onto the surface, then jump to the last frame of the shot, draw the second frame, and then slowly interpolate in-betweens and fix.

The complex part is when you want to start painting shadows. Because they have to be contained by the character/object.

It's been a while since I've used the vanilla version of "bake mesh to grease pencil" because back then it left a lot to be desired, edges where split, sometimes weird edges got converted to grease pencil strokes themselves, so a lot had to be cleaned up.

That's why now I use addons like Sketchn'carve because it offers a lot of control and refinement in the baking process.

Once the mesh is converted to grease pencil I can just select that as a mask and paint my shadows freely. Then use the same method, go to the last frame, draw again, then go to the middle, interpolate, then go to cleanup "Reproject strokes" and getting them to snap from camera view into the surface. This does better than snapping sometimes with each vertex sticking to the surface

After these 3 frames are secured then routinely I just run the same interpolation in the in-betweens with reprojection until I have all frames.

There's a lot of steps and a lot of poor functionality. Meshes converted to grease pencil tend to be very heavy so if we were to have operators to simply define the mesh, do our painting at the beginning, end frame, then fix the middle if the user choses, and after that just run the whole thing in 1's or 2's or 3's and whatever interval we pick, then ditching the grease pencil layer used for masking after all is completed, there could be a more usable way to do all of this like it's 2D software. I was about to just try and put together that stuff into an addon with my limited knowledge but given your addon is very polished for a lot of the operations I thought that maybe you have good ideas for cleaner more efficient mesh conversion and how to make this stuff into an usable menu without all the clunk

I don't know if it all makes sense

EDIT: Also, I really don't know how to use those masks and then automatically delete everything drawn outside of the masked area from camera view. If I was to do all that stuff automatically then there needs to be a procedure to clean it all before deleting the grease pencil conversion. So little details like that keep me just using a dumb step by step method

[chsh2]

To generate a better mask, these tools of this add-on may be related:

1. I am developing a new function to convert an object to Grease pencil strokes (https://github.com/chsh2/nijiGPen/discussions/36). Compared with the official one, this one may have better meshes and do not split edges. However, it requires rendering and therefore much slower than the official one.
2. Performing a Boolean union on the results of Bake Mesh to Grease Pencil may more or less clean up the topology.

Talking about cleaning, would it be OK to hide the mask (e.g. by setting the layer opacity to zero) instead of deleting the mask? In my opinion, it is not an easy task to automatically delete stroke points outside the mask. Even if we can achieve it, the points near the boundary may not look perfect.

[Dereuz24]

Hey that's a lot more sophisticated. The output in your example with Suzanne looks very clean so that would be even better

Are you working with a fixed parameter for the toon shader or are you thinking the user could pick the posterization value with a slider/number? If that's possible and the users just uses a single value that would give a clean flat mask with the minimum amount of points already, if not and you are doing a fixed shader then with a checkbox to output a flat color GP object could be enough. However if there's a way to pick the amount of colors that could be very powerful for custom effects too

Originally I didn't find a clean way to erase GP points from a masked area, but all of that was only necessary because when I tried working with the typical mesh to GP conversions you end up with GP objects that are so dense that it becomes unpractical to use in long sequences with many objects in the scene. And once you are done then you have a lot of GP objects that are only useful in a context and can't be really unloaded to improve performance. If Blender was more like Houdini this wouldn't be an issue but with the amount of strokes this method created it really wasn't practical

I took a look at what that studio did a few months ago and tried to replicate it but also found that it would be too complex to try to separate the shading so I totally forgot about it after that

And now that I think about it, the method I mentioned of interpolation and masking is only a way to brute force a flat result from an 3d object. But with your conversion tool you are already working on 2D space so any user can just draw and interpolate normally as long as the masks are generated for a specific object and a selected view.

If you eventually get to work on finishing this script and for it to update on each camera frame that would be more than enough even with a black silhouette. All the other steps I mentioned before are unnecessary with this tool

[chsh2]

In the new add-on version 0.10, I tentatively added this function in the sub-menu of [Import/Export] - [Flat Color]. If the number of colors is set to 1 or 2, the output should be a mask.

[Dereuz24]

It works perfectly. Thank you so much. The toon shading is great to get half of the way there too

For a mask, are settings like sample length and min stroke area affecting performance as well? When it renders a single frame is almost an instant conversion without the animation rendering, just going frame by frame. Is this because the operator works with the output and not the viewport?

[Dereuz24]

Also I'm trying to use other render engines and depending on the configuration it fails because I imagine it can't find the rendered output. The workaround I have is to render the mesh in a .PNG sequence, reimport it as a plane set on emission and then run the conversion from camera view after fitting the sequence in the camera area

Would it be possible to have the option to import an image sequence straight into conversion instead? That way you wouldn't have to account for every single possibility and avoid having to do all those steps. For it to work the user would have to set up the desired start and ending frames that way it matches the viewport

Given the nature of the grease pencil having its own engine this would avoid a lot of unexpected issues

Edit: I still can't believe how well this works. I've been testing it out and you did absolute magic, this is the greatest!! Thank you so much again for sharing this

[chsh2]

For a mask, are settings like sample length and min stroke area affecting performance as well?

These options only have slight performance impact on a mask.

As for the rendering engine, rendering is required whether it is a single frame or multiple frames. I have done some basic tests with Workbench and Cycles to verify that it works, but have not tested them thoroughly. For unofficial render engines, I cannot guarantee that they are supported. Which engine did you use and what error messages did you get?

Would it be possible to have the option to import an image sequence straight into conversion instead?

Yes. You can just use the normal [Flat Color] button (without the submenu) to select multiple image files, and enable the option Image Sequence and Fit to Camera.

[Dereuz24]

I tried using Malt. I thought it wouldn't initialize because it uses its own render nodes for the image output

The image sequence works perfectly!