Change face?

[tjasmin111]

Very interesting project, thank you. But is there a way to do something like make nose smaller using your approach?

[AdonaiVera]

Hi @tjasmin111 Tina, Thank you for your interest in the project! I think there are two approaches that you can use:

Using a Different Mask: By utilizing a mask with a smaller nose, you can effectively alter the appearance of the nose in the project. The mask acts as a template, so choosing one with the desired nose size will reflect in the final output.

Manipulating 3D Points: Post-processing is another viable approach. Once you have the 3D coordinates (X, Y, Z) of the points defining the nose, you can manipulate these points to reduce the size of the nose. This could be done by scaling the coordinates relative to a central point, ensuring a uniform reduction in size from all directions.

Both methods have their own set of advantages, and the choice between them would depend on the specific requirements of your project and the level of precision and control you need over the modification.

Feel free to reach out if you have any more questions or need further clarification!

[tjasmin111]

Thanks. Which one do you think is easier? I was thinking about the second approach. But it's not about just changing the point coordinates, but the nose texture along with it. So I have no clue how to do that.

[AdonaiVera]

Hi Tina,

The first approach is indeed the easier one. By creating a new mask, there's no need to modify the code. You simply need to adapt the mask template by making the nose smaller, and the algorithm will take care of texturizing it accordingly.

[tjasmin111]

Hmm ok. Can you please add a bit more details? Sorry not familiar with masks. Mask you mean like similar to how we augment glasses on the face? Is it like a smaller nose goes on top of a bigger original nose? Then it looks bad! Haha

[tjasmin111]

Would it be a possibility you could point me to some samples or prepare a demo of this nose mask feature? Of course doesnt have to be perfect, but just to see what it means.

[AdonaiVera]

When I refer to "mask" I am talking about the filters: https://github.com/AdonaiVera/idomai/tree/master/filters

You can extend the functionality by creating additional filters with specific parameters. Here's a step-by-step guide on how to do it:

Create a New Filter:

Design your filter and save it in the filters directory. Register the New Filter:

Update app.py to include your new filter.

type: Literal["mapa puntos", "bigote", "gafas", "sombrero", "anonymous", "monstruo", "oxigeno1", "Mascara Covid 1", "Mascara Covid 2", "Mascara Covid 3", "Mascara Covid 4", "Mascara Covid 5", "Mascara Covid 6", "Mascara Covid 7"]`

Read the New Filter:

self.filter14 = cv2.imread('filters/filter14.png', cv2.IMREAD_UNCHANGED)`

Define Filter Type:

Specify whether the filter applies to the head, eyes, mustache, or the entire face. Update the relevant section in app.py:

if self.type == "anonymous":
                        typeFilter = "completeFace"
                        filterImg = self.filter1`

Add to Dropdown Menu:

Make your filter selectable from the dropdown menu.

```python
webrtc_ctx.video_transformer.type = st.radio(
            "Selecciona el tipo de filtro que deseas aplicar", ("mapa puntos", "bigote", "gafas", "sombrero", "anonymous", "monstruo", "oxigeno1", "Mascara Covid 1", "Mascara Covid 2", "Mascara Covid 3", "Mascara Covid 4", "Mascara Covid 5", "Mascara Covid 6", "Mascara Covid 7")
        )

The other option is play with the points that you use to put the mask, think that you are matching the image (Filter or mask) in an specific part of your face, and you face has multiple points, you can change the points that you are using here:

def classPoints(self, results, image, typeFilter):
            if typeFilter == "completeFace":
                # Transform matrix, making sure the points are specified in 
                # top-left, top-right, bottom-right, and bottom-left order.
                ## Multiplying a scalar.
                topLeft = [float(results[0].landmark[54].x * image.shape[1]) - (float(results[0].landmark[54].x * image.shape[1])*0.10), float(results[0].landmark[54].y * image.shape[0]) - (float(results[0].landmark[54].y * image.shape[0]))*0.10]
                topRight = [float(results[0].landmark[284].x * image.shape[1]) + (float(results[0].landmark[284].x * image.shape[1])*0.10) , float(results[0].landmark[284].y * image.shape[0]) - (float(results[0].landmark[284].y * image.shape[0])*0.10)]
                bottomRight = [float(results[0].landmark[365].x * image.shape[1])+ (float(results[0].landmark[365].x * image.shape[1])*0.10) , float(results[0].landmark[365].y * image.shape[0]) + (float(results[0].landmark[365].y * image.shape[0])*0.10)]
                bottomLeft = [float(results[0].landmark[136].x * image.shape[1]) - (float(results[0].landmark[136].x * image.shape[1])*0.10) , float(results[0].landmark[136].y * image.shape[0]) + (float(results[0].landmark[136].y * image.shape[0])*0.10)]

            elif typeFilter == "mask":
                ## Taking Real points
                topLeft = [float(results[0].landmark[127].x * image.shape[1]), float(results[0].landmark[127].y * image.shape[0])]
                topRight = [float(results[0].landmark[356].x * image.shape[1]) , float(results[0].landmark[356].y * image.shape[0])]
                bottomRight = [float(results[0].landmark[365].x * image.shape[1]) , float(results[0].landmark[152].y * image.shape[0])]
                bottomLeft = [float(results[0].landmark[136].x * image.shape[1]) , float(results[0].landmark[152].y * image.shape[0])]

            elif typeFilter == "bigote":
                ## Taking Real points
                topLeft = [float(results[0].landmark[205].x * image.shape[1]), float(results[0].landmark[205].y * image.shape[0])]
                topRight = [float(results[0].landmark[425].x * image.shape[1]) , float(results[0].landmark[425].y * image.shape[0])]
                bottomRight = [float(results[0].landmark[436].x * image.shape[1]) , float(results[0].landmark[436].y * image.shape[0])]
                bottomLeft = [float(results[0].landmark[216].x * image.shape[1]) , float(results[0].landmark[216].y * image.shape[0])]

            elif typeFilter == "eyes":
                ## Taking Real points
                topLeft = [float(results[0].landmark[21].x * image.shape[1]), float(results[0].landmark[21].y * image.shape[0])]
                topRight = [float(results[0].landmark[251].x * image.shape[1]) , float(results[0].landmark[251].y * image.shape[0])]
                bottomRight = [float(results[0].landmark[323].x * image.shape[1]) , float(results[0].landmark[323].y * image.shape[0])]
                bottomLeft = [float(results[0].landmark[93].x * image.shape[1]) , float(results[0].landmark[93].y * image.shape[0])]

            elif typeFilter == "head":
                ## Taking Real points
                topLeft = [float(results[0].landmark[54].x * image.shape[1]) , float(results[0].landmark[54].y * image.shape[0]) - (float(results[0].landmark[54].y * image.shape[0]))*0.50]
                topRight = [float(results[0].landmark[251].x * image.shape[1]) , float(results[0].landmark[54].y * image.shape[0]) - (float(results[0].landmark[54].y * image.shape[0])*0.50)]
                bottomRight = [float(results[0].landmark[251].x * image.shape[1]) , float(results[0].landmark[251].y * image.shape[0])]
                bottomLeft = [float(results[0].landmark[54].x * image.shape[1]) , float(results[0].landmark[54].y * image.shape[0])]

            dstMat = [ topLeft, topRight, bottomRight, bottomLeft ]
            dstMat = np.array(dstMat)
            return dstMat

You can find the points in images like this:

This is a good guide: mediapipe_google

Sorry I didn't make the demo, but I'm super full of work.

[tjasmin111]

Thanks for the details. I get that how to create a new filter. But what I'm asking is a makeup filter that makes your actual nose wing appear smaller. So like in your mesh photo above, points 32-36 get closer to each other. How is it related to a new mask?

[AdonaiVera]

Hi Tina, You can select different points and reduce the size. This article will be very helpful for you. https://developers.google.com/mediapipe/solutions/vision/face_landmarker

As you can see in the image, each of the points has a number

If you want to make the nose smaller, try changing the number and using a different one. Also, I will be giving a talk about this tomorrow, but it's in Spanish. Feel free to attend if you're available.

https://forms.gle/dWmTouP61ZVR5BrL9

Let me know if this helps.

[tjasmin111]

Thanks. Yes I know the points, but I don't know what you mean of "change the number and use a different one". That means we have to transform/warp the actual face texture along with it. Which I don't know how.

Great! Unfortunately I don't know Spanish. But aren't you planning to prepare a simple demo to demonstrate this?

[tjasmin111]

Hi @AdonaiVera any updates on this? Do you have videos/samples of your talk? Again, I didn't get what you mean by changing numbers of landmark.

[AdonaiVera]

Hi @tjasmin111, I appreciate your patience.

The bounding box, defined by the points (Min X, Min Y, Max X, Max Y), is used to position and scale the mask relative to the face. Adjusting these points will indeed resize the entire mask, not just the nose. To specifically make the nose smaller, you would need to modify the filter image itself, not just the bounding box coordinates.

Here's a more detailed guide on how to achieve a smaller nose effect:

Modify the Filter Image: Navigate to the "filters" folder in the project repository. Use an image editing tool to manually adjust the nose size on your desired filter. Make sure to maintain the transparency and save it in a format that preserves it, like PNG.

Integrate the New Filter: Place your modified filter image in the "filters" directory. In app.py, register your new filter by adding it to the list of available filters. For example:

FILTER_TYPES = ["original", "new_small_nose_filter", ...]

Load your filter image similarly to how existing filters are loaded:

self.new_small_nose_filter = cv2.imread('filters/your_new_filter.png', cv2.IMREAD_UNCHANGED)

Apply the New Filter: Update the logic in app.py to apply your new filter when selected. You'll need to adjust the facial landmarks used to align the filter with the face. For a smaller nose, focus on landmarks around the nose area and scale your filter accordingly.

Adjust Landmark Points: If you still want to adjust the bounding box, you can fine-tune the landmark points used to place the mask. This involves selecting landmarks that define a smaller region around the nose.

You can refer to the MediaPipe Face Mesh for a detailed map of facial landmarks and choose the ones that best fit the smaller nose area.

Testing: After making these changes, test the application to ensure the new filter is applied correctly and the nose appears smaller as intended.

Remember, the key to making just the nose smaller is in editing the filter image itself and possibly fine-tuning the landmark points for mask placement. The bounding box method is more about the overall placement and scaling of the mask on the face.

I hope this clarifies the process for you. If you have any more questions or need further assistance, feel free to ask.

[tjasmin111]

Thanks. It would be awesome if you could add these as a sample script as well :)

[AdonaiVera]

Hi @tjasmin111, I'm currently swamped but I'll consider adding the nose alteration feature in the upcoming months; for now, it's not on the immediate roadmap.

[tjasmin111]

Ok sure! Thanks