Opening its own photos

[flavienbwk]

Hi,

When will the photo opening be implemented in StyleFlow?

Will we have to adjust the initial parameters (such as yaw for example) or will it be automatically detected?

Thanks!

[RameenAbdal]

The framework depends on Microsoft API for the annotations. The lighting parameters can be derived from here. After getting the projections of the images, These files can be replaced.

[flavienbwk]

@RameenAbdal thanks, will you consider adding a tutorial for this ?

[RameenAbdal]

Will try to include real image projection and setup in the next update.

[flavienbwk]

This issue should be re-opened and closed only when the feature is available 👍

[flavienbwk]

Thanks @RameenAbdal !

[maloyan]

Has anyone succeeded in editing their own images?

[PaulCzaban]

I was hoping to hijack the current image to be edited and try to find a similar sg2latents image that matches lighting, attributes etc. Unfortunately its not that simple as images are encoded via stylegan-encoder.

I don't have days to burn on this so hopefully the real image projection etc will be as simple as we all hope.

[iPsych]

here

So if I want to upload and 'modify' the image from elsewhere, the image should processed via Microsoft API and DRP for lighting?

[PaulCzaban]

here

So if I want to upload and 'modify' the image from elsewhere, the image should processed via Microsoft API and DRP for lighting?

Yes.

I believe you'll need to change 4 files. TSNE.npy (These look like the circles you click on to select an index. Maybe just keep the circle for the first index) light.npy (https://github.com/zhhoper/DPR) sg2latents.pickle (https://github.com/spiorf/stylegan-encoder) attributes.npy (Microsoft Face Detection)

The outputs from those repos will need further formatting to match the files above. Some backward engineering is needed to get it just right I think.

[iPsych]

Thanks. If I understood correctly, 1) I have to change 4 files to process images from elsewhere, right? 2) Or process images using Microsoft API and DPR(lighting), and change codes for 'load' it? Is there any plan for support custom image processing in near future?

[PaulCzaban]

Quick and dirty code to help you get started. I've excluded the lighting bit, but its detailed above if you want to include it (its optional).

Step 1 - Align your image Use Align_face_image in utils.py. This will change your input image and align the detected face. It's best to crop the photo manually to avoid GPU memory issues.

Step 2 - Project your aligned image Project your custom image via https://github.com/rolux/stylegan2encoder. I used the cmd "python project_images.py aligned_images/ generated_images/" after dropping my aligned image into aligned_images. This will product a npy file and a preview of the achieved image.

Step 3 - Get the attributes Go to https://azure.microsoft.com/en-us/services/cognitive-services/face/ upload your face and make a note of the values using detection model 1: 0-Gender (girl is 0) 1-Glasses (no glasses is 0) 2-Yaw 3-Pitch 4-Baldness 5-Beard 6-Age 7-Expresion (Smile I think)

Step 4 - Insert into StyleFlow This is up to you. However, I hijacked the existing code to bypass the requirement for lighting file.

I changed the sample indexes to the first 100. self.keep_indexes = [i for i in range(0,100)]

I then overwrote the first 3 latent with my custom images (3 as an example):

self.raw_w = pickle.load(open("data/sg2latents.pickle", "rb"))
self.raw_w['Latent'][0][0] = np.load('data/DW.npy')
self.raw_w['Latent'][1][0] = np.load('data/JS.npy')
self.raw_w['Latent'][2][0] = np.load('data/RC.npy')

I then overwrote the first 3 attributes with my custom data:

        self.all_attr[0][0][0] = 0      #Gender
        self.all_attr[0][1][0] = 0      #Glasses
        self.all_attr[0][2][0] = -1.5   #Yaw
        self.all_attr[0][3][0] = -9.2   #Pitch
        self.all_attr[0][4][0] = 0.02   #Baldness
        self.all_attr[0][5][0] = 0      #Beard
        self.all_attr[0][6][0] = 25     #Age
        self.all_attr[0][7][0] = 3      #Expresion

        #1
        self.all_attr[1][0][0] = 0      #Gender
        self.all_attr[1][1][0] = 0      #Glasses
        self.all_attr[1][2][0] = -9.9   #Yaw
        self.all_attr[1][3][0] = -0.2   #Pitch
        self.all_attr[1][4][0] = 0.24   #Baldness
        self.all_attr[1][5][0] = 0      #Beard
        self.all_attr[1][6][0] = 30     #Age
        self.all_attr[1][7][0] = 1      #Expresion

        #3
        self.all_attr[2][0][0] = 0      #Gender
        self.all_attr[2][1][0] = 0      #Glasses
        self.all_attr[2][2][0] = -5.1   #Yaw
        self.all_attr[2][3][0] = -5.1   #Pitch
        self.all_attr[2][4][0] = 0.5    #Baldness
        self.all_attr[2][5][0] = 0      #Beard
        self.all_attr[2][6][0] = 29     #Age
        self.all_attr[2][7][0] = 1      #Expresion

I also modified the blue circles to help find the first 3 indexes:

count = 0
        for point in self.X_samples:
            ######### don't use np.uint8 in tuple((point*1024).astype(int))
            if count < 3:
                cv2.circle(self.map, tuple((point * 1024).astype(int)), 7, (0, 0, 100), -1)
            else:
                cv2.circle(self.map, tuple((point * 1024).astype(int)), 6, (0, 0, 255), -1)
            count = count + 1

[provostm]

Hello @PaulCzaban, I think I manage to do what you said. I can open my pictures but when I move the cursors the face does not get modify by a lot. For exemple gender does not switch to female for a male and vice-versa. I see small modifications of the face but not a gender switch. Do you know what I can do to improve the results or maybe what I missed?

[PaulCzaban]

Hello @PaulCzaban, I think I manage to do what you said. I can open my pictures but when I move the cursors the face does not get modify by a lot. For exemple gender does not switch to female for a male and vice-versa. I see small modifications of the face but not a gender switch. Do you know what I can do to improve the results or maybe what I missed?

I also observed this behaviour. Some pictures work better than others so do try another picture. I suggest you also try increasing the max/min boundaries of the attribute sliders. I posted an issue regarding this below.

https://github.com/RameenAbdal/StyleFlow/issues/37#issue-787534611

[provostm]

Thanks I edited the min and max and retry. I will try with another picture. Do you think if I increase the number of iterations in step 2 it could be better? I tried 2000 but the execution crashed (not enough memory in my GPU, it's only a GTX 1050.

[PaulCzaban]

The best bet is to try a selection of photos, go for passport worthy, decent definition photos.

Its my belief the model just doesn't respond perfectly well to all faces... Maybe they sit too far away from the training data.

It's also possible that the alignment and projection process didn't run perfectly. If the features of the face aren't found then it might result in this unresponsive behaviour.

Either way I don't think there is an easy fix, other than trying a different photo.

[OdinGitDat]

It seems the training set from this repo suffers greatly from bias making it perform terrible on any custom portrait I've tried.

[RameenAbdal]

For projected faces, since there are many methods to project the images (see https://arxiv.org/abs/2012.09036 ) with variable quality and editing capabilities, it becomes difficult to handle all these frameworks. Currently, https://github.com/eladrich/pixel2style2pixel also does a good job striking a balance between this trade-off. Also, we add a discussion regarding the editability in the final version that will be online soon. After projecting the images, also make sure to modify (increase for the real images) the degree (here) and ranges (here) for the edits.

[OdinGitDat]

Are you suggesting that the transformation results are not as expected due to bad image projection? If this is the case do you have any settings regarding the projection used throughout the project that led to the best results?

Whenever I run the UI and play with the sliders, the results are nowhere near as striking as shown in the demos. The switch in gender or glasses is not convincing (even after adjusting the slider ranges and degree). This makes it look like the model doesn't work as well on any unseen data.

[VictorBulba]

Has anyone achieved good results when projecting custom images? I used encoders suggested by @PaulCzaban and results are bad. But I noticed that if I set --num-steps to a small value (https://github.com/rolux/stylegan2encoder) it produces another face and this face works great. I can transform gender and beard and other, but if I run more steps projection will fit the given image and stop working.

[zhanghongyong123456]

For projected faces, since there are many methods to project the images (see https://arxiv.org/abs/2012.09036 ) with variable quality and editing capabilities, it becomes difficult to handle all these frameworks. Currently, https://github.com/eladrich/pixel2style2pixel also does a good job striking a balance between this trade-off. Also, we add a discussion regarding the editability in the final version that will be online soon. After projecting the images, also make sure to modify (increase for the real images) the degree (here) and ranges (here) for the edits. .I would like to know if you plan to add attribute editing of custom images. I edited the actual images I took by myself and processed them, and the photos I got seemed to be made up（cosmetics），Not real . I can't find the reason。

[tali055]

Quick and dirty code to help you get started. I've excluded the lighting bit, but its detailed above if you want to include it (its optional).

Step 1 - Align your image Use Align_face_image in utils.py. This will change your input image and align the detected face. It's best to crop the photo manually to avoid GPU memory issues.

Step 2 - Project your aligned image Project your custom image via https://github.com/rolux/stylegan2encoder. I used the cmd "python project_images.py aligned_images/ generated_images/" after dropping my aligned image into aligned_images. This will product a npy file and a preview of the achieved image.

Step 3 - Get the attributes Go to https://azure.microsoft.com/en-us/services/cognitive-services/face/ upload your face and make a note of the values using detection model 1: 0-Gender (girl is 0) 1-Glasses (no glasses is 0) 2-Yaw 3-Pitch 4-Baldness 5-Beard 6-Age 7-Expresion (Smile I think)

Step 4 - Insert into StyleFlow This is up to you. However, I hijacked the existing code to bypass the requirement for lighting file.

I changed the sample indexes to the first 100. self.keep_indexes = [i for i in range(0,100)]

I then overwrote the first 3 latent with my custom images (3 as an example):

self.raw_w = pickle.load(open("data/sg2latents.pickle", "rb"))
self.raw_w['Latent'][0][0] = np.load('data/DW.npy')
self.raw_w['Latent'][1][0] = np.load('data/JS.npy')
self.raw_w['Latent'][2][0] = np.load('data/RC.npy')

I then overwrote the first 3 attributes with my custom data:

        self.all_attr[0][0][0] = 0      #Gender
        self.all_attr[0][1][0] = 0      #Glasses
        self.all_attr[0][2][0] = -1.5   #Yaw
        self.all_attr[0][3][0] = -9.2   #Pitch
        self.all_attr[0][4][0] = 0.02   #Baldness
        self.all_attr[0][5][0] = 0      #Beard
        self.all_attr[0][6][0] = 25     #Age
        self.all_attr[0][7][0] = 3      #Expresion

        #1
        self.all_attr[1][0][0] = 0      #Gender
        self.all_attr[1][1][0] = 0      #Glasses
        self.all_attr[1][2][0] = -9.9   #Yaw
        self.all_attr[1][3][0] = -0.2   #Pitch
        self.all_attr[1][4][0] = 0.24   #Baldness
        self.all_attr[1][5][0] = 0      #Beard
        self.all_attr[1][6][0] = 30     #Age
        self.all_attr[1][7][0] = 1      #Expresion

        #3
        self.all_attr[2][0][0] = 0      #Gender
        self.all_attr[2][1][0] = 0      #Glasses
        self.all_attr[2][2][0] = -5.1   #Yaw
        self.all_attr[2][3][0] = -5.1   #Pitch
        self.all_attr[2][4][0] = 0.5    #Baldness
        self.all_attr[2][5][0] = 0      #Beard
        self.all_attr[2][6][0] = 29     #Age
        self.all_attr[2][7][0] = 1      #Expresion

I also modified the blue circles to help find the first 3 indexes:

count = 0
       for point in self.X_samples:
           ######### don't use np.uint8 in tuple((point*1024).astype(int))
           if count < 3:
               cv2.circle(self.map, tuple((point * 1024).astype(int)), 7, (0, 0, 100), -1)
           else:
               cv2.circle(self.map, tuple((point * 1024).astype(int)), 6, (0, 0, 255), -1)
           count = count + 1

When trying to align an image I get a Name Error: name 'image' is not defined. Any ideas? I'm calling the function from powershell using python -c 'import utils; utils.Align_face_image(image.jpg)'

[bauerdavid]

Quick and dirty code to help you get started. I've excluded the lighting bit, but its detailed above if you want to include it (its optional).

Step 1 - Align your image Use Align_face_image in utils.py. This will change your input image and align the detected face. It's best to crop the photo manually to avoid GPU memory issues.

Step 2 - Project your aligned image Project your custom image via https://github.com/rolux/stylegan2encoder. I used the cmd "python project_images.py aligned_images/ generated_images/" after dropping my aligned image into aligned_images. This will product a npy file and a preview of the achieved image.

Step 3 - Get the attributes Go to https://azure.microsoft.com/en-us/services/cognitive-services/face/ upload your face and make a note of the values using detection model 1: 0-Gender (girl is 0) 1-Glasses (no glasses is 0) 2-Yaw 3-Pitch 4-Baldness 5-Beard 6-Age 7-Expresion (Smile I think)

Step 4 - Insert into StyleFlow This is up to you. However, I hijacked the existing code to bypass the requirement for lighting file.

I changed the sample indexes to the first 100. self.keep_indexes = [i for i in range(0,100)]

I then overwrote the first 3 latent with my custom images (3 as an example):

self.raw_w = pickle.load(open("data/sg2latents.pickle", "rb"))
self.raw_w['Latent'][0][0] = np.load('data/DW.npy')
self.raw_w['Latent'][1][0] = np.load('data/JS.npy')
self.raw_w['Latent'][2][0] = np.load('data/RC.npy')

I then overwrote the first 3 attributes with my custom data:

        self.all_attr[0][0][0] = 0      #Gender
        self.all_attr[0][1][0] = 0      #Glasses
        self.all_attr[0][2][0] = -1.5   #Yaw
        self.all_attr[0][3][0] = -9.2   #Pitch
        self.all_attr[0][4][0] = 0.02   #Baldness
        self.all_attr[0][5][0] = 0      #Beard
        self.all_attr[0][6][0] = 25     #Age
        self.all_attr[0][7][0] = 3      #Expresion

        #1
        self.all_attr[1][0][0] = 0      #Gender
        self.all_attr[1][1][0] = 0      #Glasses
        self.all_attr[1][2][0] = -9.9   #Yaw
        self.all_attr[1][3][0] = -0.2   #Pitch
        self.all_attr[1][4][0] = 0.24   #Baldness
        self.all_attr[1][5][0] = 0      #Beard
        self.all_attr[1][6][0] = 30     #Age
        self.all_attr[1][7][0] = 1      #Expresion

        #3
        self.all_attr[2][0][0] = 0      #Gender
        self.all_attr[2][1][0] = 0      #Glasses
        self.all_attr[2][2][0] = -5.1   #Yaw
        self.all_attr[2][3][0] = -5.1   #Pitch
        self.all_attr[2][4][0] = 0.5    #Baldness
        self.all_attr[2][5][0] = 0      #Beard
        self.all_attr[2][6][0] = 29     #Age
        self.all_attr[2][7][0] = 1      #Expresion

I also modified the blue circles to help find the first 3 indexes:

count = 0
       for point in self.X_samples:
           ######### don't use np.uint8 in tuple((point*1024).astype(int))
           if count < 3:
               cv2.circle(self.map, tuple((point * 1024).astype(int)), 7, (0, 0, 100), -1)
           else:
               cv2.circle(self.map, tuple((point * 1024).astype(int)), 6, (0, 0, 255), -1)
           count = count + 1

I cannot proceed with Step 3 (getting face attributes from Azure) as there's no option to run face detection demo online (I'm wondering if I'm alone with this problem). Any ideas how can I resolve this? I guess another option would be to use the API from code, but then I would have to write unreasonably high amount of code just to run this simple step.

[csyxwei]

Has anyone achieved good results when projecting custom images? I used encoders suggested by @PaulCzaban and results are bad. But I noticed that if I set --num-steps to a small value (https://github.com/rolux/stylegan2encoder) it produces another face and this face works great. I can transform gender and beard and other, but if I run more steps projection will fit the given image and stop working.

You can try to use https://github.com/omertov/encoder4editing to obtain the inverted latent code instead of https://github.com/rolux/stylegan2encoder. In my experiment, e4e achieves better editing results.

[chobao]

Hi, does anyone successfully use microsoft azure api now? The microsoft seems to limit the usage of face api unitl a form is applied for access