added image sort tool to faceswap

[iperov]

I added image sort tool to faceswap, which very useful to extract one face from various faces

Example original aligned folder:

Sort it by similarity: python.exe faceswap\sorttool.py -i %WORKSPACE%\data_src\aligned -by similarity result:

easy delete faces which you dont need:

Sort by blur: python.exe faceswap\sorttool.py -i %WORKSPACE%\data_src\aligned -by blur

most sharp 00000.png:

most blurred 00140.png:

[Kirin-kun]

If I understood correctly, it sorts them, then orders them by renaming them?

[iperov]

yes

[Kirin-kun]

Ideally, a tool for making a face set would make a folder for each "type" of face and keep a single one in the original directory. Up to you afterward to move some interesting faces back to the training folder if you wish to use them.

When extracting a video, event at 1fps, you will end up with a few hundreds frames with some of them nearly identical, whereas the training process would ultimately not need them and just waste time learning them "again".

My B face folder is neatly prepared, with different poses and lighting. But the A folder is very tedious to prune each time I want to use a different target video.

But thanks for the work. It's a step in the right direction to make it easier to have good face sets.

[ghost]

Works great, really does, thought it was broken it was so fast. -by blur, chuck out the trash, then -by similarity to get rid of obvious dupes. Epic time saver.

[deepfakesclub]

Kind of like your lossA/lossB balancing, sorting and getting rid of very similar faces probably saves time so your model isn't training the same angle to perfection, while other poses are only sampled rarely.

Basically, the ultimate version would be to bin faces by similarity (which takes into account the pose primarily, I think), calculate the loss on each subset, and then optimize by training on the worst subsets.

A simple example would be you notice that your side profiles are terrible while your full front face is very good. Out of 10,000 images, you probably have 5 side profiles and 5000 full front faces. Those side profiles are sampled rarely, so you would prioritize training the side profiles after the front view is good.

Some of us already do this at an unsophisticated level manually with data curation, but it could be done more systematically and automatically.

The pose bins would be based on the target replacement video. i.e. if there are no side poses in video A, there's no point in optimizing it.

[iperov]

@deepfakesclub cannot understand what you mean. So you agree or disagree with optional functions loss_balancing and sorting faces ?

[ghost]

@iperov can I ask for feature request lol? -by cull, its sort of like sort -by blur, but deletes all pics that fall below a set threshold for blur, its hard to make decision of whats blurry or not when dealing with multiple sets. Id rather know Ive culled to the same standard of pics mathematically, from each different set rather than what my mind thinks is blurry.

[iperov]

i dont need this feature for myself, but you can implement it and create new PR after my PR will be merged

[ghost]

Ill try lol, I have no idea what Im doing, I need to maybe get the blur score and do something with it hehe.

[kcimit]

hi iperov, would be nice to have a possibility to remove or move all visually similar images, including blurred ones without renaming the rest.

[deepfakesclub]

@iperov Your loss balancing and sorting are both good.

Sorry, I was just speculating on a different kind of extreme sorting. Your loss balancing got me thinking about what other optimizations are possible.

[AbysmalBiscuit]

Hi,

I've extended the sorttool to now be able to sort images into folders, both by blur and similarity. It uses the same similarity and blur sorting approaches as the the current version. I have also added the option of specifying an output directory that defaults to the input directory if none is given (which preserves the original functionality of the renaming approach).

The folder grouping by similarity takes an optional float 'minimum score threshold' parameter, which acts as the minimum similarity score for an image to be considered part of the group (right now the default is 0.5, but I haven't tested it extensively).

The folder grouping by blur takes an optional integer 'bins' to sort the images based on how blurry they are into bins. If the number of images doesn't fit equally among the bins, the exess get added to the last folder (which is the bluriest).

@kcimit and @i5L4NDOF5T4BiLiTY when sorted into folders you could just delete the unecessary folder(s).

Here is a gist : https://gist.github.com/AbysmalBiscuit/03f34da39a6cf96d84312d9d309d9c8d

If it's good enough, can I receive a pull request? (note: the code styling is for PEP8 and not the repo styling, I will reformat it to match the repo style)

[iperov]

@AbysmalBiscuit already told you , your version is outdated.

[AbysmalBiscuit]

@iperov my bad, didn't see.

[iperov]

also cannot understand why need work with additional folders

[AbysmalBiscuit]

The main advantages as I see it are: 1) It's more convenient in terms of having clear separations between how the images are grouped together. Especially when working with very large datasets like when you extract faces from several minutes of video.

2) It would also allow you to preserve the original filenames which would make it possible to use them for the aligned folder when converting.

3) It would also make it easier to delete the images that you don't want (since you can look inside the folder and see right away if it's not the face/angle/bluriness you want), which was something people in the thread were asking for.

[Kirin-kun]

The usual process, at least for me, is usually the following:

1 - Have neat B facesets ready. Sorted, various poses, etc... 2 - Extract all frames from target video 3 - Extract all the faces from all frames, in order to have an aligments.json ready 4 - Prune from the extracted faces the "false positives" (wasn't a face, wasn't the desired face, was a too small face...) 5 - Start training with all the remaining faces in A, because sorting them and creating manually a good training set is a PITA. 6 - Once the loss and preview look ok, merge a small framerange to see how it looks 7 - ???? 8 - Profit

Now, point 5 is what @iperov attempted to address with his tool, which is a huge improvement, because the trainer wastes a lot of time learning very similar faces. I think that's one of the problems the IAE model has too. The GAN model seems to be affected also.

Except that, by renaming the files, it defeats the functionality that allowed to not merge the faces deleted from the aligned directory, but present in the alignments file.

So, putting similar faces in different folders would allow to quickly build a set for training by choosing various poses/faces, while allowing to build also an aligned directory ready for conversion (just have to prune the false positives and wrong faces).

[iperov]

Except that, by renaming the files, it defeats the functionality that allowed to not merge the faces deleted from the aligned directory, but present in the alignments file.

I dont use sorttool for dest data. Only for source data, where alignments absolutely unused.

[AbysmalBiscuit]

My work process is more or less the same as @Kirin-kun.

@iperov I think there has been a misundurstanding about what each person needs the image sorting for. What I need it for is sorting the extracted 'faces', be it for training data or conversion data. If I'm not mistaken the people in this thread who want to be able to sort the faces into folders are in the same use case as I am.

Here's a more detailed example to demonstrate my use case: Let's say I extract 2000 'faces' (this includes false positives) from a 5 min video, the renaming approach wouldn't help me much since I'd still have to scroll through everything looking for when the groups change/unwanted images appear. However, if you sort the images into folders, and if the threshold used to determine similarity is discriminating enough, you will have folders that you can delete (if it's a folder of non-faces), or just pick a few images out of to get the lighting conditions/face angle/other criteria that you're looking for.

@Kirin-kun I have updated my gist to have the latest functionality from the repo, with the addition of my folder sorting: https://gist.github.com/AbysmalBiscuit/03f34da39a6cf96d84312d9d309d9c8d Could you try downloading and testing it to see how it works for you? The 'face' method also uses the threshold parameter. With the face method, folder 0 is always the non-face folder. Also it may not display a progress bar for everything if you're grouping into folders. Also backup your data before you test it just in case. :p

[Kirin-kun]

@iperov why not both?

[Kirin-kun]

@AbysmalBiscuit just did a quick test with simple args:

python sort.py -i h:\Fakes\data_A\aligned -o h:\Fakes\data_A\aligned\sorted -g folders -by face

And I just have a couple remarks:

• Why delete from input dir? It defeats the purpose of later use of the aligned dir for merging, because you'd have to move back everything from the subfolders. The space saved by not duplicating isn't worth the trouble (disks are cheap!). I'd just need to remove unwanted faces and false positives. Maybe it could be optional.

• With just "-by face", I had in the same folder faces facing different directions (left, right, facing the camera) and sometimes I had a hard time differentiating why a face had landed in a different folder when it looks about the same :). I guess the problem is the threshold, because it's the face recognition library at work.

So, maybe an explanation of what values the thresholds can be for hist and blur respectively in the usage message.

[AbysmalBiscuit]

In general I've refactored the sort to match the code style and to be arranged symmetrically in terms of methods, with the original methods followed by are my folder methods.

@Kirin-kun It doesn't actually delete from the input dir, it moves the images to the grouped dirs. This is because if I would need to move the images in the sorted directories into the aligned directory, I could write a short bash (shell) script that would do it for me, so I didn't see it as much of an issue. However, I will add an option to keep the original files where they are. ^_^

I'll explain the thing with how it sorts when using face and hist, since it should help clear up the weird behavior (although I have no idea how to abbreviate this, so it seems that I'd have to write a man page for this :p ). The way the original algorithm for 'face' and 'hist' works is by using a selection sort, which creates a smooth 'gradient' (like an increasing curve) of differences between the images. To do this a score gets calculated comparing two images and the images are then sorted based on the score. The way I group the images together is using the same score but instead checking it against the threshold value to decide if the image being checked should go into a given group (where each group has a reference image, which is the first image assigned to that group), and if no suitable group is found a new group is created. If you're using the face method, images where no faces are detected are sorted out before being compared to the threshold.

Also while testing to get it to work I used 12 images, 4 of face A and 4 of face B with both having tiny differences since they are sequential frames from a video, 3 images of a window/wall with rotations and effects, and 1 images of mountains. All scaled to 256x256px. So basically I haven't tested the threshold value enough to pick a good default.

I did just have a new idea how to better test whether an image belongs in a given group, although this may impact performance drastically.

I'll add the default values used to the cli options description and try to explain a bit better.

TL;DR: I'will add the improvements you mentioned and try to improve the way it groups the images together.

[AbysmalBiscuit]

@Kirin-kun I've added an option to keep the original files (-k, --keep) and improved the descriptions to be more verbose and to say the default values.

Can you take a look and tell me if it's clearer/better now?

I'll try working on improving the grouping now.

[AbysmalBiscuit]

@Kirin-kun After optimizing the face method (and concequently familirizing myself with the library used for it), I know why it found the same face but looking in different directions. It's because that's what it's meant to do. It is meant to find the same face in a variety of conditions.

Hence I have come up with a sample workflow:

1. Extract video that will have a variety of faces (A, B, C, D).
2. Sort them by face.
3. Sort each face by hist, as that will be sensitive to changes such as lighting, angle, etc...

Edit: I have messed about with a dataset of about 150 images of multiple faces and have deterimined sane defaults for the threshold value. I will update the description as well.

[AbysmalBiscuit]

@Kirin-kun I have optimized the group matching, set sane threshold defaults and explained the the threshold better. When you have the time take a look and tell me what you think. :)

[iperov]

also there is code duplication

img_list = [ [x, cv2.calcHist([cv2.imread(x)], [0], None, [256], [0, 256])] for x in tqdm(self.find_images(input_dir), desc="Loading") ]

[kcimit]

Would it be possible to create a log renames.log containing list [new file name];[original filename]

[iperov]

@AbysmalBiscuit also I suggest to add -by face-cnn

it uses FaceLandmarksExtractor.

example:

def process_face(self):
        input_dir = self.arguments.input_dir

        print ("Sorting by face similarity...")

        from lib import FaceLandmarksExtractor

        img_list = []
        for x in tqdm( self.find_images(input_dir), desc="Loading"):
            d = FaceLandmarksExtractor.extract(cv2.imread(x), 'cnn', True)
            img_list.append( [x, np.array(d[0][1]) if len(d) > 0 else np.zeros ( (68,2) ) ] )

        img_list_len = len(img_list)
        for i in tqdm ( range(0, img_list_len-1), desc="Sorting"):
            min_score = 9999999
            j_min_score = i+1
            for j in range(i+1,len(img_list)):

                fl1 = img_list[i][1]
                fl2 = img_list[j][1]
                score = np.sum ( np.absolute ( (fl2 - fl1).flatten() ) )                

                if score < min_score:
                    min_score = score
                    j_min_score = j            
            img_list[i+1], img_list[j_min_score] = img_list[j_min_score], img_list[i+1]

        self.process_final_rename (input_dir, img_list)

        print ("Done.")

[torzdf]

You can add it as an import (just tested).

add the import as normal, and execute the command from the faceswap dir as:

python -m tools.sort

I admit, it's not perfect and we may need to update some documentation, but it is doable.

[AbysmalBiscuit]

@kcimit I'll add a log feature.

@iperov can you explain in what way the img_list creation code is duplicated? It only occurs once in the process_hist_folders() method, and besides missing two formatting spaces it's the exact same method used in the process_hist() method.

I will add the -by face-cnn method.

@torzdf I think it would be better to add it to faceswap.py as an additional main command so as to have: extract, train, convert, sort. The way the sort tool parses the cli arguments would have to be modified but it's doable.

[Clorr]

@AbysmalBiscuit I don't want to add a new sort command to faceswap.py at least for now. You guys think in term of end user usage, and in that case I understand your point: there is a workflow where you extract, sort, train and then convert things. On my side, I don't think in term of end user usage, I do think in term of Machine Learning constraints.

The extract is not just extracting faces, it is also a preprocessing (for now align only, but we could some other that mask the background or generate different types of alignment) of images that have an impact on how the model is trained.

The convert is also specific as it takes output of the model (which can include mask, or other information) and post-processes images.

Sorting has no impact on model training, it is just here to make easier the cleanup process of extraction that is not perfect. If at some point the sorting or filtering or clustering of images can improve the model, we will see it coming and if it can't be made at extraction, I could consider adding another step, however for now, I'm just waiting until there is a need for that.

[iperov]

@AbysmalBiscuit also add

hist-unsim

first places photos which maximum non similar with each other

    def process_hist_unsim(self):
        input_dir = self.arguments.input_dir

        print ("Sorting by histogram unsimilarity...")

        img_list = [ [x, cv2.calcHist([cv2.imread(x)], [0], None, [256], [0, 256]), 0] for x in tqdm( self.find_images(input_dir), desc="Loading") ]

        img_list_len = len(img_list)
        for i in tqdm ( range(0, img_list_len), desc="Sorting"):
            score_total = 0
            for j in range( 0, img_list_len):
                if i == j: continue
                score_total += cv2.compareHist(img_list[i][1], img_list[j][1], cv2.HISTCMP_BHATTACHARYYA)

            img_list[i][2] = score_total

        print ("Sorting...")            
        img_list = sorted(img_list, key=operator.itemgetter(2), reverse=True)         
        self.process_final_rename (input_dir, img_list)

        print ("Done.")

also make same with face-unsim

example:

at first maximum non similar:

at end maximum similar:

[iperov]

^ so we can cut photos which useless for nn model

[AbysmalBiscuit]

@Clorr cheers for explaining the general direction of the project and intention for faceswap.py. :) I guess I do mostly have an end user perspective.

Would you be open to the idea of having something like a faceswap-tools.py or tools.py file? Since this would allow for a clear separation of the machine learning aspects while allowing for easy integration and usage of end user tools/features (maybe also integrating some of the more commonly used ffmpeg commands).

@iperov I'll add it in.

[Clorr]

Yup, i told that here ;-)

[AbysmalBiscuit]

Ah. I hadn't seen that. ^_^ The file paths issue with imports brought up in that thread was the reason why I wanted to use a root dir command script as well. :p

From experimenting earlier today with integrating sort into faceswap.py, I had re-written the sort.py cli parsing approach to more or less match scripts/train.py, so I have also just made a tools.py.

However there is an issue, since if there is a tools.py file and a directory called tools, the interpreter seems to always pick the file as the target for the import, do you know a way around this besides renaming one of them? For now I have renamed the tools directory to extra_tools.

Here is a gist for tools.py: https://gist.github.com/AbysmalBiscuit/391cc9e38ef452210a446b91faac559d

I have also updated my sort.py gist so that it can be used with tools.py. I have added the sort -by face-cnn method, but it doesn't seem to group properly into folders, and I haven't yet figured out why. Sorting by renaming works really well though, so it seems thatthe issue is something to do with how I group the images together/threshold values I have tried. the problem was the threshold values I was using, as the scores that get calculated are much bigger than 0.5. I still have to add face-unsim, hist-unsim and a file renaming logger.

[AbysmalBiscuit]

@iperov I have updated my gist with the hist-dissim and face-dissim methods; 'dissim' because it's dissimilarity and not unsimilarity. :)

Both methods work, but because I'm pretty tired at the moment I'm not sure if I have converted the face-dissim to work correctly, could you have a look and tell me what you think?

I still need to add the renaming logger option.

Edit: I forgot to add that the dissimilarity methods don't make much sense when organising things into distinct groups, so I have added logic to make it so that if someone uses the dissim methods with grouping by folders it defaults to the ordinary method: E.g. python tools.py sort -g folders -by face-dissim is the same as python tools.py sort -g folders -by face

[iperov]

looks ok

[AbysmalBiscuit]

@Clorr @iperov I have added the renaming/moving logging using a json file. I have also fixed the issue I had with importing from the tools directory, now it's possible to import from it directly (I had to create an empty __init__.py file in the tools directory).

As far as I see it, it's all done and ready to be commited. :)

I'm going to use a separate branch for submitting my pull request.

[iperov]

good job