Latest Staging Branch - Bugfixes

[torzdf]

Hi

I have just pushed some quite significant refactoring to the staging branch and would welcome testers to make sure that any final bugs are ironed out. Most of the work is behind the scenes, so there is no change to how the scripts are run. The following bugs have been squashed, however:

• Extract - Fix skip existing. Turns out this has been broken since the naming convention of output faces changed. It is not broken any more
• Train - Fix the -w flag bug. Using the -w flag would not write out the image if preview mode was enabled. Now it does
• Train - Enable the process to exit when target iterations have been reached. This currently only works when preview mode is enabled, but it used to not exit at all, so it's something
• Convert - Remove backwards compatibility for skip frames, and fix a Nonetype bug. I figure enough time has passed now
• Convert - Change what happens when there is no alignments.json. Since the move to face_alignments for face detection, not having an alignments file has been problematic, with OOM errors. The check for alignments file is now done at the beginning, and if a file does not exist, it is generated prior to moving on to conversion.

https://github.com/deepfakes/faceswap/tree/staging

Thanks

[Kirin-kun]

For the alignments.json not present, your fix extracts the faces as if an extract command had been run before?

That's where I think iperov's idea would be useful: embedding the landmarks information in the png of the extracted face. He didn't want to keep the alignments.json compatibility so "stormed out", but it was a nice idea anyway. I can see no downside to it. It would remove the need for "skipface", the alignments.json would become optional and it would help with the port of DFaker model that uses the landmarks for src AND dst.

[bryanlyon]

I agree we should look into it, but I don't think we should remove skip, since sometimes you want to rerun extract. I would want to keep that as default since skip is already available on it's own.

[torzdf]

To be clear. Skip is still in there, I just removed backward compatibility for old naming convention.

Yes, alignments.json is generated entirely if it doesn't pre exist... But only if it doesn't pre exist. This would need to happen anyway if the info was embedded in the pngs

[Kirin-kun]

Found a bug in "Convert - Change what happens when there is no alignments.json."

If you attempt to convert when there isn't an alignments.json, then it launches an extract.

BUT, it creates an empty "aligned" directory and proceeds to extract faces to... the "merged" directory you just specified on the command line, where you intended to put the converted faces.

I understand it uses the defaults options for extraction, but if it creates a directory, then it should use it.

[torzdf]

It shouldn't save faces at all if it's generating alignments.json from convert. I will take a look. Thanks!

[torzdf]

Ok, I'm confused...

It does create the output_dir for the merged folder at extract stage, but that is just an effect of the get_folder function which creates a folder if it doesn't exist.

The extract function still uses this if called from convert, but ultimately does not save any faces into it, as the convert function explicitly tells it not to save faces.

The created folder is then used when convert is run when it saves the converted faces to it. I don't really see this as a problem, as ultimately the folder is used, it's just a question of when it is created (prior to the extract or convert).

Ultimately the empty 'aligned' directory IS the final merged folder. This is because input_dir/output_dir is used for both the extract and convert functions. The extra coding required to ultimately reach the same end goal doesn't seem worth it.

[Kirin-kun]

I stand corrected. It doesn't save any aligned face. But I actually wanted it to... :/

My scenario: I had a directory with more pictures of B (celeb), but I didn't have an alignments.json nor an aligned directory.

I decided to test swapping (BtoA) on these pictures, so I directly ran convert, telling it to store the result in a "merged" subdirectory.

It noticed there wasn't an alignments.json, so proceeded to create an empty "aligned" directory and started to convert and save them in the merged directory.

Now, I think it would have been useful if it really extracted the faces, so I could re-use them eventually.

Also, when I checked the "cnn" extractor on this occasion, I had:

Failed to convert image: H:\Fakes\00SRC_DATA\200205121016267.JPG. Reason: Error while calling cudaGetLastError() in file C:\Users\Kirin\AppData\Local\Temp\pip-install-v8ubmnln\dlib\dlib\dnn\gpu_data.cpp:117. code: 2, reason: out of memory

So I had to rerun my "convert" with hog which detects less faces.

[torzdf]

What graphics card do you have?

I'm using a GTX1080, and this change means that it can extract and convert without generating OOM. It's still not perfect, but it's better than it was before.

Ultimately not extracting the faces is a deliberate choice, as if you want the faces to exist then you are far better off running extract, as you have far more control over how they are extracted and can review prior to running convert.

I would guess that in 90% of cases, anyone running convert without an alignments file already present is just looking to quickly generate swapped image with little care for keeping the extracted faces.

Either way, this change should make it easier to add an option to save the extracted faces at the convert stage in the future. This was my end goal here, rather than adding more options at this stage.

[Kirin-kun]

I have a GTX1060 6Gb.

[torzdf]

Ok, well that's slightly concerning, but still an improvement on before I think. Before it wasn't possible to convert at all without having an alignments file.

Ultimately I would like to get back to the position where it gets faces on the fly when it converts without an alignments file, but that will require further refactoring, and this is just a step towards that.

[Kirin-kun]

Erm..

I switched to staging and I still have the problem where if I don't give the -a parameter at convert time, I have a "NoneType" error.

In convert.py (in staging), I see:

def check_skipface(self, filename, face_idx):
        """ Check whether face is to be skipped """
        if self.faces_to_swap is None:
            return False
        face_name = "{}_{}{}".format(Path(filename).stem, face_idx, Path(filename).suffix)
        face_file = Path(self.args.input_aligned_dir) / Path(face_name)
        skip_face = face_file not in self.faces_to_swap
        if skip_face:
            print("face {} for frame {} was deleted, skipping".format(
                face_idx, os.path.basename(filename)))
return skip_face

I misread or it's still using the parameter, even if it's "None"?

[torzdf]

Thx, Will take a look.

[torzdf]

self.faces_to_swap is set to None if either no aligned directory is provided or there are no faces in the aligned directory

if self.faces_to_swap returns None then skip_face returns false

So this should work fine. Can you provide a traceback?

[Kirin-kun]

Problem seems to be on my side. I checked out staging, it said it's up to date but the scripts are those of master.

And when I pulled, it said:

C:\Users\Kirin\faceswap>git status
On branch staging
nothing to commit, working tree clean

C:\Users\Kirin\faceswap>git pull
There is no tracking information for the current branch.

I recloned a clean repo and it seems really up to date now.

C:\Users\Kirin\faceswap>git status
On branch staging
Your branch is up to date with 'origin/staging'.

nothing to commit, working tree clean

C:\Users\Kirin\faceswap>git pull
Already up to date.

Color me confused.

[Kirin-kun]

It's on purpose that the option "-D all" isn't available anymore?

[torzdf]

No. I'll take a look.

[Kirin-kun]

Also, on extraction, there's a cosmetic problem. Not really important though.

Starting, this may take a while...
  0%|                                                  | 0/126 [00:00<?, ?it/s]I
nfo: initializing keras model...
100%|████████████████████████████████████████| 126/126 [01:06<00:00,  1.89it/s]

[torzdf]

Yeah, I noticed that. #won't fix. May look at in the future.

I have pull 'all' back in for detectors. It got pulled out because I merged the extract and convert options together where possible. Extract had the 'all' option and convert did not. I happened to select the convert arguments.

I have pushed a fix, so you can re-pull if you need the all option.

[Kirin-kun]

Conversion on staging branch is broken, at least on my config. When I switched back to master, it worked.

Staging:

python c:\users\kirin\faceswap\faceswap.py convert --input-d
ir H:\fakes\pldg-sue --output-dir h:\fakes\pldg-sue\merged -m H:\Fakes\modelo.00
src_data.pldg-sue -b 4 -e 2 -S -D cnn -a h:\fakes\pldg-sue\aligned
C:\Program Files\Python36\lib\site-packages\h5py\__init__.py:36: FutureWarning:
Conversion of the second argument of issubdtype from `float` to `np.floating` is
 deprecated. In future, it will be treated as `np.float64 == np.dtype(float).typ
e`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Output Directory: h:\fakes\pldg-sue\merged
Input Directory: H:\fakes\pldg-sue
Loading Extract from Extract_Align plugin...
Using json serializer
Alignments filepath: H:\fakes\pldg-sue\alignments.json
Alignments filepath: H:\fakes\pldg-sue\alignments.json
Loading Model from Model_Original plugin...
loaded model weights
Loading Convert from Convert_Masked plugin...
  2%|?                                         | 2/126 [00:00<00:15,  8.13it/s]2
018-04-24 19:10:35.493314: E T:\src\github\tensorflow\tensorflow\stream_executor
\cuda\cuda_event.cc:49] Error polling for event status: failed to query event: C
UDA_ERROR_OUT_OF_MEMORY
2018-04-24 19:10:35.500314: F T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_event_mgr.cc:203] Unexpected Event status: 1

On master branch:

git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

python c:\users\kirin\faceswap\faceswap.py convert --input-d
ir H:\fakes\pldg-sue --output-dir h:\fakes\pldg-sue\merged -m H:\Fakes\modelo.00
src_data.pldg-sue -b 4 -e 2 -S -D cnn -a h:\fakes\pldg-sue\aligned
Input Directory: H:\fakes\pldg-sue
Output Directory: h:\fakes\pldg-sue\merged
Filter: filter.jpg
Using json serializer
Starting, this may take a while...
Loading Model from Model_Original plugin...
C:\Program Files\Python36\lib\site-packages\h5py\__init__.py:36: FutureWarning:
Conversion of the second argument of issubdtype from `float` to `np.floating` is
 deprecated. In future, it will be treated as `np.float64 == np.dtype(float).typ
e`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
2018-04-24 19:11:11.048347: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:1344] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 5.54GiB
2018-04-24 19:11:11.059348: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0
2018-04-24 19:11:11.690384: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1
edge matrix:
2018-04-24 19:11:11.697385: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:917]      0
2018-04-24 19:11:11.701385: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:930] 0:   N
2018-04-24 19:11:11.706385: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:
0/task:0/device:GPU:0 with 5331 MB memory) -> physical GPU (device: 0, name: GeF
orce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
loaded model weights
Loading Convert from Convert_Masked plugin...
Reading alignments from: H:\fakes\pldg-sue\alignments.json
100%|████████████████████████████████████████| 126/126 [01:00<00:00,  2.08it/s]
-------------------------
Images found:        126
Faces detected:      118
-------------------------
Done!

There's a double line "Alignments filepath"? The filter.jpg doesn't appear?

[Kirin-kun]

Maybe something is loaded twice and eats memory.

[torzdf]

filter.jpg won't appear, as I changed the default to None. It didn't make sense having a default of a file that doesn't exist.

[Kirin-kun]

CUDA_ERROR_OUT_OF_MEMORY seems to point to tensorflow being loaded twice.

[torzdf]

It does print twice, I'll look to fix that. Once when it checks whether you have alignments and the other when it loads them. Basically it loads face alignments twice, which isn't necessary.

I don't see that you should be running out of memory though. Can you run the -v flag on staging so it generates the tensorflow debug

[Kirin-kun]

Strangely, this time, I closed a browser where there was a video running and it passed.

python c:\users\kirin\faceswap\faceswap.py convert -v --inpu
t-dir H:\fakes\pldg-sue --output-dir h:\fakes\pldg-sue\merged -m H:\Fakes\modelo
.00src_data.pldg-sue -b 4 -e 2 -S -D cnn -a h:\fakes\pldg-sue\aligned
C:\Program Files\Python36\lib\site-packages\h5py\__init__.py:36: FutureWarning:
Conversion of the second argument of issubdtype from `float` to `np.floating` is
 deprecated. In future, it will be treated as `np.float64 == np.dtype(float).typ
e`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Output Directory: h:\fakes\pldg-sue\merged
Input Directory: H:\fakes\pldg-sue
Loading Extract from Extract_Align plugin...
Using json serializer
Alignments filepath: H:\fakes\pldg-sue\alignments.json
Alignments filepath: H:\fakes\pldg-sue\alignments.json
Loading Model from Model_Original plugin...
2018-04-24 20:03:59.449570: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:1344] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 5.65GiB
2018-04-24 20:03:59.459570: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0
2018-04-24 20:04:00.055604: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1
edge matrix:
2018-04-24 20:04:00.062605: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:917]      0
2018-04-24 20:04:00.066605: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:930] 0:   N
2018-04-24 20:04:00.069605: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:
0/task:0/device:GPU:0 with 5439 MB memory) -> physical GPU (device: 0, name: GeF
orce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
loaded model weights
Loading Convert from Convert_Masked plugin...
100%|████████████████████████████████████████| 126/126 [00:43<00:00,  2.90it/s]
-------------------------
Images found:        126
Faces detected:      118
-------------------------
Done!

Only difference I see:

totalMemory: 6.00GiB freeMemory: 5.54GiB vs now: totalMemory: 6.00GiB freeMemory: 5.65GiB

So, maybe it pushed the memory consumption just over the edge. Something you didn't see because you have a 1080.

[torzdf]

Also, if you get a chance could you try this:

in convert.py, remove from the top:

from scripts.extract import Extract

and insert it here (line 54):

    def generate_alignments(self):
        """ Generate an alignments file if one does not already
        exist. Does not save extracted faces """
        print('Alignments file not found. Generating at default values...')
        from scripts.extract import Extract
        extract = Extract(self.args)
        extract.export_face = False
        extract.process()

[torzdf]

I'm not sure it will make any difference, to be honest, but I can't test.

If it doesn't, then I'll probably have to dive into FaceLandmarksExtractor and only load the dlib components when they're requested.

[torzdf]

Oh, sorry. Didn't see that it worked.

Ok, maybe it's edge case. Ultimately GPU loading/unloading needs to be improved, for sure. Hopefully this issue doesn't re-occur when purely using faceswap.

[Kirin-kun]

Well, if your refactor eats more VRAM than before, it's hardly an improvement in that regard. Some people with the 2Gb adapters will be out.

[torzdf]

It shouldn't eat more VRAM.

It will probably eat more VRAM than the pre-staging version, because it removes the hacky fix to only load dlib modules when required (#315)... but this was a hack and far from ideal.

That should be mitigated by the fact that it now only loads modules for the script that is running. The version before the 'hack' used to load every module for every task, regardless of whether you were using that task or not. The current version only load modules related to that task, so DLIB will get loaded for extract/convert but not for Train.

Ultimately the refactor needs to go deeper, loading and unloading VRAM items when they're required, but it's one step at a time.

Ultimately, if this failed to work in every instance, then I would re-instate the hack, but hacks are not good, and it's better to fix at source, which i could hopefully do after this got merged.

[torzdf]

Thanks for your feedback @Kirin-kun I made some minor amends from your feedback #374. This has now been merged to the master branch

[Kirin-kun]

Is the filter/nfilter function broken?

C:\Users\Kirin\face>python c:\users\kirin\faceswap\faceswap.py extract -i H:\Fak
es\reach -o H:\Fakes\reach\aligned -D cnn -ae -r on -f h:\fakes\reach0306_0.png

C:\Program Files\Python36\lib\site-packages\h5py\__init__.py:36: FutureWarning:
Conversion of the second argument of issubdtype from `float` to `np.floating` is
 deprecated. In future, it will be treated as `np.float64 == np.dtype(float).typ
e`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Output Directory: H:\Fakes\reach\aligned
Input Directory: H:\Fakes\reach
Loading Extract from Extract_Align plugin...
Filter: ['h:\\fakes\\reach0306_0.png']
Traceback (most recent call last):
  File "c:\users\kirin\faceswap\faceswap.py", line 32, in <module>
    ARGUMENTS.func(ARGUMENTS)
  File "c:\users\kirin\faceswap\lib\cli.py", line 37, in execute_script
    process = script(*args)
  File "c:\users\kirin\faceswap\scripts\extract.py", line 19, in __init__
    self.faces = Faces(self.args)
  File "c:\users\kirin\faceswap\scripts\fsmedia.py", line 160, in __init__
    self.filter = self.load_face_filter()
  File "c:\users\kirin\faceswap\scripts\fsmedia.py", line 184, in load_face_filt
er
    facefilter = FaceFilter(filter_files[0], filter_files[1], self.args.ref_thre
shold)
  File "c:\users\kirin\faceswap\lib\FaceFilter.py", line 14, in __init__
    self.encodings = list(map(lambda im: face_recognition.face_encodings(im)[0],
 images))
  File "c:\users\kirin\faceswap\lib\FaceFilter.py", line 14, in <lambda>
    self.encodings = list(map(lambda im: face_recognition.face_encodings(im)[0],
 images))
IndexError: list index out of range

Same thing happens with nfilter.

[torzdf]

Ok, that's weird. I tested both of these options and they worked fine... will check,

[torzdf]

Ok, I just tested and I don't have this issue. It may be a Windows path thing...

python faceswap.py extract -i /mnt/fakes/Masters/A/white/frames/hope/25fps -o ~/fake/tmptest/hope -D cnn -ae -r on -f /mnt/fakes/Masters/A/white/faces/hope/1fps/hope_1fps_000039_0.png
/home/user/fake/env/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Output Directory: /home/user/fake/tmptest/hope
Input Directory: /mnt/fakes/Masters/A/white/frames/hope/25fps
Loading Extract from Extract_Align plugin...
Filter: ['/mnt/fakes/Masters/A/white/faces/hope/1fps/hope_1fps_000039_0.png']
Using json serializer
Alignments filepath: /mnt/fakes/Masters/A/white/frames/hope/25fps/alignments.json
Starting, this may take a while...
  0%|                                                                                                                                                                          | 0/1092 [00:00<?, ?it/s]Info: initializing keras model...
WARNING:tensorflow:From /home/user/fake/env/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:1349: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
  0%|▋                                                                                                                                                               | 5/1092 [00:28<1:44:28,  5.77s/it]No face encodings found
No face encodings found
  1%|▉                                                                                                                                                               | 6/1092 [00:29<1:30:16,  4.99s/it]No face encodings found
No face encodings found
  1%|█                                                                                                                                                               | 7/1092 [00:30<1:20:04,  4.43s/it]

etc. I don't suppose you have traceback kicking around that outputs the filter file name pre-change? I don't have a Windows testbox up at the moment.

[torzdf]

Does that file exist? The double backslash appears because python is escaping the backslash in the Windows path, but it should still work.

[torzdf]

Old code:

def load_filter(self):
        nfilter_files = self.arguments.nfilter
        if not isinstance(self.arguments.nfilter, list):
            nfilter_files = [self.arguments.nfilter]
        nfilter_files = list(filter(lambda fn: Path(fn).exists(), nfilter_files))
        filter_files = self.arguments.filter
        if not isinstance(self.arguments.filter, list):
            filter_files = [self.arguments.filter]
        filter_files = list(filter(lambda fn: Path(fn).exists(), filter_files))

        if filter_files:
            import_FaceFilter()
            print('Loading reference images for filtering: %s' % filter_files)
            return FaceFilter(filter_files, nfilter_files, self.arguments.ref_threshold)

New code:

    def load_face_filter(self):
        """ Load faces to filter out of images """
        facefilter = None
        filter_files = [self.set_face_filter(filter_type)
                        for filter_type in ('filter', 'nfilter')]

        if any(filters for filters in filter_files):
            facefilter = FaceFilter(filter_files[0], filter_files[1], self.args.ref_threshold)
        return facefilter

    def set_face_filter(self, filter_list):
        """ Set the required filters """
        filter_files = list()
        filter_args = getattr(self.args, filter_list)
        if filter_args:
            print("{}: {}".format(filter_list.title(), filter_args))
            filter_files = filter_args
            if not isinstance(filter_args, list):
                filter_files = [filter_args]
            filter_files = list(filter(lambda fnc: Path(fnc).exists(), filter_files))
        return filter_files

So, as far as I can see it builds in the same way, except that it now also loads if only nfilter is provided.

[Kirin-kun]

After various tries, I think it comes from my filter file.

It detects two faces in it (which is the case) , so it crashes.

Sorry, I'm an edge cases specialist :grinning:

[Kirin-kun]

And btw, there's no check if the filter files actually exist. If they don't exist, it proceeds happily.

[torzdf]

This actually excludes files which don't exist, but it does it silently: list(filter(lambda fnc: Path(fnc).exists(), filter_files) Maybe it's because it's a jpg rather than a png? I've never actually dug into the filter file code.

[Kirin-kun]

list(map(lambda im: face_recognition.face_encodings(im)[0], images))

Is probably expecting a single face. When it gets more, it errors.

[Kirin-kun]

No, no. It's a png. It's a face extracted previously, but there's another face in the background.

I actually wanted to make a second extraction pass so it doesn't extract that second face at all.

[Kirin-kun]

Yup, tested again and it's because it gets two faces in that pic. One in front and one blurry in the back.

[torzdf]

I guess that's technically a bug, so if you want to raise an issue, go ahead :)

[Kirin-kun]

As it is an edge case, I don't think it's worth it.

Maybe add a mention in the help for the extract command that the filter and/or nfilter images should contain a single face.