Use own face detection module

[mark-selyaeff]

Hello, I want to use my own network for face detection, so I tried to pass face_detector=None to FaceAlignment class, but it gives me an error.

Is there any functionality to pass cropped faces or its bounding boxes to the get_landmarks method?

[1adrianb]

Hi @mark-selyaeff, please see the FolderDetector here https://github.com/1adrianb/face-alignment/blob/master/face_alignment/detection/folder/folder_detector.py which should do what you want

[mark-selyaeff]

@1adrianb, thank you for a fast response! I am no quite sure how to use it though since no documentation is provided. Assuming that I have a list of face bounding boxes in the format list[(x1,y1,x2,y2),...], I don't understand why according to docstring it searches for a file with the same name as the input image. My guess is that I should provide bounding boxes for an image in this file. Right?

Even if so, running this code:

import face_alignment
from face_alignment.detection.folder import FaceDetector as FolderDetector

folder_det = FolderDetector('cpu')
fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, 
flip_input=False, face_detector=folder_det)

raises an error:

Traceback (most recent call last):
  File "/landmarks_2d.py", line 8, in <module>
    fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, 
flip_input=False, face_detector=folder_det)
  File "/lib/python3.7/site-packages/face_alignment/api.py", line 64, in __init__
    face_detector_module = __import__('face_alignment.detection.' + face_detector,
TypeError: can only concatenate str (not "FolderDetector") to str

[1adrianb]

@mark-selyaeff You don't really need to instantiate the detector outside, simply passing face_detector='folder' as a string should do the job. The format of the files containing the bbs should follow that from https://github.com/1adrianb/face-alignment/blob/6a7731168dbb1a15f9ecd5fe4c79c992f179a622/face_alignment/detection/folder/folder_detector.py#L22 You can simply modify the above the function in case you have a widely different format.

[Bao150297]

@mark-selyaeff Did you find the way to perform your idea? If you done, plz share with me.

[rakadambi]

Hello, I am also trying to do the same. For instance, I have the file with bbs in the current working directory and I am calling the aligner like this:

fa = api.FaceAlignment(api.LandmarksType._2D, face_detector=os.getcwd(). device = 'cpu')

However, I get this after I call:

ModuleNotFoundError: No Module named detection....

What am I doing wrong?

[rakadambi]

I tried passing face_detector=none to FaceAlignment constructor and it says

ModuleNotFoundError: No Module named detection.none.

Confused here!

[1adrianb]

@rakadambi you need to pass face_dector='folder' which will use this class: https://github.com/1adrianb/face-alignment/blob/master/face_alignment/detection/folder/folder_detector.py

Please check its code to see the expected format, but basically it will look for a a numpy or torch file with the same name as the image (but different extension).

[Bao150297]

I could use my own detection model by modifying this line preds = fa.get_landmarks(input_img, [[0, 0, img.shape[0], img.shape[1])[-1] in file detect_landmarks_in_image.py , with input_img is the face as output of my detection model. For @1adrianb , will you release other models? Your work is amazing, but with 20fps I got when running on GTX 1080, it wasn't enough. You released lua version, but using it as a plugin for a Python project is hard for me :(

[rakadambi]

Hi @1adrianb

I passed face_detector='folder'. It seemed to proceed a bit. This is what I did.

1. I detected the face bounding box and saved it in an array [x1, y1, x2, y2].
2. Saved this using np.save to a file. The file has the same base name as the image file name.
3. Call get_landmarks(image_path, np_saved_file). The second argument is the saved file (with .npy extension).
4. This fails in line 150 of api.py file when trying to get "center".

I think it should ideally go to folder_detector.detect_from_image. So, this is what I did.

In api.py

I put detect_faces = np.load(filename.npy)

This got me detected_faces = [x1, y1, x2, y2]

Still fails after enumerate(detected_faces)

So, I traced through your code with sfd. The detected faces is of the format:

detected_faces = [array([x1, y1, x2, y2, val].

What is "val" here? Is it scale?

I would appreciate your input.

[1adrianb]

@rakadambi val is the confidence of the detector, simply setting it to 1 should be fine

[rakadambi]

I got this to work. This is how one can do it. Thanks to @1adrianb for his patience with all my questions.

Method to use your own face detector.

1. Call FaceAlignment with face_detector = folder argument.
2. Detect all faces that you want from your face detector. Add the 5th value to each numpy array of 1 (confidence factor). Store them in a python list of numpy arrays.
3. Call get_landmarks(). The arguments should be the image absolute file path and the list of arrays that you got from step 2 above.

Done.

Notes: I don't see folder_detector function being used at all.

[Coderx7]

@rakadambi Thanks a lot. it'd be a treat if you could make that into a PR, or link your own repo, so we dont go through the hassle of that again . Your feed back is greatly appreciated so far. Thank you both

[rakadambi]

Hi @Coderx7 . I was just testing out the changes and this worked in my local repo. I am not pushing this to any repo as it is experimental. But I expect to do so in the coming weeks and I will post the link here. In the meantime, If you have other questions on this implementation, I will be happy to reply here.

[Coderx7]

@rakadambi Thank you very much. sure do, if anything comes up I'll be asking here Have a great day sir :)

[SaddamBInSyed]

@rakadambi Hi, as agreed, can you post the link over here? so that others can benefit.

[rakadambi]

Hi @SaddamBInSyed I have not pushed it to any repo. But the snippet of code works well with your own face detector. It is really these two lines of code: fa = FaceAlignment(LandmarksType._2D, face_detector='folder', device='cpu') landmarks = fa.get_kandmarks(my_image, <list of faces>)

Please let me know if I can help further

[SaddamBInSyed]

Thanks for the reply.

[andreasmarxer]

Hi @1adrianb @rakadambi @SaddamBInSyed

---

Edit: Problem solved, the fault was in the line: bb_npy = list(np.load(bb_path)), this needs to be changed to: bb_npy = [np.load(bb_path)].

---

I'm currently having problems when using my own face detector. I'm only interested in one specific face per image, first I'm saving this face in a npy file per image as following:

bb_list = [x, y, x+w, y+h, 1]
out_path = SAVE_FOLDER + filename_wo + '.npy'
np.save(out_path, bb_list)

1) Is it correct that the 5th value, the confidence is also needed? Because in the description of the class FolderDetector this 5th value is not mentioned. (see https://github.com/1adrianb/face-alignment/blob/master/face_alignment/detection/folder/folder_detector.py)

Further, I have adapted the code as proposed by @rakadambi:

# landmark detection
bb_folder = 'bb'
fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, flip_input=False, face_detector='folder' )

bb_path = filename_wo + '.npy'
bb_npy = list(np.load(bb_path))
print('bb_npy: ')
print(bb_npy)
print(type(bb_npy))

landmarks = fa.get_landmarks(image, detected_faces=bb_npy) #[-1] # np array with floats

2. Is it correct to have the string 'folder' as an input to the function in line 3? (I think previously I used the path and it did not work either)

The output then throws me the error: IndexError: Invalid index to scalar variable and looks like:

Anyone sees my fault or has an idea? Thanks in advance and stay save :)

[rakadambi]

Hi @andreasmarxer I think the API expects a Python list of np arrays. The np arrays themselves are the face bounding boxes. I set the 5th element to 1.

Can you just try with a single face. Take your np array that you read from the filename, add 1 as confidence factor to this np array, add it to a python list and it should work.

[andreasmarxer]

Thank you @rakadambi . As edited in my comment above, I already have solved the issue. The confidence factor I already had considered, the problem was the conversion into a list, that didn't work as expected.

Thank you!

[hermankant]

@1adrianb Thank you for still being active and responsive!

@rakadambi Thank you for all your help here. Was wondering if you could help me with something as I can't figure this out. I'm trying to use haar cascade detector. I'm taking just one face for testing, appending 1 to the list of the returned bb yet the landmark API returns errors. This is the final relevant attempt with the detected face:

for f in range(len(faces)):
    tempL = faces[f].tolist()
    tempL.append(1)
    # newF.append(tempL)

preds = fa.get_landmarks(inp, tempL)

The error it returns is:

Traceback (most recent call last):
  File "C:\Users\herma\anaconda3\lib\site-packages\torch\autograd\grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\herma\anaconda3\lib\site-packages\face_alignment\api.py", line 149, in get_landmarks_from_image
    [d[2] - (d[2] - d[0]) / 2.0, d[3] - (d[3] - d[1]) / 2.0])
TypeError: 'int' object is not subscriptable

Any idea what I should do? I thought maybe the values of the bb should be ints, and I changed them to floats floated = [float(i) for i in tempL] and ran the get_landmarks with the floated version, yet then I got an error saying TypeError: 'float' object is not subscriptable

So I'm not sure what I should call it with there and what am I not understanding

[rakadambi]

How did you initialize "fa"? I can't see it in the code above?

[hermankant]

@rakadambi fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._3D, face_detector='folder', device='cpu')

as for faces, it looks like that: faces = face_cascade.detectMultiScale(inp,scaleFactor=1.1,minNeighbors=5)

While we're at it. Are there other builtin detectors I could use and type in here detector = 'other'? I just prefer speed and haar provided me that, with sufficient accuracy.

[rakadambi]

For the landmarks call, you should send a list of np arrays. In your case the list will have just one np array with 5 elements, the fifth being 1 (which you are appending). Are you sure your list has one np array?

[hermankant]

@rakadambi I though a list is good enough. After what you wrote I added a line just in case:

for f in range(len(faces)):
    tempL = faces[f].tolist()
    tempL.append(1)
    array = np.array(tempL)
    # floated = [float(i) for i in tempL]
    # newF.append(tempL)

preds = fa.get_landmarks(inp, array)

when array contains (after the last interation):

and got the following error:

  File "C:\Users\herma\anaconda3\lib\site-packages\torch\autograd\grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\herma\anaconda3\lib\site-packages\face_alignment\api.py", line 149, in get_landmarks_from_image
    [d[2] - (d[2] - d[0]) / 2.0, d[3] - (d[3] - d[1]) / 2.0])
IndexError: invalid index to scalar variable.```

[rakadambi]

I think your "array" is an np array. Your "array" should be a list. This list should have np arrays. Each such np array has 5 elements

[hermankant]

@rakadambi I see. But now I'm getting a different error. Frustrating.. changed to:

newL = []
for f in range(len(faces)):
    tempL = faces[f]
    np.append(tempL,[1])
    newL.append(tempL)

preds = fa.get_landmarks(inp, newL)

where newL is:

and the error is

Traceback (most recent call last):
  File "C:\Users\herma\anaconda3\lib\site-packages\torch\autograd\grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\herma\anaconda3\lib\site-packages\face_alignment\api.py", line 153, in get_landmarks_from_image
    inp = crop(image, center, scale)
  File "C:\Users\herma\anaconda3\lib\site-packages\face_alignment\utils.py", line 113, in crop
    newImg = np.zeros(newDim, dtype=np.uint8)
ValueError: negative dimensions are not allowed

regardless are you ware of a simpler way of just specifying a different detector?

And again, thanks!!!!

[rakadambi]

From your newL, it looks like it is [x, y, w, h]

I checked my code. It should be [x, y, x+w, y+h, 1]. In other words, the width and height need to be replaced by actual coordinates.

Can you try that?

[hermankant]

@rakadambi I think it did the trick. Thank you!