Append new entries to pickle file (KNNClassifier object)

face_recognition version: v1.22
Python version: 3.6
Operating System: Mac

Description

I am trying to add new encodings and names to saved pickle file (KNNClassifier object) - but unable to append.

What I Did

        # Save the trained KNN classifier
        if os.path.getsize(model_save_path) > 0:  
            if model_save_path is not None:
                with open(model_save_path, 'rb') as f:
                    unpickler = pickle.Unpickler(f)
                    clf = unpickler.load()

            newEncodings = X, y
            clf.append(newEncodings)

            with open(model_save_path,'wb') as f:
                pickle.dump(clf, f)

        else:
            if model_save_path is not None:
                with open(model_save_path, 'wb') as f:
                     pickle.dump(knn_clf, f)

Getting error : KNeighborsClassifier' object has no attribute 'append' Is there any way to achieve this? Please advice.

Other questions, if I train all images for every new training requests, does it going to impact the verification process as the pickle file is in use or OS can handle that?

I am working on moving to MySQL, if anyone did this please share your thoughts. Thank you!

* face_recognition version: v1.22

* Python version: 3.6

* Operating System: Mac
Description

I am trying to add new encodings and names to saved pickle file (KNNClassifier object) - but unable to append.

What I Did
        # Save the trained KNN classifier
        if os.path.getsize(model_save_path) > 0:  
            if model_save_path is not None:
                with open(model_save_path, 'rb') as f:
                    unpickler = pickle.Unpickler(f)
                    clf = unpickler.load()

            newEncodings = X, y
            clf.append(newEncodings)

            with open(model_save_path,'wb') as f:
                pickle.dump(clf, f)

        else:
            if model_save_path is not None:
                with open(model_save_path, 'wb') as f:
                     pickle.dump(knn_clf, f)
Getting error : KNeighborsClassifier' object has no attribute 'append' Is there any way to achieve this? Please advice.

Other questions, if I train all images for every new training requests, does it going to impact the verification process as the pickle file is in use or OS can handle that?

I am working on moving to MySQL, if anyone did this please share your thoughts. Thank you!

Hello, first off as the error suggests, a classifier behaves differently than a python list and you cannot append objects and arrays to it.

What it think you meant to do here is to update the classifier using new embeddings. Sadly within for most of the models, the online training option is not included. So this requires you to train the KNN once again from scratch (clf.fit()) with all the samples from the database which does not seem feasible.

I would recommend you to look up river which supports updating a classifier using only one sample. They also have a KNNClassifier instance, the difference is that you use model = model.learn_one(x, y) to train on one instance. So the replacement would be easy.

Thank you @KiLJ4EdeN

The online learning library river, provide learn_one(x, y) & predict_one(x) methods in KNNClassifier. I have rewritten my code to include river here:

def train_online(train_dir, model_save_path=None, n_neighbors=None, knn_algo='ball_tree', verbose=False):
    try:

        from river import neighbors
        from river import preprocessing
        from river import compose

        X = []
        y = []

         # Loop through each person in the training set
        for class_dir in os.listdir(train_dir):
            if not os.path.isdir(os.path.join(train_dir, class_dir)):
                continue

            # Loop through each training image for the current person
            for img_path in image_files_in_folder(os.path.join(train_dir, class_dir)):
                image = face_recognition.load_image_file(img_path)
                face_bounding_boxes = face_recognition.face_locations(image)
                # face_bounding_boxes = face_recognition.face_locations(image, number_of_times_to_upsample=2, model="cnn")

                if len(face_bounding_boxes) != 1:
                    # If there are no people (or too many people) in a training image, skip the image.
                    if verbose:
                        print("Image {} not suitable for training: {}".format(img_path, "Didn't find a face" if len(face_bounding_boxes) < 1 else "Found more than one face"))
                else:
                    # Add face encoding for current image to the training set
                    # X.append(face_recognition.face_encodings(image, known_face_locations=face_bounding_boxes, num_jitters=100)[0])
                    X.append(face_recognition.face_encodings(image, known_face_locations=face_bounding_boxes)[0])
                    y.append(class_dir)

        if n_neighbors is None:
            n_neighbors = int(round(math.sqrt(len(X))))
            if verbose:
                print("Chose n_neighbors automatically:", n_neighbors)

        model = compose.Pipeline(preprocessing.StandardScaler(), neighbors.KNNClassifier(n_neighbors=n_neighbors, algorithm=knn_algo, weights='distance'))

        for encoding, name in zip(X, y):
            model.learn_one(encoding, name)

        if model_save_path is not None:
            with open(model_save_path, 'wb') as f:
                pickle.dump(model, f)
        return 1

    except Exception as e:
        return "Error: {}".format(e)

Please help me in understanding the below items:

How do I save the model, as I did it in pickle format?
How to load and get this model, when I need it for prediction? (predict_one).

This is new to me, kindly advice.

I am closing this issue, as I have implemented using PostgreSQL, and I feel that is much better solution in my case.

If any starter facing same scenario, have a look at here it might help.

@KiLJ4EdeN thanks for your reply, I got to learn about online training.

ageitgey / face_recognition