search

ageitgey / face_recognition

The world's simplest facial recognition api for Python and the command line
MIT License
52.35k stars 13.36k forks source link

GPU memory not get released after using CNN for face location in web_service_example.py (flask example) #722

Open kshitij005 opened 5 years ago

kshitij005 commented 5 years ago

Description

GPU memory not released after adding "CNN for face location" line in web_service_example.py

What I Did

By adding **line of finding face location using CNN** in web_service_example.py like this ::
    **# Load the uploaded image file
    img = face_recognition.load_image_file(file_stream)

    _# Get face locations for any face in the uploaded image
    unknown_face_location = face_recognition.face_locations(img, number_of_times_to_upsample=1, model="cnn")_

    # Get face encodings for any faces in the uploaded image
    unknown_face_encodings = face_recognition.face_encodings(img, unknown_face_location)**

i found that my GPU does not release memory after CNN,
So when i try to upload image 1st time, it uses 77% of GPU's memory and when i upload image 2nd time it starts using GPU' memory from 77% ( that means GPU is not releasing its memory after complition of request),
after multiple attempts GPU's memory reach to 100% and then it shows ::
  **return cnn_face_detector(img, number_of_times_to_upsample)
RuntimeError: Error while calling cudaMalloc(&data, n) in file /home/dlib-19.13/dlib/cuda/cuda_data_ptr.cpp:28. code: 2, reason: out of memory**

i tried ::
**del unknown_face_location
gc_collect()**
below unknown_face_location
but it didn't work, 
when i restart flask only then GPU releases memory (which is not good option to use)

i used following version::
Nvidia Driver : nvidia-396
CUDA Version 9.0.176
Cudnn Version 7.4.2
Dlib Version 19.13

I need to know how to overcome from this memory issue by using another drivers and version or by changing the code.
NOTE: i need CNN and when i use CNN in a code where i don't use Flask and other web service applications it works fine.(I am new to this So spare me for any small mistakes done by me)
tutugreen commented 5 years ago

+1 for adding cnn for knn training.

mohitwadhwa2 commented 3 years ago

Did you find any solution to resolve the Memory issue? If yes, Please share because I am also facing this memory issue with face recognition module

mohitwadhwa2 commented 3 years ago
  • face_recognition version: 1.2.3
  • Python version: 3.5.2
  • Operating System: Ubuntu 16.04 LTS
  • GPU: NVIDIA GeForce GTX 1050TI SSE2

Description

GPU memory not released after adding "CNN for face location" line in web_service_example.py

What I Did

By adding **line of finding face location using CNN** in web_service_example.py like this ::
    **# Load the uploaded image file
    img = face_recognition.load_image_file(file_stream)

    _# Get face locations for any face in the uploaded image
    unknown_face_location = face_recognition.face_locations(img, number_of_times_to_upsample=1, model="cnn")_

    # Get face encodings for any faces in the uploaded image
    unknown_face_encodings = face_recognition.face_encodings(img, unknown_face_location)**

i found that my GPU does not release memory after CNN,
So when i try to upload image 1st time, it uses 77% of GPU's memory and when i upload image 2nd time it starts using GPU' memory from 77% ( that means GPU is not releasing its memory after complition of request),
after multiple attempts GPU's memory reach to 100% and then it shows ::
  **return cnn_face_detector(img, number_of_times_to_upsample)
RuntimeError: Error while calling cudaMalloc(&data, n) in file /home/dlib-19.13/dlib/cuda/cuda_data_ptr.cpp:28. code: 2, reason: out of memory**

i tried ::
**del unknown_face_location
gc_collect()**
below unknown_face_location
but it didn't work, 
when i restart flask only then GPU releases memory (which is not good option to use)

i used following version::
Nvidia Driver : nvidia-396
CUDA Version 9.0.176
Cudnn Version 7.4.2
Dlib Version 19.13

I need to know how to overcome from this memory issue by using another drivers and version or by changing the code.
NOTE: i need CNN and when i use CNN in a code where i don't use Flask and other web service applications it works fine.(I am new to this So spare me for any small mistakes done by me)

Why are you calculating the face locations and then calculating the embeddings? If you just want the embeddings, then you can directly pass the image and it will calculate the face embeddings. Did you find any solution to resolve this memory issue?

kshitij005 commented 3 years ago

Yes, go inside the library and delete model variable

On Tue, Mar 23, 2021, 12:05 PM mohitwadhwa2 @.***> wrote:

  • face_recognition version: 1.2.3
  • Python version: 3.5.2
  • Operating System: Ubuntu 16.04 LTS
  • GPU: NVIDIA GeForce GTX 1050TI SSE2

Description

GPU memory not released after adding "CNN for face location" line in web_service_example.py What I Did

By adding line of finding face location using CNN in web_service_example.py like this :: **# Load the uploaded image file img = face_recognition.load_image_file(file_stream)

_# Get face locations for any face in the uploaded image
unknown_face_location = face_recognition.face_locations(img, number_of_times_to_upsample=1, model="cnn")_

# Get face encodings for any faces in the uploaded image
unknown_face_encodings = face_recognition.face_encodings(img, unknown_face_location)**

i found that my GPU does not release memory after CNN, So when i try to upload image 1st time, it uses 77% of GPU's memory and when i upload image 2nd time it starts using GPU' memory from 77% ( that means GPU is not releasing its memory after complition of request), after multiple attempts GPU's memory reach to 100% and then it shows :: return cnn_face_detector(img, number_of_times_to_upsample) RuntimeError: Error while calling cudaMalloc(&data, n) in file /home/dlib-19.13/dlib/cuda/cuda_data_ptr.cpp:28. code: 2, reason: out of memory

i tried :: del unknown_face_location gc_collect() below unknown_face_location but it didn't work, when i restart flask only then GPU releases memory (which is not good option to use)

i used following version:: Nvidia Driver : nvidia-396 CUDA Version 9.0.176 Cudnn Version 7.4.2 Dlib Version 19.13

I need to know how to overcome from this memory issue by using another drivers and version or by changing the code. NOTE: i need CNN and when i use CNN in a code where i don't use Flask and other web service applications it works fine.(I am new to this So spare me for any small mistakes done by me)

Why are you calculating the face locations and then calculating the embeddings? If you just want the embeddings, then you can directly pass the image and it will calculate the face embeddings. Did you find any solution to resolve this memory issue?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ageitgey/face_recognition/issues/722#issuecomment-804656787, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKEFMPTGEPKWXCQH4ZM5DGLTFAZBJANCNFSM4GR3TYJQ .

epicchen commented 3 years ago

https://github.com/ageitgey/face_recognition/issues/899#issuecomment-809383111

flariut commented 2 years ago

I'm also having this issues with memory leaking. I wrote a service that continously compares known faces, it's works flawlessly but slowly and steadily the GPU memory usage goes up by a little, and without proper control it maxes out and CUDA starts failing, I triple checked my code but I can't find any issues with it. Is this a face_recognition or dlib issue? Is there perspective on fixing this rather than just resetting the variables or the program itself, as this requires to deserialize the model again and again, losing uptime? @ageitgey