search

hukkelas / DSFD-Pytorch-Inference

A High-Performance Pytorch Implementation of face detection models, including RetinaFace and DSFD
Apache License 2.0
218 stars 58 forks source link

DSFD stopping without error #16

Closed QuentinSpalla closed 3 years ago

QuentinSpalla commented 3 years ago

Hi,

I am currently using the DSFDDetector on thousands of images. This is working well for a few images but after 1k or 2k images the script stopped without any bug or any errors. I know that the script is not finished because I do not see my ending prints.

Just to be clear, I am also running the exact same script with the RetinaNetResNet50 detector and it is working perfectly.

I am working on a GCP VM in Debian 9. Python3.7.6, torch==1.2.0, face-detection==0.1.4 Currently I am using a Tesla P100 GPU for the inferences.

Has somebody ever encountered a similar issue ?

hukkelas commented 3 years ago

Could it be the program running out of memory?

If you are using different shapes for each image, there could be an increase in memory consumption per image. We are caching the priors for each image shape (see here).

You could try to disable the caching, however, this will increase the runtime quite a bit.

QuentinSpalla commented 3 years ago

Thank you for you quick answer.

Could it be the program running out of memory?

Yes maybe it could be possible but why RetinaNetRestNet50 is not running out of memory on the same images ? Both detectors do not save in cache the images ?

You could try to disable the caching, however, this will increase the runtime quite a bit.

Ok thank you for the advice it I want to disable the caching I juste have to comment the line you sent ? And you are saying that it will be slower, but do you know approximately in what proportions ?

hukkelas commented 3 years ago

Yes maybe it could be possible but why RetinaNetRestNet50 is not running out of memory on the same images?

That's true. We're caching the priors in RetinaNet somewhat different, therefore, it might not run into the same issue.

Ok thank you for the advice it I want to disable the caching I juste have to comment the line you sent ?

Yep!

And you are saying that it will be slower, but do you know approximately in what proportions ?

This is highly dependent on the dataset you have. If all images has a different shape, then it should be equally fast.

Another point is that it might be that you're running out of memory on your GPU. You could limit the max resolution to (for example) 1080.