MegaDetector crashes on VM instance - Githubissues

agentmorris / MegaDetector

MegaDetector is an AI model that helps conservation folks spend less time doing boring things with camera trap images.

MIT License

117 stars 26 forks source link

MegaDetector crashes on VM instance #45

Closed agentmorris closed 1 year ago

agentmorris commented 1 year ago

I'm trying to run the MegaDetector over some images on a GCP VM instance with a Tesla T4 GPU. I created a conda virtual environment using the environment-detector.yml file with the tensorflow-gpu version set to 1.13.1, and I am trying to run the run_tf_detector_batch.py script as instructed in the MegaDetector page. However, the process crashes while processing the first image (below is a screenshot of my terminal window when that happens):

Screen Shot 2021-03-25 at 1 34 27 AM

I've tried to trace the program but haven't been able to figure out what the issue is. Any idea why this might be happening and how to solve it? Thanks!

Issue cloned from Microsoft/CameraTraps, original issue posted by pbick on Mar 25, 2021.

agentmorris commented 1 year ago

Update: if I change the tensorflow-gpu version to 1.14.0, the last error ("axis attribute is out of bounds") no longer happens, but the memory errors persists and the process is still killed.

(Comment originally posted by pbick)

agentmorris commented 1 year ago

Sorry you're having trouble with MegaDetector... a couple questions:

In the new scenario, with TF 1.14, do you see "...exceeds 10% of system memory", then just "Killed"? There's no error before "Killed"?
Confirm that you are not running inside Docker or any other virtualized environment?
When all else fails, try TF random versions... if it's not too much trouble, do you mind trying TensorFlow 1.15 also?

(Comment originally posted by agentmorris)

agentmorris commented 1 year ago

I changed the tensorflow-gpu version to 1.15.0 and it seems like it worked! Thanks!

(Comment originally posted by pbick)

agentmorris commented 1 year ago

Glad we got it running!

"Do random stuff until it works": still the most reliable solution to technology problems.

(Comment originally posted by agentmorris)