agentmorris
commented
1 year ago Thanks for sharing! I will try to answer one of your questions here:
What exactly are ecologists looking to extract out of camera trap images?
It varies from project to project, but some common themes:
- Maybe 85% of our collaborators are focused on estimating wildlife populations. Among those, most go through every image and quantify the number of individuals present for every species present. Some add additional features (e.g. gender, behavior, antler size), but species is the least common denominator. Most of these projects will also quantify humans and vehicles, but as a secondary goal.
- Most of the remaining 15% are primarily focused on humans and vehicles for offline surveys, e.g. park managers who want to quantify trail use.
- A small but growing number of users are interested in detecting humans and vehicles in real-time for anti-poaching applications. This is a very small fraction of global camera trap use, but is growing fast, and is a critical case for AI.
- Estimating weather from camera traps is something we've encountered, but it's rare.
I think these are relatively representative of the larger camera trap community, but take what I say with a grain of salt, there are a number of biases in whom we end up in contact with.
Hope that helps!
-Dan
(Comment originally posted by agentmorris)
Hey all,
Not sure if it falls under an issue (I'm happy to post or x-post it somewhere more suitable). This repo may be the best place to ask some questions about what's needed to be integrated in camera traps, since individuals with technical backgrounds, with some understanding on research needs, are likely to stumble upon the repo.
I'm an AI for Earth grantee (agritech and their integration within smart cities). I learned about camera traps during the AI for Earth Summit, and coincidentally, as a part of an online competition (Microsoft's Discover AI Upskilling Journey) where organizers have asked participants to look into helping out with camera traps.
However, what is not particularly clear is, what exactly are ecologists looking to extract out of camera trap images? Animals (detection and classifying which animal)? Other objects related to poaching (people, cars, trucks)? Other features (weather conditions, time stamps of when detection happens)?
Understanding what needs to be extracted can help us (researchers and hobbyists) understand what needs to be developed, as well as tailor the hardware and integration to better fit what is needed.
Overview of my project:
The goal of this project is to highlight how effective Edge AI can be in eliminating/filtering out a significant number of inter-steps between data collection and obtaining classification and detection results, potentially along with other data/features that need to be collected or extracted.
More information: The project aims to tie-in features from machine learning and internet of things with hardware devices, similar in some aspects to what is currently used on the ground, in order to reduce the overall time spent ecologists spend on classifications, collecting data, etc... As someone who has never interacted with ecologists, it's difficult to understand what is currently needed for them to better allocate their time and focus on research aspects, rather than countless hours on manual data labelling.
Model: I trained a custom object detection and classification (Yolov4) model for 17 classes, which are all the classes of animals that I was able to manually make out in the MegaDetector demo video. The classes are Armadillo, Bear, Bird, Bull, [Car], Cat, Cattle, Deer, Dog, Fox, Monkey, [Person], Pig, Raccoon, Sheep, Tiger, [Truck]. I added a few classes in [brackets], just in case someone would find the model useful as is.
The images used for training are obtained from Open Images Dataset V6.
Then, I used the model to run object detection and classification on some test images (the sheep image in the overview is one of the test images used) and the MegaDetector demo video.
Testing images (after ~5hrs training): Yolov4 TFLite (TensorFlow Lite model, converted from yolov4)
Yolov4
Video: Running the yolov4 model on the MegaDetector demo video (less than 2 minutes for the model to detect and classify the video).
It seems to be pretty ok for some parts, needs more training for other parts. Some examples for detection/classification:
Some examples for bad detection/classification (model needs more training?):
Extracting more information from images
I was reading that ecologists also look at camera trap images to extract more information about the ecosystem, such as weather conditions.
My hand-waving approach was to use Custom Vision to train weather data. I stumbled upon a Multi-class Weather Dataset for Image Classification, which contains 4 classes of weather conditions. The dataset was used for training.
However, testing the model on some images, the model does not seem to be the best for camera-traps. Even though some classifications were on point, the accuracy is really low.
Another hand-waving approach to extract weather data from camera-trap images (haven't implemented this yet):
It seems that some images display temperature data in each image. Another approach is to extract temperature information as well as time/date stamps from images (OCR). Then use historical weather datasets (such as openweather API) to fetch other weather features such as humidity, wind direction/speed, weather conditions (rainy, cloudy, partially sunny, sunny), etc..
Raspberry pi as an Edge AI Device
As implied earlier, the main reason to convert the Yolov4 model to a TensorFlow Lite model is for it to be run on the raspberry pi. This means that as an image or video is taken, object detection and classification happens on the device itself, where the results can be either uploaded directly to the internet or saved on an SD card.
As the weather information is relatively more challenging to extract directly from images and videos, integrating relatively cheap sensors may provide a more custom overview of the ecosystem. For example, it is possible to add sensors to collect data about emissions such as COx, NOx, NH3, VOCs as well as air quality. It would also allow for more customizability when it comes to choosing a camera that may be a better fit for custom applications, as well as detection sensors (in this example, pyroelectric/ infrared sensor in addition to an ultrasonic sensor).
Issue cloned from Microsoft/CameraTraps, original issue posted by ejri on Jun 25, 2020.