Hotspot classification

[Marcel-Simon]

One low hanging fruit in our project is the hotspot classification, i.e. taking the hotspot detections provided by their algorithm and deciding which one actually corresponds to a seal and which not. Here are possible action items:

• Crop color image and IR image patches at the annotated hotspot locations and create an easy to use dataset out of it. The cropping can be done as in the visualization tool of @Athapan Make sure the seals are not tightly cropped, i.e. there is still at least 30 pixels border around the seal.
• Train an image classification model on color images only with any library you like on this dataset. This is a nice task for somebody, who has not much experience with machine learning but who would be interested in learning it. You can use any library and model you like, but a good starting point is pytorch and AlexNet / ResNet-50. It would be great to have an overview how different models perform. Feel free to ask for more details, if needed, in this issue thread.
• Investigate the fusion of color and IR images for classification. Possible would be stacking images, i.e. making a 4 channel image out of color + IR (more promising but can be tricky to get this running), or separate models for each type of image (easier to get it running, but maybe worse accuracy).

[CIPop]

@Marcel-Simon here's my idea for this item: I plan to recycle existing very deep models (VGG16 / inception-v3) using transfer learning:

• will detach the top layer and last 3 layers
• will freeze middle layers that will re-use imagenet training
• will re-attach a new top layer to account for the different size (based on #10 - all color boxes for anomaly vs seal are 512x512)
• will attach the last 3 conv, pool and backprop layers to account for the two classes

Variations:

• add data augmentation using Keras' ImageDataGenerator

[RealDataLLC]

@clpop

I agree. This seems to be the methodology most of the other endangered wildlife detectors are also using.

deep neural net +transfer learning + data augmentation.

Do we have the hotspot image data set though? Like would it make sense to overlay the thermal images over the normal images, basically adding a new dimension similar to using RGB images vs Grayscale in the input?

Which framework/example are you building it off of?

[cosminpa]

I'm also interested in this

[Marcel-Simon]

Happy about all the interest.

@CIPop The general idea as also described by @nelsonlin2708968 is perfectly right. However, I think, there is no need to change so much of the network architecture, you can keep the network input size and just resize the input images accordingly on the fly during training.

[RealDataLLC]

Hey there,

Just got some replies for other projects people have done.

The people working on the snow leopard identification (Mark Hamilton) sent me this for help making a transfer learning architecture. Step 1: https://mmlspark.blob.core.windows.net/website/index.html#install Go to Databricks section -> follow instructions to create and import library and examples Step 2: Run the flowers Image Featurizer example notebook to see some transfer learning examples! Let me know if you run into issues or need more help!

This is from the AI for earth people. • If you aren’t an expert in deep learning frameworks, you can leverage the Custom Vision service in Azure, which is doing transfer learning under the covers and there is less of a learning curve here. You can access the service at http://customvision.ai. I give a quick demo of this in my talk at https://channel9.msdn.com/Events/Ignite/Microsoft-Ignite-Orlando-2017/BRK3296 - start at time 40:40 for a slide or two and then a demo.

[CIPop]

Created a pre-release with a 1.5GB model based on VGG16 + transfer learning. The model is checked in at https://github.com/Microsoft/arcticseals/blob/master/src/hotspotdetection/VGG16transferlearning.ipynb