Lecture 11: Detection and Segmentation

[hon9g]

Semantic Segmentation

• No objects, just pixels
• Input: Image
• Output: the decision of a category for every pixel. (각 픽셀이 어떤 category에 해당하는지)

Classification + Localization

• Singel object
• not only What the category is, but also Where is the object in the image?
• output: Bounding Box, label

Object Detection

• Multiple objects
• output: bounding box, label
• R-CNN -> Fast R-CNN -> Faster R-CNN
• Sliding window approach
• Region Proposals

Instance Segmentation

• Multiple objects
• bounding box 대신 segmentation

reference

[cs231n lecture 11: Detection and Segmentation](https://www.youtube.com/watch?v=nDPWywWRIRo&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=11)

[hon9g]

Semantic Segmentation

• Label each pixel in the image with a category label.
• Don't differentiate instances, only care about pixels.

Sliding Window

• take an input image and break it up into many many tiny local crops of image.
• now, treating as a classification problem for tiny crops.
• Weakness: It is super super computationally expensive.
  • because we want to take a label of every pixel in the image.
  • we would need a separate crop for every pixel in the image. (it will super super expensive for the run forward and backward)
  • patches are overlapped. we can actually share computation between different patches.
  • It is terrible. but it is first thing might you think of semantic segmentation.

Fully Convolutional

• Rather than extracting individual patches from the image,
• Just having the network be a whole giant stack of convolutional layers with no fully connected layers.
• Each convolutional layer preserves the spatial size of input.
• output: classification score for every pixel in the image
• weakness: It is super super computationally expensive (and also take a ton of memory), since applying a bunch of convolutions that are all keeping the same spatial size.

Downsampling & Upsampling

• computational efficient than maintain hign resolution.

In-Network upsampling: "Upooling"

In-Network upsampling: "Max-Upooling"

Learnable Upsampleing: Transpose Convolution

• backward 연산을 하면 convolution 연산과 같다.

[hon9g]

Classification + Localization

• Single object
• not only What the category is, but also Where is the object in the image?
• output: Bounding Box, label

[hon9g]

Object Detection

• output: bounding box, label
• Multiple objects
  • Object Detection is different from localization because there might be a different number of objects in the image. (you don't know how many you expect to find)
• Sliding window
  • Apply a CNN to many different crops of the image, CNN classifies each crop as object or background.
  • problem: Need to apply CNN to a huge number of locations and scales, very computationally expensive!
• Region Proposals
  • Find "blobby" image regions that are likely to contain objects.
  • Relatively fast to run; e.g. Selective Search gives 2000 region proposals (that seems object is there) in a few seconds on CPU
  • It might not correct mostly, but It provides high recall
  • CPU computation
• R-CNN -> Fast R-CNN -> Faster R-CNN

[hon9g]

R-CNN

• given input image

  • -> get region proposals of it

• these regions proposed could have different size

  • -> warp them to that fixed square size
  • -> run each of them through a convolutional network.
  • -> classification

• problem: It still computationally pretty expensive. because if we've got 2000 region proposals, and running them independently.

  • Training is slow, takes a lot of disk space.
  • Inference (detection) is slow.

Fast R-CNN

• Rather than processing each region of interest separately, we're going to run the entire image through some convolutional layers all at once to give a high-resolution convolutional feature map corresponding to the entire image.
• still using some region proposal like a fixed version of Selective Search.
• However, rather than cropping out the pixels of the image corresponding to the region proposals, instead, we imagine projecting those region proposals on to the convolutional feature map.
• allow reusing a lot of expensive convolutional computation across the entire image.
• reshaping of those crop for "ROI pooling layer".
• get classification score at fully connected layer.
• on training
  • a multi-task loss
  • do backpropagation, it learn all jointly

R-CNN vs SPP vs Fast R-CNN

• fast R-CNN 10 times faster than R-CNN to train
• bacause of it sharing all the computation between different feature maps.
• test time of fast R-CNN is super fast.
• Fast R-CNN problem: Runtime dominated by region proposals! so it ends up being bottlenecked by just computing of region proposals.

Faster R-CNN

• The idea of Faster R-CNN is to make just predict its own region proposals itself.

• ROI pooling layer

R-CNN vs SPP vs Fast R-CNN vs Faster R-CNN