Learning Two-Branch Neural Networks for Image-Text Matching Tasks

[chullhwan-song]

https://arxiv.org/abs/1704.03470 참고) https://github.com/chullhwan-song/Reading-Paper/issues/26 이논문 이후의 연구임.

[chullhwan-song]

What ?

• 따라서, https://github.com/chullhwan-song/Reading-Paper/issues/26 이 base
• ROI(localization)관련 정보를 이용.

input 형태 - phrase localization

embedding network

• 이 https://github.com/chullhwan-song/Reading-Paper/issues/26 연구에서 살펴봄.

  Similarity network의 소개

  

  • embedding network와 거의 유사
    • element-wise product = two branch를 합침
    • 이후, fully connected layers
    • loss : logistic regression loss
      • positive pair = +1 , negative pair는 = -1 의형태로 주어 학습
  • two tasks
• phrase localization = given a phrase, find a corresponding bounding box in the image
• bi-directional image-sentence search = given a query image/sentence, retrieve matching sentences/images from a database

  related works

• CCA
• CCA+deep
• Ranking-based methods : DeVISE
  • DeVISE가 ranking 기반 방법으로 분류되는지 이해못하고 있음. > metric learning이라고만 이해하고 있었음.ㅠ > 자세히 살펴볼요가 있음. > 다음은 이 문제의 식(=hinge rank loss)..
    • ranking이란 의미가 정확히?? > 힌트 "applies a margin-based penalty to an incorrect annotation when it gets ranked higher than a correct one for describing an image."
      • 정확한 라벨 score보다 부정확한 라벨 스코어가 더 높은(ranked higher) 경우 penalty를 주는..case
    • "single-directional ranking loss," > metric learning과 구분하기 위해 억지로..한느낌인데.. false인경우에만 penalty , 양방양은 true일때도..-penalty를 주는 모양새인듯~
      • Metric learning and Siamese networks.
  • "bi-directional ranking loss"

    학습

    • Preserving Neighborhood Structure관련 triplet loss : https://github.com/chullhwan-song/Reading-Paper/issues/26 연구 참조
    • Similarity Network
• 이 연구는 기존 연구에 이 network를 덧붙힌 연구 > 복잡하게 triplet loss를 사용하지 말고 그냥 간단히 match, no-match로 구분하는 loss만으로도 이전 연구보다 좋은(비슷한)성능을 보임
• The network then merges the output of the two branches using element-wise product, followed by a series of FC and ReLU layers.
  • others merge : concatenation, bilinear pooling, compact bilinear pooling 해봤지만, element-wise product 가 가장 좋았다.
• given input pair (xi, yj) , generate a score pij
  • to match the correct ground truth label (+1 for positive pairs and -1 for negative pairs).
• so, object function = using logistic regression : 일종의 sigmoid 함수
  • given the samples {xi, yj, zij}, xi==zij > +1, else -1
    • SGD
• need positive and negative image-text pairs > 여기서는 triplet 샘플링방법보다..더 효과적이라고 말하고 있음.
  • equal number of positives and negatives in every mini-batch.

실험

• 실험 조건
  • Network Architecture : Linear branch structure vs Nonlinear branch structure
  • Selecting Positive Pairs : Single positive vs Augmented positive(IOU>0.7)
  • Embedding Loss Functions : Single-directional(Devise)vs Bi-directional(pure triplet loss)
  • Neighborhood Sampling and Constraints : Neighborhood sampling vs Neighborhood constraints( 3 = 0.1, =0.1)
• detail
  • image feature > vgg + faster rcnn > roi 4096 feature
  • texture feature > https://github.com/chullhwan-song/Reading-Paper/issues/26 동일
    • https://github.com/chullhwan-song/Reading-Paper/issues/26 에서 말했듯이, tfxidf feature로도 비슷한 결과
  • branch nets = 4096>1024>512(image), 6000>1024>512(text) : two fc > https://oss.navercorp.com/vl/NextImageSearch_Study/issues/26 동일
  • similarity nets = 512x512(elements wise)> 256 (fc) > 1 (logistic regression)
• image to sentence : Flickr30K Enitity
• image to sentence : Flickr30K
• sentence to sentence
• combined image-sentence & region phrase
  • d(x,y) image-senetence간의 distance
  • d_rp(x,Y) rois - phrase간의 distance인데 이들에 대한 평균 distance
  • 알파 = 0.2