Bags of Local Convolutional Features for Scalable Instance Search

Abstract
- cnn feature를 bag of word 형태로 만드는 방법
- assignment map
  - fast spatial re-ranking
  - obtaining object localizations that are used for query expansion
- We demonstrate the suitability of the BoW representation based on local CNN features for instance retrieval
BAG OF WORDS FRAMEWORK
- pre-trained CNN as local feature
- discard fc & softmax - conv feature - keeping aspect ratio
- 1 channel -> a vector -> Bag of Word -> Inverted indexed
- Bag of word
  - using K-means
  - making assignment map
IMAGE RETRIEVAL : a spatial re-ranking, and query expansion.
- initial search - 후보군 서치
  - cosine similarity between the BoW vector of the query image and the BoW vectors of the full images in the database.
    - use a sparse matrix based inverted index, GPU..
  - two types - bounding box의 유무
    - Global search (GS)
    - Local search (LS)
  - Local Re-Ranking
  - Top T images 가 대상
  - localization score 기반으로 re-ranking
    - re-ranking을 위한 ROI 구성
      - w= { W, W/2, W/4 } , h = {H, H/2, W/4} : W, H of assignment map 기준으로,
      - sliding window strategy으로 가능한 모든 조합의 크기로 구성
      - assignment map 과 overlap 비율이 50%넘어야함.
      - query의 aspect ratio와 비슷한 것만 선택
    - 구성된 ROI에 대해
      - BoW vector representation 구성하고
      - query's BoW vector representation cosine similarity
      - BoW vector representation
        
        spatial pyramid matching
        
        vector 값에 대한 loss 감쇠, locality에 대한 감쇠
        
        level = 2
  - Query Expansion (?)
    - Global query expansion (GQE): The BoW vectors of the N images at the top of the ranking are averaged together with the BoW of the query to form the new representation for the query.
    - Local query expansion (LQE): Locations obtained in the local reranking step are used to mask out the background and build the BoW descriptor of only the region of interest of the N-top images in the ranking. These BoW vectors are averaged together with the BoW of the query bounding box to perform a second search.
    - 다른 논문 읽다가 비슷한 용어가 나오길래.. 다름 아닌 MAC 논문임.
      - "Query expansion (QE). Re-ranking brings positive images at the very top ranked positions. Then, we collect the 5 top-ranked images, merge them with the query vector, and compute their mean. Finally, the similarity to this mean vector is adopted to re-rank once more the top N images."
    - 즉, 일반 결과중에 상위 결과 Top-K 결과가 어느정도 타당성을 가지고 있다면, query+Top-K vector들 평균값을 취해, 새로운 query vector로 사용한다는 의미이다.
    - 참조 paper : "PARTICULAR OBJECT RETRIEVAL WITH INTEGRAL MAX-POOLING OF CNN ACTIVATIONS"
    - 이외 query expansion 알고리즘을 다양하게 존재 - ex) manifold learning
Experiments
- instance dataset - landmark data : oxford building, paris building,,,
- caffe - vgg16: conv5_1, conv5_2, conv5_3 > best : conv5_1
- PCA - whitening
- dictionary - 25K

chullhwan-song / Reading-Paper

Bags of Local Convolutional Features for Scalable Instance Search #9