search

MKLab-ITI / multimedia-indexing

A framework for large-scale feature extraction, indexing and retrieval.
Apache License 2.0
59 stars 19 forks source link

The training and testing (loading and using) of a model should be in the same class or docs should be added #7

Open futurely opened 9 years ago

futurely commented 9 years ago

The classes in the gr.iti.mklab.visual.quantization package train the various models used by the classes in the gr.iti.mklab.visual.datastructures package. It's hard to construct the data pipeline of training and testing the same complete search structure.

lefman commented 9 years ago

I think that it's better to distinguish classes that implement data structures from classes that train the quantizers used from this data structures. That said, adding documentation and/or a class that shows how the complete pipeline (of training the quantizers and using them in a data structure) can be constructed would indeed be very helpful.

futurely commented 9 years ago

CoarseQuantizerLearning and CodebookLearning are essentially the same except for the input data format. They can be unified.

futurely commented 9 years ago

I've successfully built a small IVFPQ index and the image retrieval results were not bad although I didn't benchmark and tune the parameters. The complete steps are as follows.

  1. Downloading images (using my own Apache HttpComponents based multi-threaded downloader)
  2. Run gr.iti.mklab.visual.examples.FeatureExtraction to extract local features. A few file path issues have to be fixed. SIFT and RootSIFT failed to detect key points in some of my images while SURF worked well.
  3. Optionally sample the local features by running gr.iti.mklab.visual.quantization.SampleLocalFeatures. 4. Copy the local features of each image into a single file with cat feature_dir/*.txt > somewhere_else/features.csv or cat feature_dir/*.csv > somewhere_else/features.csv.
  4. Run gr.iti.mklab.visual.quantization.CodebookLearning to learn a local features codebook.
  5. Run gr.iti.mklab.visual.examples.FolderIndexingMT to extract VLAD features of the images downloaded in step 1 using the codebook learned in step 5 and store them in a linear index.
  6. Run gr.iti.mklab.visual.quantization.CoarseQuantizerLearning to learn a coarse quantizer from the VLAD features extracted in step 6.
  7. Run gr.iti.mklab.visual.quantization.ProductQuantizationLearning to learn a product quantizer from the VLAD features extracted in step 6 and the coarse quantizer learned in step 7.
  8. Run gr.iti.mklab.visual.examples.IndexTransformation to transform the linear index of the VLAD features extracted in step 6 into an IVFPQ index using the product quantizer learned in step 8 and the coarse quantizer learned in step 7.
  9. Run the image retrieval example gr.iti.mklab.visual.examples.Example using the linear index (step 6), IVFPQ index (step 9), coarse quantizer (step 7) and product quantizer (step 8) to get the most similar images of each image in the linear index. I adapted the example to generate the HTML pages each of which visualized a query image and its retrieval results.
lefman commented 9 years ago

The pipeline seems correct.