How to run reverse_image_search?

[msharmavikram]

I am not sure how to run the reverse image search example and I am very interested in it!

I noticed a few things missing in the example and I tried to fix it but did not succeed in understanding how to actually run it.

1. parser --> readers.
2. I see some of the video paths are hardcoded. ahewhope.m4v (i downloaded a path myself and converted it take it as input)
3. faster_rcnn_coco.toml missing. ( i downloaded and succeeded in ensuring it takes in)
4. Missing faiss installation. (I was able to install it and resolve the dependency in this example)
5. Query image I added a screenshot of from the video. I am not sure if that works. Do you need dimension?

Even after all of this, I still did not understand how to host the server and perform the search. Kind of stuck. I was able to run the server but failed to load the data to the server. Appreciate if you can provide some information on how to use server - search scheme of running.

Regards Vikram.

[willcrichton]

Sorry, that application is from quite some time ago and has yet to be updated. I'm going to remove it from the repository entirely. However, you could still approximate it by looking at the TensorFlow examples (e.g. the one in scannertools), except extract the last layer feature vectors instead of the actual labels.

[msharmavikram]

Thanks. Reverse image search is a very good use case for Scanner. Although I am sad that you are removing it, I am wondering if there is any specific reason apart from updating the example to latest code base?

[willcrichton]

Mostly just that we're focusing more on video applications than image ones.

[msharmavikram]

Seems like I have misunderstood the example then.

I thought the example would allow searching across multiple videos with a frame of interest. For example, let's say, I have a movie screen snapshot and I want to figure out which movie it belongs to. I could perform a reverse image search on a large corpus of video database to find the specific movie. I anticipated this example would reflect such a scenario. (at least basic)

(Why I am interested in this? Sometimes I would like to know where the meme pictures are actually derived from! What season of tv series, movie or news, etc.)

[fpoms]

Hey @msharmavikram! I think you understood the application correctly: it was used to search for frames from a video. However, as @willcrichton pointed out, it hasn't been updated for the newest version of our API.

Unfortunately, we don't have the time to update it right now. Would you be interested in trying to get it working again? I can answer any questions you have while trying to revive the example and provide advice on what to replace any missing components with! I think the main parts that need to be updated our:

1. Update all the calls to Scanner to use the new API
2. Change from faster rcnn to a more modern model, such as from our tensorflow or detectron example

Let me know!

[msharmavikram]

Thanks for the response @apoms I would be happy to help fix this example. Currently, I am playing around with FAISS and once done, I will start working on this. (probably over the weekend).

I would definitely need some help on the APIs. I tried to hack a little bit to reach a stage where further modifications did not help. I also experimented what @willcrichton suggested using the scannertools but noted that it used older scanner base and I could not point to the reference in scanner to get the thing working. With help, I think I can revive it to become alive again!

A suggestion: Can we have tags to point out in scannertool to tell the base commit of scanner? Given that scanner is actively getting developed, it makes sense to have reference point across scannertool and scanner.

[willcrichton]

Note here: Scannertools has been updated with matching versions to Scanner.

[willcrichton]

@msharmavikram going to close this for now, but please reopen if you have an issue.