Scaling the search capability -PoC with Elastic search

arampacha / CLIP-rsicd

Apache License 2.0

208 stars 31 forks source link

Scaling the search capability -PoC with Elastic search #34

Closed INF800 closed 2 years ago

INF800 commented 2 years ago

Hi, I am planning to build a (scalable) vector search engine using this model. I need to understand couple of things for it. Will you guys be helping me?

These are the current challenges I face

Not able to find downloadable gdrive/MyDrive/RSICD/RSICD_images.rar. Drive link in RSICD_optimal is prompting "request access"

INF800 commented 2 years ago

Hmm, found this link from somewhere https://mega.nz/folder/wCpSzSoS#RXzIlrv--TDt3ENZdKN8JA

sujitpal commented 2 years ago

Hi @INF800 hope this is enough information to start:

you can download the images and captions directly from the RSICD repository (https://github.com/201528014227051/RSICD_optimal) -- the link is also available from the README.md file for this project.
once done, you can download one of our models (again link and example code available on README.md) and generate the vectors from the image-caption pairs.
at this point I suspect you would be ready to upload them to your Elasticsearch instance. We used NMSLib for the demo (code and demo links available in README.md) but in another variation we used the Vespa engine, which offers text, vector and hybrid (text+vector) searching.

INF800 commented 2 years ago

I am stuck at the first step currently. (Because I am also looking at the training code which I will use in future)

These two files are actually downloadable

The new download source of RSICD-MEGA
dataset.json (2.8MB)
imgs.rar (252.9MB)
The new download source of Sydney-captions and UCM-catpions-MEGA.
annotations_rsicd.rar (836KB)
RSICD_images.rar (459.8MB)

The first file ( 252MB) is subset of second file (459MB) right? Or are they both different?
And are we using only second file (459MB) for training the best model(s) ?

INF800 commented 2 years ago

And also, can I know what are we using for searching in HF spaces? The speed is appreciable.

sujitpal commented 2 years ago

We didn't use the MEGA data these appear to have been added after we were done with the project. Not sure if one is a subset of the other. If you are going to retrain with more data, then might make sense to download both these archives and check.

And as I mentioned earlier, we are using NMSLib for searching in the HF-spaces demo.

nikky4D commented 2 years ago

@sujitpal for the data used, does the RSICD data also contain Sydney and UCM-Merced data? If not, did you use all three in your training or only the RSICD data?

sujitpal commented 2 years ago

For the training I believe we used RSICD plus the Sydney and UCM-Merced datasets, but I will defer to @arampacha for the authoritative answer.

arampacha commented 2 years ago

IIRC the data is not overlapping and we used all 3 datasets. This is also consistent with model cards we have. And don't see a reason not to trust those ;)