search

naver / splade

SPLADE: sparse neural search (SIGIR21, SIGIR22)
Other
751 stars 84 forks source link

Running SPLADE in production (Render python server) #37

Closed arainey2022 closed 1 year ago

arainey2022 commented 1 year ago

Hi team, I'm working with my development team on using SPLADE for sparse embeddings (alongside dense embeddings from OpenAI Ada), with the end goal of having a hybrid search setup.

However, we keep running into memory issues creating embeddings for chunks of text.

I was wondering if you have any tips for running this in production or better still if there is an API available that you're aware of that can take text as an input and output spare embeddings?

Any help would be hugely appreciated.

thibault-formal commented 1 year ago

hi @arainey2022 ,

We are currently discussing the release of an API for SPLADE. Regarding your question, it might just be that your batch size is too large (SPLADE embeddings take a bit more memory than dense embeddings due to the large output space).

arainey2022 commented 1 year ago

Thank you very much for this!

I would love to be your first API customer! We'd be very interested in a service like this. I don't suppose you have any timelines for this?

Is there a batch size you'd recommend?

And have you seen cases where someone has SPLADE running on a Render/Heroku type of setup?

thibault-formal commented 1 year ago

I would love to be your first API customer! We'd be very interested in a service like this. I don't suppose you have any timelines for this?

Nice to hear! Unfortunately, no timeline yet!

Is there a batch size you'd recommend?

The batch size really depends on your hardware.. on a single GPU, it should be easy to have a few hundred documents per batch.

And have you seen cases where someone has SPLADE running on a Render/Heroku type of setup?

Not that I am aware of.

Sayed-Ameer commented 1 year ago

You can try https://github.com/pinecone-io/pinecone-text/#splade but you have to figure out how to build an index for the output. I just found out about this repo but i would really like if someone can build a searchable index that accept indices, values.

arainey2022 commented 1 year ago

Thanks Sayed. We're actually trying this library, but having trouble keeping the server from crashing.

thibault-formal commented 1 year ago

I am closing this issue, feel free to re-open it if needed!