cadurosar
commented
1 year ago I'm very impressed by SPLADE, particularly the newest efficient versions. However, it is only trained on English texts.
Hi @nickchomey thanks for your words and your interest in SPLADE! Actually, we have some initial results that should hit arxiv this week, as we used a multilingual version of SPLADE in the WSDM CUP 23 - MIRACL competition. But is still early work and it should take us a while to have a model that we can share on huggingface.
There's an mMARCO dataset that has 14 languages, which is already in use by SBERT and other projects. Importantly, there's a doc2query mt5 model that uses this dataset. It seems to me that anyone using non-english (or multiple) languages would have no choice but to use this.
So, this is something that we have been looking into for a while, and as I spoiled have done a little bit for a competition, but not on our languages of mMARCO, but around the same amount (15 of MIRACL). There are two main problems:
1) getting a good model initialized in a multilingual fashion that works with SPLADE. Some people have tried to use out-of-the-shelf pretrained multilingual models with mixed results (for example https://desires.dei.unipd.it/2022/papers/paper-06.pdf most notably look into Figure 2). As we have seen in the efficiency study for SPLADE, a good initialization is paramount, but pretraining multi-lingual models is quite costly.
2) SPLADE training and inference cost increases a lot with vocabulary size. For a single language, a tokenizer of around 30k might suffice (which is the size we use for English), but going into multilingual we need to increase vocabulary size. Going for 120k vocab size (mbert) makes it a lot more costly, reducing the max length we can go and/or requiring larger gpus. If we try 240k vocab (infoxlm-roberta) than it is even harder
A SPLADE version would be fantastic, especially if compared to the mT5 version of doc2query on BEIR zero-shot data!
I'm sorry but I'm not sure to understand here. I can't see how to compare multilingual models on BEIR, as that would only judge the models capabilities in English which could skew the results.
Even better would be if you could somehow use the FLORES-200 dataset, which is used by the cutting edge NLLB-200 translation model!
On the subject of FLORES and NLLB, while we think this is still far in the future, using NLLB to translate from any language to English and then using the SPLADE++ models is a proxy that we know to work well. For example in TREC-neuCLIR this is what worked the best out of the different approaches we tested.
Would you consider implementing a multilingual version in a future iteration of SPLADE? I think this would provide immense value to the global community!
Also, its not clear to me that the SPLADE++ methods were used as part of your efficient version. So, it would be great if you could use and compare it with the other methods.
Yes, it is kinda unclear because we submitted both at the same time, so we had to write them as if the other one did not exist. I will take this chance to write an explanation for this, and I can maybe point people here when I see this question. What we found is that there is a trade-off between them. SPLADE++ was the best that we could (effectiveness-wise) do at the moment with out-of-the-shelf LMs at the moment we submitted, while the efficient trades some of that effectiveness (mostly out-of-domain effectiveness) for efficiency. We still haven't found a way to have the best of both worlds.
The efficiency paper kinda starts from SPLADE++. Actually, the baseline we use there ("improvements i) and ii)") are basically SPLADE++ ensemble distil, but taking the models with fewer flops from Figure 1 in https://arxiv.org/pdf/2205.04733.pdf.
The efficiency paper than introduces 4 actual novel improvements to SPLADE. Out of those 4, 3 actually makes it harder for working on out-of-domain data, especially BEIR. Each of the improvements has a different reason for worsening the BEIR score:
1) Improvement 3) suggests separating the document and query encoder. However, some of the datasets on BEIR, such as QUORA, have no separation between what is a document and what is a query. Thus, separating the concepts and using the "wrong" concept encoder lessens the effectiveness on those datasets
2) Improvement 5) suggests doing MLM+FLOPS middle-training on MSMARCO. While the MLM+FLOPS training helps a lot in preparing the model for SPLADE, it further specializes in the model on MSMARCO, which makes it harder for the model to generalize to BEIR.
3) Improvement 6) suggest changing the query encoder to a more efficient one. This not only increases the problem we had seen on Improvement 3), but it also creates a difference in concepts between the document and query encoder. While this does not affect much on MSMARCO, it makes it so that less used words/unseen concepts point to different parts of the latent space
Note that we would have loved to have that on the papers, but the main problems were 1) they were submitted at the same time and thus could not cite each other; 2) At SIGIR22 full paper submission time, it was not clear how to make them a single study (and they were not ready by then, the extra time of the short paper really helped); 3) as we were constrained by the 4 pages of the short paper it was hard to find the place to add a detailed explanation such as this one
nickchomey
thibault-formal
I'm very impressed by SPLADE, particularly the newest efficient versions. However, it is only trained on English texts.
There's an mMARCO dataset that has 14 languages, which is already in use by SBERT and other projects. Importantly, there's a doc2query mt5 model that uses this dataset. It seems to me that anyone using non-english (or multiple) languages would have no choice but to use this. A SPLADE version would be fantastic, especially if compared to the mT5 version of doc2query on BEIR zero-shot data!
Even better would be if you could somehow use the FLORES-200 dataset, which is used by the cutting edge NLLB-200 translation model!
Would you consider implementing a multilingual version in a future iteration of SPLADE? I think this would provide immense value to the global community!
Also, its not clear to me that the SPLADE++ methods were used as part of your efficient version. So, it would be great if you could use and compare it with the other methods.