FLOPs calculation

[namespace-Pt]

I recently read your SPLADE paper and I think it's quite interesting. I have a question concerning FLOPs calculation in the paper.

I think computing FLOPs for an inverted index involves the length of the activated posting lists(the overlapping terms in query and document). For example, a query a b c and a document c a e, since we must inspect the posting list of the overlapping terms a and c, the flops should at least be

posting_length(a) + posting_length(c)

because we perform summation for each entry in the posting list. However, in the paper you compute FLOPs by the probability that a, b, c are activated in the query and c, a, e are activated in the document. I think this may underestimate the flops of SPLADE because the less sparse the document, the longer posting lists in the inverted index.

[thibault-formal]

hi @namespace-Pt Sorry for the late answer!

I am not sure I completely got your point. For the FLOPS estimation, we rely on the derivation from Minimizing FLOPs to Learn Efficient Sparse Representations. The probabilities are directly estimated from the length of the posting lists (for both documents and queries, where for the later we simply "index" them).

let us know if you need more details, Thibault

[namespace-Pt]

Thank you @thibault-formal! I'll read the paper.

BTW, do you have a checkpoint of SPLADEv2-distill? I tried to reproduce your result but failed. I found that using the distillation we donot need a ground-truth passage to each query, so did you use the queries that are absent from qrels.train.tsv to train the student model?

[namespace-Pt]

I would appreciate it if you could also provide the SPLADEv2-max (MRR@10=0.34) checkpoint. I want to see how the model learns to distribute the tokens.

[sclincha]

Hi @namespace-Pt ,

• We have some models available here: https://europe.naverlabs.com/research/machine-learning-and-optimization/splade-models/. New models will come soon, the SPLADEv2-max is not there yet but we can add it even if we have better models than this one trained without distillation.
• For your original question on the FLOPS, I understood that you were asking whether we should measure the FLOPS (Q*D) versus FLOPS(Q) + FLOPS(D). We have tried that and it did not change much the results. Sometimes, it is better to adjust differently the query and document representation.

[thibault-formal]

@sclincha @namespace-Pt the weights for SPLADEv2-max can actually be found in the weights folder in this repo.

[namespace-Pt]

@sclincha @thibault-formal Thank you! I'll check it out.