configuration for splade++ results

[seanmacavaney]

Hi-- thanks for the nice work.

I'm trying to index+retrieve using the naver/splade-cocondenser-ensembledistil model. Following the readme, I've done:

export SPLADE_CONFIG_FULLPATH="config_default.yaml"
python3 -m src.index \
  init_dict.model_type_or_dir=naver/splade-cocondenser-ensembledistil \ # <--- (from readme, using the new model)
  config.pretrained_no_yamlconfig=true \
  config.index_dir=experiments/pre-trained/index \
  index=msmarco  # <--- added

export SPLADE_CONFIG_FULLPATH="config_default.yaml"
python3 -m src.retrieve \
  init_dict.model_type_or_dir=naver/splade-cocondenser-ensembledistil \ # <--- (from readme, using the new model)
  config.pretrained_no_yamlconfig=true \
  config.index_dir=experiments/pre-trained/index \
  config.out_dir=experiments/pre-trained/out-dl19 \
  index=msmarco \  # <--- added
  retrieve_evaluate=msmarco # <--- added

Everything runs just fine, but I'm getting rather poor results in the end:

MRR@10: 0.18084248646927734
recall ==> {'recall_5': 0.2665353390639923, 'recall_10': 0.3298710601719197, 'recall_15': 0.3694364851957974, 'recall_20': 0.3951050620821394, 'recall_30': 0.4270654250238777, 'recall_100': 0.5166069723018146, 'recall_200': 0.5560768863419291, 'recall_500': 0.606984240687679, 'recall_1000': 0.6402578796561604}

I suspect it's a configuration problem on my end, but since the indexing process takes a bit of time, I thought I'd just ask before diving too far into the weeds: Is there a configuration file to use for the splade++ results, and how do I use it?

Thanks!

[thibault-formal]

Hi @seanmacavaney, great to see you trying our code! You are right, I made a mistake; it shouldn't be config_default.yaml, because it initializes parameters for the toy/debug config (especially max_length: 10 which should explain your low results).

Simply use:

export SPLADE_CONFIG_NAME="config_splade++_cocondenser_ensembledistil"

and this should work!

[seanmacavaney]

Thanks -- that did the trick!