search

embeddings-benchmark / mteb

MTEB: Massive Text Embedding Benchmark
https://arxiv.org/abs/2210.07316
Apache License 2.0
1.98k stars 277 forks source link

Avoid using global seeds #942

Open KennethEnevoldsen opened 5 months ago

KennethEnevoldsen commented 5 months ago

Currently, we use a global seed (in fact, multiple, which might invalidate each other). This is problematic as computation happening outside the software might also set a global seed, which will then invalidate benchmark results). A solution instead is to define a random_state or pass around an RNG object.

see e.g. https://docs.astral.sh/ruff/rules/numpy-legacy-random/

This is not a pressing issue; the current performance metric aligns with the third decimal (as far as I am aware and can test)

KennethEnevoldsen commented 2 months ago

A documented case where this causes problems: https://github.com/embeddings-benchmark/results/pull/19#discussion_r1741573217