erikbern / ann-benchmarks

Benchmarks of approximate nearest neighbor libraries in Python
http://ann-benchmarks.com
MIT License
4.73k stars 715 forks source link

Skip loading base/config.yml on Windows. #497

Closed koron closed 3 months ago

koron commented 3 months ago

load_configs() failed always because of ann_benchmarks/algorithms/base/config.yaml is empty. yaml.safe_load() returns None for empty steam (https://pyyaml.org/wiki/PyYAMLDocumentation).

This fixes it with skipping None which returned from yaml.safe_load()

maumueller commented 3 months ago

@koron I'm not sure why this should have happened on your machine, we explicitly skip the base/config.yml in get_config_files(...). Could you check whether you made any additional changes that led to this behavior?

https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/definitions.py#L126-L131

koron commented 3 months ago

I haven't confirmed it properly, but I'm using Windows and the directory separator is \, so - {f"{base_dir}/base/config.yml"} may not be able to exclude it.

koron commented 3 months ago

I tried instantly.

So it should be - {os.path.join(base_dir, "base", "config.yml")}

>>> import os, glob
>>> files = glob.glob(os.path.join("ann_benchmarks/algorithms", "*", "config.yml"))
>>> set(files) - {f"ann_benchmarks/algorithms/base/config.yml"}
{
    'ann_benchmarks/algorithms\\balltree\\config.yml',
    'ann_benchmarks/algorithms\\hnswlib\\config.yml',
    'ann_benchmarks/algorithms\\mrpt\\config.yml',
    'ann_benchmarks/algorithms\\faiss_gpu\\config.yml',
    'ann_benchmarks/algorithms\\dummy_algo\\config.yml',
    'ann_benchmarks/algorithms\\tinyknn\\config.yml',
    'ann_benchmarks/algorithms\\qsg_ngt\\config.yml',
    'ann_benchmarks/algorithms\\qdrant\\config.yml',
    'ann_benchmarks/algorithms\\pg_embedding\\config.yml',
    'ann_benchmarks/algorithms\\n2\\config.yml',
    'ann_benchmarks/algorithms\\onng_ngt\\config.yml',
    'ann_benchmarks/algorithms\\subprocess\\config.yml',
    'ann_benchmarks/algorithms\\vearch\\config.yml',
    'ann_benchmarks/algorithms\\vald\\config.yml',
    'ann_benchmarks/algorithms\\opensearchknn\\config.yml',
    'ann_benchmarks/algorithms\\qg_ngt\\config.yml',
    'ann_benchmarks/algorithms\\luceneknn\\config.yml',
    'ann_benchmarks/algorithms\\glass\\config.yml',
    'ann_benchmarks/algorithms\\elasticsearch\\config.yml',
    'ann_benchmarks/algorithms\\bruteforce\\config.yml',
    'ann_benchmarks/algorithms\\kgraph\\config.yml',
    'ann_benchmarks/algorithms\\redisearch\\config.yml',
    'ann_benchmarks/algorithms\\dolphinnpy\\config.yml',
    'ann_benchmarks/algorithms\\nearpy\\config.yml',
    'ann_benchmarks/algorithms\\sptag\\config.yml',
    'ann_benchmarks/algorithms\\flann\\config.yml',
    'ann_benchmarks/algorithms\\nndescent\\config.yml',
    'ann_benchmarks/algorithms\\datasketch\\config.yml',
    'ann_benchmarks/algorithms\\faiss\\config.yml',
    'ann_benchmarks/algorithms\\rpforest\\config.yml',
    'ann_benchmarks/algorithms\\vespa\\config.yml',
    'ann_benchmarks/algorithms\\voyager\\config.yml',
    'ann_benchmarks/algorithms\\base\\config.yml',   # <== HERE WITH '\'!
    'ann_benchmarks/algorithms\\faiss_hnsw\\config.yml',
    'ann_benchmarks/algorithms\\elastiknn\\config.yml',
    'ann_benchmarks/algorithms\\pgvector\\config.yml',
    'ann_benchmarks/algorithms\\nmslib\\config.yml',
    'ann_benchmarks/algorithms\\panng_ngt\\config.yml',
    'ann_benchmarks/algorithms\\puffinn\\config.yml',
    'ann_benchmarks/algorithms\\pynndescent\\config.yml',
    'ann_benchmarks/algorithms\\kdtree\\config.yml',
    'ann_benchmarks/algorithms\\scann\\config.yml',
    'ann_benchmarks/algorithms\\milvus\\config.yml',
    'ann_benchmarks/algorithms\\diskann\\config.yml',
    'ann_benchmarks/algorithms\\annoy\\config.yml',
    'ann_benchmarks/algorithms\\ckdtree\\config.yml',
    'ann_benchmarks/algorithms\\weaviate\\config.yml'
}

(edited partially)
koron commented 3 months ago

Based on the suggestions and experimental results, I made the changes (added another commit).

If you prefer squash to one commit, let me know and I'll do it.

maumueller commented 3 months ago

Thanks, great fix!