Cannot decide self.index type

[loki-zhou]

error log | 日志或报错信息 | ログ

08/12/2024 15:30:44 - [INFO] -faiss.loader->>>    Loading faiss with AVX2 support.
08/12/2024 15:30:44 - [INFO] -faiss.loader->>>    Could not load library with AVX2 support due to:
ModuleNotFoundError("No module named 'faiss.swigfaiss_avx2'")
08/12/2024 15:30:44 - [INFO] -faiss.loader->>>    Loading faiss.
08/12/2024 15:30:44 - [INFO] -faiss.loader->>>    Successfully loaded faiss.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3369/3369 [01:12<00:00, 46.57it/s]
2024-08-12 15:31:56.763 | INFO     | huixiangdou.primitive.file_operation:summarize:143 - 累计555文件，成功438个，跳过0个，异常117个
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/HuixiangDou/huixiangdou/service/feature_store.py", line 384, in <module>
    retriever = cache.get(config_path=args.config_path, work_dir=args.work_dir)
  File "/root/HuixiangDou/huixiangdou/service/retriever.py", line 273, in get
    retriever = Retriever(config_path=config_path,
  File "/root/HuixiangDou/huixiangdou/service/retriever.py", line 48, in __init__
    self.faiss = Faiss.load_local(dense_path)
  File "/root/HuixiangDou/huixiangdou/primitive/faiss.py", line 176, in load_local
    raise ValueError('Cannot decide self.index type {}, open https://github.com/InternLM/HuixiangDou/issues/346 for hotfix'.format(type(index)))
ValueError: Cannot decide self.index type <class 'faiss.swigfaiss.IndexFlat'>, open https://github.com/InternLM/HuixiangDou/issues/346 for hotfix

Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
>>> index = faiss.read_index('embedding.faiss')
>>> print(type(index))
<class 'faiss.swigfaiss.IndexFlat'>

context | 编译/运行环境 | バックグラウンド

microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Python 3.10.12

how to reproduce | 复现步骤 | 再現方法

1. 2. 3.

more | 其他 | その他

embedding.faiss.zip

[tpoisonooo]

352 直接把 distance 方法存到 pkl， 不再按 class name 选。