search

AmenRa / retriv

A Python Search Engine for Humans 🥸
MIT License
174 stars 20 forks source link

fsspec==2023.12.2 does not allow '**' in path #32

Open jacobvsdanniel opened 8 months ago

jacobvsdanniel commented 8 months ago 
  File "/lib/python3.8/site-packages/retriv/dense_retriever/dense_retriever.py", line 175, in index
    self.index_aux(
  File "/lib/python3.8/site-packages/retriv/dense_retriever/dense_retriever.py", line 137, in index_aux
    self.ann_searcher.build()
  File "/lib/python3.8/site-packages/retriv/dense_retriever/ann_searcher.py", line 27, in build
    index, index_infos = build_index(
  File "/lib/python3.8/site-packages/autofaiss/external/quantize.py", line 205, in build_index
    embedding_reader = EmbeddingReader(
  File "/lib/python3.8/site-packages/embedding_reader/embedding_reader.py", line 20, in __init__
    self.reader = NumpyReader(embeddings_folder)
  File "/lib/python3.8/site-packages/embedding_reader/numpy_reader.py", line 67, in __init__
    self.fs, embeddings_file_paths = get_file_list(embeddings_folder, "npy")
  File "/lib/python3.8/site-packages/embedding_reader/get_file_list.py", line 15, in get_file_list
    return _get_file_list(path, file_format)
  File "/lib/python3.8/site-packages/embedding_reader/get_file_list.py", line 46, in _get_file_list
    file_paths = fs.glob(glob_pattern)
  File "/lib/python3.8/site-packages/fsspec/spec.py", line 606, in glob
    pattern = glob_translate(path + ("/" if ends_with_sep else ""))
  File "/lib/python3.8/site-packages/fsspec/utils.py", line 734, in glob_translate
    raise ValueError(
ValueError: Invalid pattern: '**' can only be an entire path component

this error does not occur with fsspec==2023.5.0