ioannist
commented
4 years ago I was able to get passed this error by using my own RagRetriever instead of RagPyTorchDistributedRetriever inside "transformers/examples/rag/finetune.py"
I am also using my own custom dataset and index #7763
The following changes got me past the missing index error. However, I have no idea if this is efficient or if I am doing something that I shouldn't be doing...
if self.is_rag_model:
if args.prefix is not None:
config.generator.prefix = args.prefix
config.label_smoothing = hparams.label_smoothing
hparams, config.generator = set_extra_model_params(extra_model_params, hparams, config.generator)
# commented out this line
# retriever = RagPyTorchDistributedRetriever.from_pretrained(hparams.model_name_or_path, config=config)
############### new stuff ###############
dataset = load_from_disk(args.passages_path) # to reload the dataset
dataset.load_faiss_index("embeddings", args.index_path) # to reload the index
retriever = RagRetriever.from_pretrained(
hparams.model_name_or_path, index_name="custom", indexed_dataset=dataset
)
######################################
model = self.model_class.from_pretrained(hparams.model_name_or_path, config=config, retriever=retriever)
prefix = config.question_encoder.prefix
patrickvonplaten
lhoestq
amogkam
shamanez
Environment info
transformers version: 3.3.1 Platform: Ubuntu Python version:3.6.12 PyTorch version (GPU: yes): 1.6.0 Using GPU in script?: 1 gpu Using distributed or parallel set-up in script?: no
Who can help
@patrickvonplaten @sgugger
Information
model name: facebook/rag-sequence-base
The problem arises when using the official example scripts: (give details below) The tasks I am working on is my own task or dataset: (give details below)
To reproduce
1) Make directory at examples/rag/ioannis-data and add train eval and test files in the directory 2) Run transformers/examples/rag/finetune.sh with following changes: --data_dir examples/rag/ioannis-data \ --output_dir examples/rag/ioannis-output \ --model_name_or_path facebook/rag-sequence-base
The script terminates with the following error:
datasets.search.MissingIndex: Index with index_name 'embeddings' not initialized yet. Please make sure that you call 'add_faiss_index' or 'add_elasticsearch_index' first.