Generation error with vec2text_uw.yaml

[ondovb]

On MacOS 14.5, Python 3.11.10:

(map2text) MWC36G27YR6Y:map2text bdo8$ python map2text/quick_start.py map2text/configs/cs_research_idea/vec2text_uw.yaml 
Loading KNN index from cache.
KNN index initialized.
Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead.
Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead.
  0%|                                                                                                                   | 0/200 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/vec2text/trainers/corrector.py", line 252, in generate
    hypothesis_input_ids = inputs["hypothesis_input_ids"]
                           ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'hypothesis_input_ids'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bdo8/Documents/Projects/LitVectors/map2text/map2text/quick_start.py", line 280, in <module>
    main()
  File "/Users/bdo8/Documents/Projects/LitVectors/map2text/map2text/quick_start.py", line 274, in main
    GenerationExperiment.from_config(config).run()
  File "/Users/bdo8/Documents/Projects/LitVectors/map2text/map2text/quick_start.py", line 155, in run
    results = self.generator.decode_all(queries)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdo8/Documents/Projects/LitVectors/map2text/map2text/model/base.py", line 137, in decode_all
    return [self.decode(encoding) for encoding in tqdm.tqdm(low_dim_embeddings)]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdo8/Documents/Projects/LitVectors/map2text/map2text/model/base.py", line 137, in <listcomp>
    return [self.decode(encoding) for encoding in tqdm.tqdm(low_dim_embeddings)]
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdo8/Documents/Projects/LitVectors/map2text/map2text/model/non_trainable_gen.py", line 107, in decode
    return vec2text.invert_embeddings(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/vec2text/api.py", line 106, in invert_embeddings
    regenerated = corrector.generate(
                  ^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/vec2text/trainers/corrector.py", line 261, in generate
    ) = self._get_hypothesis_uncached(inputs=inputs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/vec2text/trainers/corrector.py", line 615, in _get_hypothesis_uncached
    hypothesis_input_ids = self.inversion_trainer.model.generate(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/vec2text/models/inversion.py", line 284, in generate
    return self.encoder_decoder.generate(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/transformers/generation/utils.py", line 1993, in generate
    self._prepare_special_tokens(generation_config, kwargs_has_attention_mask, device=device)
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/transformers/generation/utils.py", line 1842, in _prepare_special_tokens
    and isin_mps_friendly(elements=eos_token_tensor, test_elements=pad_token_tensor).any()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/map2text/lib/python3.11/site-packages/transformers/pytorch_utils.py", line 325, in isin_mps_friendly
    return elements.tile(test_elements.shape[0], 1).eq(test_elements.unsqueeze(1)).sum(dim=0).bool().squeeze()
                         ~~~~~~~~~~~~~~~~~~~^^^
IndexError: tuple index out of range
  0%|                                                                                                                   | 0/200 [00:00<?, ?it/s]

[xingjian-zhang]

This seems to be an issue from vec2text but I failed to reproduce it. Did you set up the env variable OPENAI_API_KEY?

Reference: https://github.com/jxmorris12/vec2text/issues/54

[xingjian-zhang]

Could you please run the demo of vec2text and see if the error still exists?