KeyError: 'samples'

[liushiliushi]

Could you please help me with this:

│   190 │   │                                                                                      │
│   191 │   │   print(f"self.dataset: {self.dataset}")                                             │
│   192 │   │   samples_batches = self.meta_chain.initial_chain.batch_invoke(batch_inputs, self.   │
│ ❱ 193 │   │   samples_list = [element for sublist in samples_batches for element in sublist['s   │
│   194 │   │   samples_list = self.dataset.remove_duplicates(samples_list)                        │
│   195 │   │   self.dataset.add(samples_list, 0)                                                  │
│   196                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'samples'

It seems that there isn't 'samples' in samples_llist

[Eladlev]

Hi, In order to provide better help, please provide all the config file changes and the running command. This error usually happens due to one of the following reasons

1. Using too weak LLM model (if you modify the default LLM and using GPT-3.5 as the meta-prompt LLM instead of GPT-4)
2. Your connection permission issues (key/organization, etc')

[liushiliushi]

Thanks for your response. I'm sure that the connection goes on well. It might be the problem of the modified config_default.yml

use_wandb: False
dataset:
    name: 'dataset'
    records_path: null
    initial_dataset: ''
    label_schema: ["Yes", "No"]
    max_samples: 50
    semantic_sampling: False # Change to True in case you don't have M1. Currently there is an issue with faiss and M1

annotator:
    # method : 'llm'
    # config:
    #     api_url: ''
    #     api_key: 'admin.apikey'
    #     workspace: 'admin'
    #     time_interval: 5
    method : 'llm'
    config:
        llm:
            type: 'azure'
            name: 'gpt-4-0125-preview'
#            async_params:
#                retry_interval: 10
#                max_retries: 2
            model_kwargs: {"seed": 220}
        num_workers: 5
        prompt: 'prompts/predictor_completion/prediction.prompt'
        mini_batch_size: 1  #change to >1 if you want to include multiple samples in the one prompt
        mode: 'annotation'

predictor:
    method : 'llm'
    config:
        llm:
            type: 'azure'
            name: 'gpt-4-0125-preview'
#            async_params:
#                retry_interval: 10
#                max_retries: 2
            model_kwargs: {"seed": 220}
        num_workers: 5
        prompt: 'prompts/predictor_completion/prediction.prompt'
        mini_batch_size: 1  #change to >1 if you want to include multiple samples in the one prompt
        mode: 'prediction'

meta_prompts:
    folder: 'prompts/meta_prompts_classification'
    num_err_prompt: 1  # Number of error examples per sample in the prompt generation
    num_err_samples: 2 # Number of error examples per sample in the sample generation
    history_length: 4 # Number of sample in the meta-prompt history
    num_generated_samples: 10 # Number of generated samples at each iteration
    num_initialize_samples: 10 # Number of generated samples at iteration 0, in zero-shot case
    samples_generation_batch: 10 # Number of samples generated in one call to the LLM
    num_workers: 5 #Number of parallel workers
    warmup: 4 # Number of warmup steps

eval:
    function_name: 'accuracy'
    num_large_errors: 4
    num_boundary_predictions : 0
    error_threshold: 0.5

llm:
    type: 'azure'
    name: 'gpt-4-1106-preview'
    temperature: 0.8

stop_criteria:
    max_usage: 2 #In $ in case of OpenAI models, otherwise number of tokens
    patience: 10 # Number of patience steps
    min_delta: 0.01 # Delta for the improvement definition

Thanks for your help!

[Eladlev]

When using an LLM as the annotator, you should also explicitly define the instructions for the annotator in the configuration file. See the following example: https://github.com/Eladlev/AutoPrompt/blob/main/docs/installation.md#configure-llm-annotator

[liushiliushi]

Thanks for you quick response! I've changed config_default.yml to

use_wandb: False
dataset:
    name: 'dataset'
    records_path: null
    initial_dataset: ''
    label_schema: ["Yes", "No"]
    max_samples: 50
    semantic_sampling: False # Change to True in case you don't have M1. Currently there is an issue with faiss and M1

annotator:
    method : 'llm'
    config:
        llm:
            type: 'azure'
            name: 'gpt-4-1106-preview'
        instruction:
            'Does this movie review contain a spoiler? answer Yes or No.'
        num_workers: 5
        prompt: 'prompts/predictor_completion/prediction.prompt'
        mini_batch_size: 1
        mode: 'annotation'

predictor:
    method : 'llm'
    config:
        llm:
            type: 'azure'
            name: 'gpt-4-1106-preview'
#            async_params:
#                retry_interval: 10
#                max_retries: 2
            model_kwargs: {"seed": 220}
        num_workers: 5
        prompt: 'prompts/predictor_completion/prediction.prompt'
        mini_batch_size: 1  #change to >1 if you want to include multiple samples in the one prompt
        mode: 'prediction'

meta_prompts:
    folder: 'prompts/meta_prompts_classification'
    num_err_prompt: 1  # Number of error examples per sample in the prompt generation
    num_err_samples: 2 # Number of error examples per sample in the sample generation
    history_length: 4 # Number of sample in the meta-prompt history
    num_generated_samples: 10 # Number of generated samples at each iteration
    num_initialize_samples: 10 # Number of generated samples at iteration 0, in zero-shot case
    samples_generation_batch: 10 # Number of samples generated in one call to the LLM
    num_workers: 5 #Number of parallel workers
    warmup: 4 # Number of warmup steps

eval:
    function_name: 'accuracy'
    num_large_errors: 4
    num_boundary_predictions : 0
    error_threshold: 0.5

llm:
    type: 'azure'
    name: 'gpt-4-1106-preview'
    temperature: 0.8

stop_criteria:
    max_usage: 2 #In $ in case of OpenAI models, otherwise number of tokens
    patience: 10 # Number of patience steps
    min_delta: 0.01 # Delta for the improvement definition

And the running command is

python3 run_pipeline.py     --prompt "Does this movie review contain a spoiler? answer Yes or No"     --task_description "Assistant is an expert classifier that will classify a movie review, and let the user know if it contains a spoiler for the reviewed movie or not."     --num_steps 30

But I still get the error: KeyError: 'samples'. I further printed the value of samples_batches, and find that there isn't 'samples' in samples_batches. I really appreciate your help.

sample batches: [{'num_samples': 10, 'task_description': 'Assistant is an expert classifier that will classify a movie review, and let the user know if it contains a spoiler for the reviewed movie or not.', 'instruction': 'Does this movie review contain a spoiler? answer Yes or No', 'text': 'Below are 10 challenging samples designed to test the capability of the expert classifier in identifying spoilers within movie reviews.\n\n### Sample 1:\nReview: "I must say, the character development in the second act was unexpected. Watching John Doe evolve from a small-time crook to something much greater was a journey I didn\'t see coming. The climax of the movie had a surprising twist that left me in awe."\n- Spoiler: No\n\n### Sample 2:\nReview: "The film\'s homage to the noir genre was clever, especially when you realize the protagonist was the villain all along. The subtle hints throughout were a nice touch, and the ending was a fitting tribute to the classics."\n- Spoiler: Yes\n\n### Sample 3:\nReview: "I was particularly impressed by how the movie represented historical events without giving away the key turning point that anyone familiar with the period would recognize. The costumes and setting felt authentic and immersive."\n- Spoiler: No\n\n### Sample 4:\nReview: "The director\'s choice to revisit the moment from the first movie, where it all began, tied the series together beautifully. It gave a deeper meaning to the protagonist\'s journey and certainly set up an epic finale."\n- Spoiler: Yes\n\n### Sample 5:\nReview: "The film was a rollercoaster of emotions, and the performances were top-notch. When the credits rolled, I was still reeling from the implications of the scientist\'s discovery on the narrative as a whole."\n- Spoiler: No\n\n### Sample 6:\nReview: "The movie\'s take on the multiverse concept was fascinating, especially the scene where the hero meets his demise, which I think is a bold move that will be remembered for years to come."\n- Spoiler: Yes\n\n### Sample 7:\nReview: "Without revealing too much, the film\'s twist on the traditional \'chosen one\' trope was refreshing. It makes you question the nature of destiny and the paths we choose."\n- Spoiler: No\n\n### Sample 8:\nReview: "Interestingly, the final confrontation wasn\'t between who you\'d expect, and the villain\'s demise was both poetic and satisfying. It\'s rare to see such a perfect encapsulation of justice in film."\n- Spoiler: Yes\n\n### Sample 9:\nReview: "The interplay between the main characters was the highlight of the film for me. Their final decisions felt earned and resonated with the overarching theme without disclosing the fate of the kingdom they fought to protect."\n- Spoiler: No\n\n### Sample 10:\nReview: "By the end, when the true identity of the puppet master was revealed, it was both a shock and a testament to the intricate storytelling woven throughout the series. The implications for the universe are profound."\n- Spoiler: Yes\n\nThese samples are challenging because they include nuanced language that may give the impression of revealing a plot without actually doing so, or they include actual spoilers cloaked in more subtle language that might not be immediately recognized as a spoiler.'}]

[Eladlev]

The issue was with the llm 'type', you wrote 'azure' in lowercase and the system expected 'Azure'. I modified the code to be invariant to lower/upper case

[liushiliushi]

The program is running smoothly now, thank you very much！