run_generation_pipeline is not working

[oanastrut]

Hi. When I try to run the generating movie reviews example using run_generation_pipeline.py I receive this error:

ValueError: At least one label specified must be in y_true

The config_generator.yml is looking like this:

        annotator:
            method : ''

        dataset:
            max_samples: 20
            label_schema:  ["1","2","3","4","5"]

        predictor:
            method : 'llm'
            config:
                prompt: 'prompts/predictor_completion/prediction_generation.prompt'
                mini_batch_size: 1
                llm:
                    type: 'Azure'
                    name: 'my_engine_name'
                num_workers: 7

        meta_prompts:
            folder: 'prompts/meta_prompts_generation'
            warmup: 1

        eval:
            function_name: 'ranking'
            error_threshold: 4

How can I fix this issue?

[Eladlev]

Hi, This part of the configuration:

annotator:
   method : ''

This means that you are not using any annotators. It will work only if you are working with a custom dataset, and load it properly to the system with valid annotations. See here: https://github.com/Eladlev/AutoPrompt/issues/51#issuecomment-2022592389

If you want to use either LLM annotator or a human annotator, you should modify this field according to the instructions.

[oanastrut]

Does this apply to the generation task as well? In the examples.md is mentioned that the annotator section in the generation config yaml file remains empty.

[Eladlev]

The error you encounter occurs in the annotator of the first phase of the optimization, when we use the classification pipeline. The issue is in the annotator field in the config_default.yml file.

Just to explain in more detail on the optimization process and what is (probably) the issue you encounter: The generation optimization consists of two phases:

1. Optimizing the ranking prompt. This is done by converting the generation task to a ranking task and optimizing the ranking prompt treating it as a classification task (where the default labels are 1-5). For this part of the optimization we load the annotator from the config_default.yml. The error you encounter happens in this step and my guess is that it is probably that you are using there an inappropriate annotator. If you are using an LLM annotator, for example, you should pay attention that the prompt instructing the model to return the correct labels (1-5 and not 'Yes' and 'No'). Actually in this case you already have a ranking prompt so you can skip this step.
2. After having a ranking prompt we are using it as a score function and optimize the generation prompt. For this part, we are using the config_generator.yml