Running all combinations of variants in promptflow

[ZhiliangWu]

Is your feature request related to a problem? Please describe. According to https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-tune-prompts-using-variants?view=azureml-api-2#run-all-variants-with-a-single-row-of-data-and-check-outputs, the more node with variants, the more combinations you will miss in running

Each time you can only select one LLM node with variants to run while other LLM nodes will use the default variant.

This is a must-have feature for anyone familiar with grid-search in machine learning. At the beginning, I thought it was a bug in my code but according to the docs it is expected.

Describe the solution you'd like Instead of only being able to change one node variants at a time, support running flows with the combination of all variants, including the non-default ones.

Describe alternatives you've considered Use only one node with variants, but this obviously deprecate the idea of variants.

Additional context Relates to https://github.com/microsoft/promptflow/issues/944, which is labelled with long-term without arguments.

[D-W-]

Hi @ZhiliangWu , we won't support run all variants in near future. As a workaround, you can update the default variant in flow YAML before run or override single node's variant with --variant like this:

pf run create --flow web-classification --data web-classification/data.jsonl --variant '${summarize_text_content.variant_1}' --column-mapping url='${data.url}' --stream --name my_first_variant_run

[ZhiliangWu]

Hi @ZhiliangWu , we won't support run all variants in near future. As a workaround, you can update the default variant in flow YAML before run or override single node's variant with --variant like this:

pf run create --flow web-classification --data web-classification/data.jsonl --variant '${summarize_text_content.variant_1}' --column-mapping url='${data.url}' --stream --name my_first_variant_run

Any suggestions on overwriting the default variants on the fly so that we can work around it?

[D-W-]

Unfortunately, we did not provide ability to override YAML in pf run. However, you can leverage our flow-as-func feature to override YAML file like this:

f = load_flow(
    flow_path,
)
f.context.overrides = {
    "node_variants.node_name.default_variant_id": "variant_id",
}

Ref: https://microsoft.github.io/promptflow/how-to-guides/execute-flow-as-a-function.html

[ZhiliangWu]

Could you please consider adding this override functionality in the pf commands? We used pf to conduct experiments, and this flow-as-func feature is not designed for that. I am still wondering why supporting grid-search like experimentation capability is such difficult. Isn't variants a highlighted feature from prompt flow? In the current form, such variant mechanism is essentially useless for flow with more than two nodes with variants.

[ZhiliangWu]

@D-W-

[D-W-]

Sorry I missed this. We are still evaluating our long-term plan for variants support so I could not promise we can support this in short term. I'll update here once we decide to support it.