search

microsoft / promptflow

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
https://microsoft.github.io/promptflow/
MIT License
8.72k stars 781 forks source link

Is there a way to set the trace destination in the Python code, rather than via the command line? #3372

Open tyler-suard-parker opened 1 month ago

tyler-suard-parker commented 1 month ago

In this tutorial: https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-trace-local-sdk?view=azureml-api-2&tabs=python

we are told to set the trace destination using the Azure CLI. It would be really helpful if there was a way to do that in the Python code instead, like we can do with the other connections.

promptflow.trace.subscription_id = "absbas" prompflow.trace.workspace_name = "my workspace"

etc.

tyler-suard-parker commented 1 month ago

The reason why I am asking this is, I want to run PromptFlow in an Azure WebApp. I don't want to have to do the az login and set destination every time I restart that web app.

brynn-code commented 1 month ago

Hi. The az login is used to get the identity for accessing Azure resources, even if we support config the trace destination via python code like you've wrote, you will still be asked to login for getting the identity token to execute the flow.

tyler-suard-parker commented 1 month ago

@brynn-code That is really frustrating. I am working with an Azure Web App. I am able to access 6 other Azure resources by just loading my credentials into environment variables. How would you recommend I install and use the az login during deployment and then the set trace command with CI/CD with Azure Devops, so I don't have to enter those commands manually every time I deploy?

brynn-code commented 1 month ago

Would you like to show me the environment variables you've configured which could let Azure credential get token successfully?

We will call the DefaultAzureCredential to retrieve the token, that's a chained credential, az login is used to get the AzureCliCredential and that not the specific type we required, any else credential type could get the token can work as same, which means if the environment variables are in standard format, maybe it could work.

brynn-code commented 1 month ago

The standard format means that's something AzureCredential could recognize directly, in that case, maybe the env vars could work.

tyler-suard-parker commented 1 month ago

Hello Brynn, I don't think any of my environment variables allow az login to get the credentials. When I type az login, I need to open a website, enter a code, and log in manually. If there is a way around this I would appreciate some instructions on how to do it.

tyler-suard-parker commented 1 month ago

Would you like to show me the environment variables you've configured which could let Azure credential get token successfully?

We will call the DefaultAzureCredential to retrieve the token, that's a chained credential, az login is used to get the AzureCliCredential and that not the specific type we required, any else credential type could get the token can work as same, which means if the environment variables are in standard format, maybe it could work.

I'm sorry but I don't understand the question here.

tyler-suard-parker commented 1 month ago

I am able to get Promptflow Tracing to work during my build pipeline by logging in manually using the website link and authentication code using az login.

However, during the deployment phase, I am getting some errors when trying to do the same thing. How can I set my trace destination during my deployment/release pipeline so my app is ready to go when it is deployed? Do I have to run az login and then set the trace destination after my app is deployed?

tyler-suard-parker commented 1 month ago

A suggestion: just bypass all the OpenTelemetry stuff and just save things to CosmosDB using the CosmosDB SDK. That way I don't have to mess with az login during build or deployment.

tyler-suard-parker commented 1 month ago

It appears that I can't set the promptflow config during deployment, because according to pf config -h, that gets saved to ~/.promptflow/config.yaml, which is an unwriteable directory. You guys really did not think this through when writing it, it appears that your software is not compatible with Azure WebApps or Azure FunctionApps.

tyler-suard-parker commented 4 weeks ago

@brynn-code It appears that OpenTelemetry with Azure Monitoring uses just a connection string and I don't have to log in, you may want to adopt that approach: https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-enable?tabs=python

brynn-code commented 3 weeks ago

I am able to get Promptflow Tracing to work during my build pipeline by logging in manually using the website link and authentication code using az login.

However, during the deployment phase, I am getting some errors when trying to do the same thing. How can I set my trace destination during my deployment/release pipeline so my app is ready to go when it is deployed? Do I have to run az login and then set the trace destination after my app is deployed?

Does the error you mentioned here is the one in this issue? #3391

tyler-suard-parker commented 3 weeks ago

The error in #3991 is caused by this error, so the two are similar, yes. I am still unable to deploy my web app while using promptflow tracing, because promptflow tracing requires logging in via the azure command line and then setting the trace destination.