search

microsoft / promptflow

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
https://microsoft.github.io/promptflow/
MIT License
8.68k stars 778 forks source link

[BUG] Function `evaluate(...)` from promptflow.evals.evaluate can't be used / imported. #3382

Open megel opened 3 weeks ago

megel commented 3 weeks ago

Describe the bug I followed the example in MSDocs Evaluate on test dataset using evaluate() in my GitHub code space.

I'm not able to import evaluate with: from promptflow.evals.evaluate import evaluate

grafik

grafik

How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug: 1 I have used this devcontainer 

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/python
{
    "name": "Python 3.12",
    "build": {
        "dockerfile": "Dockerfile"

        // , "context": ".."
    },
    "features": {
        "ghcr.io/devcontainers/features/azure-cli:1": {
            "installBicep": true,
            "extensions": "ml"
        },
        "ghcr.io/devcontainers/features/git:1": {},
        "ghcr.io/azure/azure-dev/azd:latest": {},
        "ghcr.io/devcontainers/features/docker-in-docker:2": {},
        "ghcr.io/devcontainers/features/aws-cli": {},
        "ghcr.io/devcontainers/features/github-cli": {},
        "ghcr.io/devcontainers/features/python": {
            "version": "3.12"
        },
        "ghcr.io/devcontainers/features/docker-in-docker": {
        },
        "ghcr.io/devcontainers-contrib/features/pipenv": {
        },
        "ghcr.io/eitsupi/devcontainer-features/go-task:latest": {
        }
    },
    "customizations": {
        "vscode": {
            "extensions": [
                "prompt-flow.prompt-flow",
                "ms-azuretools.vscode-docker",
                "ms-python.python",
                "ms-toolsai.jupyter",
                "ms-azuretools.vscode-bicep",
                "rogalmic.bash-debug",
                "GitHub.copilot",
                "humao.rest-client",
                "heaths.vscode-guid",
                "ms-python.vscode-pylance",
                "ms-python.black-formatter",
                "ms-python.isort",
                "github.vscode-github-actions",
                "streetsidesoftware.code-spell-checker",
                "streetsidesoftware.code-spell-checker-german",
                "ms-azuretools.vscode-azurecontainerapps",
                "ms-vscode.azure-account",
                "wholroyd.jinja"
            ]
        }
    }
}

with this Dockerfile:

# Use a specific version of Python image to ensure consistency across builds
FROM mcr.microsoft.com/devcontainers/python:3.12-bullseye

# Combine system updates, package installations and Python upgrades into a single RUN command to reduce layers
# Also, clean up the apt cache to reduce image size
RUN sudo apt-get update && sudo apt-get install -y \
    gcc \
    cmake \
    pkg-config \
    libdbus-1-dev \
    libglib2.0-dev \
 && python -m pip install --upgrade pip \
 && sudo apt-get clean \
 && sudo rm -rf /var/lib/apt/lists/*

# Copy requirements.txt and install the Python packages
# Install keyring-related and IPython packages in the same layer to reduce the image size and build time
COPY requirements.txt .
RUN pip install -r requirements.txt \
    && pip install keyrings.alt dbus-python ipython ipykernel

# Configure the IPython kernel
RUN ipython kernel install --name "python3" --user

requirements.txt

promptflow
promptflow-tools
promptflow-azure
promptflow-evals
azure-cosmos
azure-ai-ml
azure-ai-resources
azure-ai-generative[index,evaluate,promptflow] #==1.0.0b8
azure-search-documents==11.4.0

langchain
langchain-community
azure-ai-documentintelligence

nbconvert
requests
keyrings.alt
python-dotenv
poetry

Running Information(please complete the following information): grafik promptflow 1.11.0 promptflow-azure 1.11.0 promptflow-core 1.11.0 promptflow-devkit 1.11.0 promptflow-evals 0.3.0 promptflow-tools 1.4.0 promptflow-tracing 1.11.0 Python 3.12.3

My installed python packages are:

Package                                  Version
---------------------------------------- -----------
aiohttp                                  3.9.5
aiohttp-retry                            2.8.3
aiosignal                                1.3.1
alembic                                  1.13.1
aniso8601                                9.0.1
annotated-types                          0.7.0
anyio                                    4.4.0
argcomplete                              3.3.0
asttokens                                2.4.1
attrs                                    23.2.0
azure-ai-documentintelligence            1.0.0b3
azure-ai-generative                      0.0.0b1
azure-ai-ml                              1.16.1
azure-ai-resources                       1.0.0b8
azure-common                             1.1.28
azure-core                               1.30.1
azure-cosmos                             4.7.0
azure-identity                           1.16.0
azure-mgmt-core                          1.4.0
azure-mgmt-resource                      22.0.0
azure-monitor-opentelemetry-exporter     1.0.0b26
azure-search-documents                   11.4.0
azure-storage-blob                       12.19.0
azure-storage-file-datalake              12.14.0
azure-storage-file-share                 12.16.0
azureml-mlflow                           1.56.0
beautifulsoup4                           4.12.3
bleach                                   6.1.0
blinker                                  1.8.2
build                                    1.2.1
CacheControl                             0.14.0
cachetools                               5.3.3
certifi                                  2024.6.2
cffi                                     1.16.0
charset-normalizer                       3.3.2
cleo                                     2.1.0
click                                    8.1.7
cloudpickle                              3.0.0
colorama                                 0.4.6
comm                                     0.2.2
contourpy                                1.2.1
crashtest                                0.4.1
cryptography                             42.0.7
cycler                                   0.12.1
dataclasses-json                         0.6.6
datasets                                 2.19.2
debugpy                                  1.8.1
decorator                                5.1.1
defusedxml                               0.7.1
Deprecated                               1.2.14
dill                                     0.3.8
distlib                                  0.3.8
distro                                   1.9.0
dnspython                                2.6.1
docker                                   7.1.0
docutils                                 0.21.2
dulwich                                  0.21.7
email_validator                          2.1.1
entrypoints                              0.4
evaluate                                 0.4.2
executing                                2.0.1
fastapi                                  0.111.0
fastapi-cli                              0.0.4
fastjsonschema                           2.19.1
filelock                                 3.14.0
filetype                                 1.2.0
fixedint                                 0.1.6
Flask                                    3.0.3
Flask-Cors                               4.0.1
flask-restx                              1.3.0
fonttools                                4.53.0
frozenlist                               1.4.1
fsspec                                   2024.3.1
gitdb                                    4.0.11
GitPython                                3.1.43
google-api-core                          2.19.0
google-auth                              2.29.0
google_search_results                    2.4.1
googleapis-common-protos                 1.63.1
graphene                                 3.3
graphql-core                             3.2.3
graphql-relay                            3.2.0
greenlet                                 3.0.3
gunicorn                                 22.0.0
h11                                      0.14.0
httpcore                                 1.0.5
httptools                                0.6.1
httpx                                    0.27.0
huggingface-hub                          0.23.3
idna                                     3.7
importlib_metadata                       7.1.0
importlib_resources                      6.4.0
installer                                0.7.0
ipykernel                                6.29.4
ipython                                  8.25.0
isodate                                  0.6.1
itsdangerous                             2.2.0
jaraco.classes                           3.4.0
jaraco.context                           5.3.0
jedi                                     0.19.1
jeepney                                  0.8.0
Jinja2                                   3.1.4
joblib                                   1.4.2
json-lines                               0.5.0
jsonlines                                4.0.0
jsonpatch                                1.33
jsonpath-ng                              1.6.1
jsonpickle                               3.0.4
jsonpointer                              2.4
jsonschema                               4.22.0
jsonschema-specifications                2023.12.1
jupyter_client                           8.6.2
jupyter_core                             5.7.2
jupyterlab_pygments                      0.3.0
keyring                                  24.3.1
keyrings.alt                             5.0.1
kiwisolver                               1.4.5
langchain                                0.2.1
langchain-community                      0.2.1
langchain-core                           0.2.3
langchain-text-splitters                 0.2.0
langsmith                                0.1.69
Mako                                     1.3.5
Markdown                                 3.6
markdown-it-py                           3.0.0
MarkupSafe                               2.1.5
marshmallow                              3.21.2
matplotlib                               3.9.0
matplotlib-inline                        0.1.7
mdurl                                    0.1.2
mistune                                  3.0.2
mlflow                                   2.13.2
mlflow-skinny                            2.13.1
more-itertools                           10.2.0
msal                                     1.28.0
msal-extensions                          1.1.0
msgpack                                  1.0.8
msrest                                   0.7.1
multidict                                6.0.5
multiprocess                             0.70.16
mypy-extensions                          1.0.0
nbclient                                 0.10.0
nbconvert                                7.16.4
nbformat                                 5.10.4
nest-asyncio                             1.6.0
numpy                                    1.26.4
oauthlib                                 3.2.2
openai                                   1.31.0
opencensus                               0.11.4
opencensus-context                       0.1.3
opencensus-ext-azure                     1.1.13
opencensus-ext-logging                   0.1.1
opentelemetry-api                        1.25.0
opentelemetry-exporter-otlp-proto-common 1.25.0
opentelemetry-exporter-otlp-proto-http   1.25.0
opentelemetry-proto                      1.25.0
opentelemetry-sdk                        1.25.0
opentelemetry-semantic-conventions       0.46b0
orjson                                   3.10.3
packaging                                23.2
pandas                                   2.2.2
pandocfilters                            1.5.1
parso                                    0.8.4
pexpect                                  4.9.0
pillow                                   10.3.0
pip                                      24.0
pkginfo                                  1.11.0
platformdirs                             4.2.2
ply                                      3.11
poetry                                   1.8.3
poetry-core                              1.9.0
poetry-plugin-export                     1.8.0
portalocker                              2.8.2
prompt_toolkit                           3.0.45
promptflow                               1.11.0
promptflow-azure                         1.11.0
promptflow-core                          1.11.0
promptflow-devkit                        1.11.0
promptflow-evals                         0.3.0
promptflow-tools                         1.4.0
promptflow-tracing                       1.11.0
proto-plus                               1.23.0
protobuf                                 4.25.3
psutil                                   5.9.8
ptyprocess                               0.7.0
pure-eval                                0.2.2
pyarrow                                  15.0.2
pyarrow-hotfix                           0.6
pyasn1                                   0.6.0
pyasn1_modules                           0.4.0
pycparser                                2.22
pydantic                                 2.7.3
pydantic_core                            2.18.4
pydash                                   7.0.7
Pygments                                 2.18.0
PyJWT                                    2.8.0
pyparsing                                3.1.2
pyproject_hooks                          1.1.0
python-dateutil                          2.9.0.post0
python-dotenv                            1.0.1
python-multipart                         0.0.9
pytz                                     2024.1
PyYAML                                   6.0.1
pyzmq                                    26.0.3
querystring-parser                       1.2.4
rapidfuzz                                3.9.3
referencing                              0.35.1
regex                                    2024.5.15
requests                                 2.32.3
requests-oauthlib                        2.0.0
requests-toolbelt                        1.0.0
rich                                     13.7.1
rpds-py                                  0.18.1
rsa                                      4.9
ruamel.yaml                              0.18.6
ruamel.yaml.clib                         0.2.8
scikit-learn                             1.5.0
scipy                                    1.13.1
SecretStorage                            3.3.3
shellingham                              1.5.4
six                                      1.16.0
smmap                                    5.0.1
sniffio                                  1.3.1
soupsieve                                2.5
SQLAlchemy                               2.0.30
sqlparse                                 0.5.0
stack-data                               0.6.3
starlette                                0.37.2
strictyaml                               1.7.3
tabulate                                 0.9.0
tenacity                                 8.3.0
threadpoolctl                            3.5.0
tiktoken                                 0.7.0
tinycss2                                 1.3.0
tomlkit                                  0.12.5
tornado                                  6.4
tqdm                                     4.66.4
traitlets                                5.14.3
trove-classifiers                        2024.5.22
typer                                    0.12.3
typing_extensions                        4.12.1
typing-inspect                           0.9.0
tzdata                                   2024.1
ujson                                    5.10.0
urllib3                                  2.2.1
uvicorn                                  0.30.1
uvloop                                   0.19.0
virtualenv                               20.26.2
waitress                                 2.1.2
watchfiles                               0.22.0
wcwidth                                  0.2.13
webencodings                             0.5.1
websocket-client                         1.8.0
websockets                               12.0
Werkzeug                                 3.0.3
wrapt                                    1.16.0
xxhash                                   3.4.1
yarl                                     1.9.4
zipp                                     3.19.1
brynn-code commented 3 weeks ago

Hi @singankit and @luigiw , could you please help take a look at this issue?

megel commented 3 weeks ago

My issue is related to Python 3.12.3.

I have double checked with another Python version 3.11.9 which works.

luigiw commented 2 weeks ago

Thank you @megel, we're working on a change to support Python 3.12.3.