search

Azure / azureml-examples

Official community-driven Azure Machine Learning examples, tested with GitHub Actions.
https://docs.microsoft.com/azure/machine-learning
MIT License
1.72k stars 1.4k forks source link

openai-whisper-large batch endpoint, submit job failure: Failed building wheel for pycocotools #2602

Open Debutant9527 opened 1 year ago

Debutant9527 commented 1 year ago

Operating System

Linux

Version Information

python version: 3.10 azure-ai-ml==1.9.0

Steps to reproduce

  1. Follow this tutorial: https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/system/inference/automatic-speech-recognition/asr-batch-endpoint.ipynb
  2. At step 5, "Run a batch inference job", the job failed with error "ERROR: Failed building wheel for pycocotools ERROR: Could not build wheels for pycocotools, which is required to install pyproject.toml-based projects"

Expected behavior

the job could run successfully

Actual behavior

job failed

Addition information

Full error Building wheels for collected packages: wget, pycocotools, audioread, databricks-cli, fusepy, rouge-score, seqeval, future, lit Building wheel for wget (setup.py): started Building wheel for wget (setup.py): finished with status 'done' Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9657 sha256=071432923f7fa0f344e58965dee563fade14b810bc2a1c6a6726914eb73b5fd8 Stored in directory: /root/.cache/pip/wheels/46/4a/43/6e71c9584e8b20b326931deaf89f11bf4db1bda170ac38e7b4 Building wheel for pycocotools (pyproject.toml): started Building wheel for pycocotools (pyproject.toml): finished with status 'error' Building wheel for audioread (setup.py): started Building wheel for audioread (setup.py): finished with status 'done' Created wheel for audioread: filename=audioread-3.0.0-py3-none-any.whl size=23703 sha256=b98cd616cd2952377f58e9d01dc541234b72859e3996ed77015d3f0765d14fa5 Stored in directory: /root/.cache/pip/wheels/74/25/48/ad94b69151b78e9aeba6850da119f04eda1c811d22fcf4b32d Building wheel for databricks-cli (setup.py): started Building wheel for databricks-cli (setup.py): finished with status 'done' Created wheel for databricks-cli: filename=databricks_cli-0.17.7-py3-none-any.whl size=143860 sha256=37b05d8f08afa241de784149024f3e0fec5a2b617b14fd136ffbb49fa2097cab Stored in directory: /root/.cache/pip/wheels/35/d3/b7/3c0e882c34e5809296ce94413a2555fb1fa5ba3d17da96f3c6 Building wheel for fusepy (setup.py): started Building wheel for fusepy (setup.py): finished with status 'done' Created wheel for fusepy: filename=fusepy-3.0.1-py3-none-any.whl size=10486 sha256=f71aeb82a1f9ca38b08291e7a2252726de03a7554edc50468f865c8e8b9c05b6 Stored in directory: /root/.cache/pip/wheels/db/01/ee/105147547cf0ecd1031b26901757e1f17c98902c8cc7bcaa47 Building wheel for rouge-score (setup.py): started Building wheel for rouge-score (setup.py): finished with status 'done' Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24936 sha256=cc8972bffa71c19cf1b84ef8caa9a23f4dcb23e9dcff54a678c236bd85b6a202 Stored in directory: /root/.cache/pip/wheels/5a/a8/47/66a086bcca0e429716cd68c4da6b7f9ac7976868735937eb63 Building wheel for seqeval (setup.py): started Building wheel for seqeval (setup.py): finished with status 'done' Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16165 sha256=c27dd16520ef0bac42384e6ba4f6cc2c88710976a24bc6f90bb16dd081eaa937 Stored in directory: /root/.cache/pip/wheels/e3/30/9b/6b670dac34775f2b7cc4e9b172202e81fbb4f9cdb103c1ca66 Building wheel for future (setup.py): started Building wheel for future (setup.py): finished with status 'done' Created wheel for future: filename=future-0.18.3-py3-none-any.whl size=492025 sha256=3bc99b21dadc4732c8c37ad8bc84a9ea6920c5948a41cb38a8cfc1b8f878eb59 Stored in directory: /root/.cache/pip/wheels/a6/db/41/71a0e5d071a14e716cc11bb021a9caa8f76ec337eca071487e Building wheel for lit (pyproject.toml): started Building wheel for lit (pyproject.toml): finished with status 'done' Created wheel for lit: filename=lit-16.0.6-py3-none-any.whl size=93584 sha256=27c730695537db680b4090e97584f0438edfb220fd34bcb8bf25648d1cbaee01 Stored in directory: /root/.cache/pip/wheels/31/26/a6/958a0b82f632a115bc37080c7f3b04baaaa7f55f07d5ffa6c0 Successfully built wget audioread databricks-cli fusepy rouge-score seqeval future lit Failed to build pycocotools

failed Pip subprocess error: error: subprocess-exited-with-error

× Building wheel for pycocotools (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [23 lines of output] running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-cpython-38 creating build/lib.linux-x86_64-cpython-38/pycocotools copying pycocotools/coco.py -> build/lib.linux-x86_64-cpython-38/pycocotools copying pycocotools/mask.py -> build/lib.linux-x86_64-cpython-38/pycocotools copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-cpython-38/pycocotools copying pycocotools/init.py -> build/lib.linux-x86_64-cpython-38/pycocotools running build_ext Compiling pycocotools/_mask.pyx because it changed. [1/1] Cythonizing pycocotools/_mask.pyx building 'pycocotools._mask' extension creating build/common creating build/temp.linux-x86_64-cpython-38 creating build/temp.linux-x86_64-cpython-38/common creating build/temp.linux-x86_64-cpython-38/pycocotools gcc -pthread -B /azureml-envs/azureml_6d341b69b79ddf697bca4a07cf2a1ac6/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip-build-env-966dvlnc/overlay/lib/python3.8/site-packages/numpy/core/include -I./common -I/azureml-envs/azureml_6d341b69b79ddf697bca4a07cf2a1ac6/include/python3.8 -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-38/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99 gcc: error: ../common/maskApi.c: No such file or directory /tmp/pip-build-env-966dvlnc/overlay/lib/python3.8/site-packages/Cython/Compiler/Main.py:384: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-gr5kge21/pycocotools_111ec10f337f48e1b69eedd76c6c836d/pycocotools/_mask.pyx tree = Parsing.p_module(s, pxd, full_module_name) error: command '/usr/bin/gcc' failed with exit code 1 [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pycocotools ERROR: Could not build wheels for pycocotools, which is required to install pyproject.toml-based projects

CondaEnvException: Pip failed

The command '/bin/sh -c ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_6d341b69b79ddf697bca4a07cf2a1ac6 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name pycache -exec rm -rf {} + && ldconfig' returned a non-zero code: 1 2023/08/27 21:33:22 Container failed during run: acb_step_0. No retries remaining. failed to run step ID: acb_step_0: exit status 1

Run ID: crea failed after 3m15s. Error: failed during run, err: exit status 1

Debutant9527 commented 1 year ago

The same issue happened when submitting job with batch endpoint which created in Workspace UI or with the notebook tutorial https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/system/inference/automatic-speech-recognition/asr-batch-endpoint.ipynb