search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.6k stars 3.01k forks source link

Couldn't create nuclio functions #2067

Closed jinzishuai closed 2 years ago

jinzishuai commented 4 years ago

Hi there,

I am new to cvat and am trying to follow instructions at https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/installation.md#semi-automatic-and-automatic-annotation to setup automatic annotation with nuclio.

The project in nuclio was created without any problem but I got errors creating the functions. Here is what I see:

(base) seki@xubuntu-20:~/src/cvat$ nuctl deploy --project-name cvat     --path serverless/openvino/dextr/nuclio     --volume `pwd`/serverless/openvino/common:/opt/nuclio/common     --platform local
20.08.23 16:05:46.718                     nuctl (I) Deploying function {"name": ""}
20.08.23 16:05:46.718                     nuctl (I) Building {"versionInfo": "Label: 1.4.17, Git commit: 278c7a4fb23a93973d16d87dbaaad87823e9644f, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
20.08.23 16:05:46.960                     nuctl (I) Cleaning up before deployment
20.08.23 16:05:47.050                     nuctl (I) Staging files and preparing base images
20.08.23 16:05:47.051                     nuctl (I) Building processor image {"imageName": "cvat/openvino.dextr:latest"}
20.08.23 16:05:47.051     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.17-amd64"}
20.08.23 16:05:48.333     nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-479579805/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-bt1egf28hvo5093nhgs0 -f /tmp/nuclio-build-479579805/staging/Dockerfile.onbuild  --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler --build-arg NUCLIO_LABEL=1.4.17 --build-arg NUCLIO_ARCH=amd64  .", "stderr": "unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n"}
20.08.23 16:05:48.334                     nuctl (W) Create function failed, setting function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:233\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:194\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:51\nFailed to build processor image\n    /nuclio/pkg/processor/build/builder.go:250\nFailed to build processor image", "errCauses": [{"error": "Failed to build image artifacts", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:233\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:194\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:51\nFailed to build image artifacts", "errorCauses": [{"error": "Failed to copy objects from onbuild", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:233\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:194\nFailed to copy objects from onbuild", "errorCauses": [{"error": "Failed to build onbuild image", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:233\nFailed to build onbuild image", "errorCauses": [{"error": "stdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n", "errorCauses": [{"error": "exit status 1"}]}]}]}]}]}

Error - exit status 1
    /nuclio/pkg/cmdrunner/cmdrunner.go:124

Call stack:
stdout:

stderr:
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory

    /nuclio/pkg/cmdrunner/cmdrunner.go:124
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:233
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:194
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:51
Failed to build processor image
    /nuclio/pkg/processor/build/builder.go:250

I assume we are missing files under the /tmp/nuclio-build-xxx folder but I don't see anything like that in the /tmp folder and I made sure I do have the permission to create a new folder within /tmp.

Has anyone seen this problem before? Any advice is greatly appreciated.

Thanks a lot

jinzishuai commented 4 years ago

BTW, I see a similar error message for this function (in error state) in the nuclio dashboard

Error - exit status 1
    /nuclio/pkg/cmdrunner/cmdrunner.go:124

Call stack:
stdout:

stderr:
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-033503775: no such file or directory

    /nuclio/pkg/cmdrunner/cmdrunner.go:124
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:233
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:194
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:51
Failed to build processor image
    /nuclio/pkg/processor/build/builder.go:250
nmanovic commented 4 years ago

@jinzishuai , are you able to deploy any other serverless functions? Please run ls -la command inside /tmp/ directory.

jinzishuai commented 4 years ago

@nmanovic I think you are on the right direction

My nucli version is 1.4.17

(base) seki@xubuntu-20:~/src/gtl2via$ nuctl version
Client version:
"Label: 1.4.17, Git commit: 278c7a4fb23a93973d16d87dbaaad87823e9644f, OS: linux, Arch: amd64, Go version: go1.14.3"(base)

But my dashboard is running on version 1.4.8. Do you think that is a problem?

jinzishuai commented 4 years ago

I did downgrade to the same nuctl-1.4.8 version but still got the same error.

I don't see anything /tmp/ being created when running a command.

MPiecuch commented 3 years ago

Hello,

I have the same problem. I have tried nucleo versions 1.5.16 and the newest one 1.6.1. Does anyone have any suggestions?

kiteklan commented 3 years ago

I have the same problem did may be 10 new installs on different ubuntu version 18.04 19 20 20.10 same same same

nmanovic commented 3 years ago

@kiteklan , could you please post your error message please? Let's try to investigate the issue together.

kiteklan commented 3 years ago

@kiteklan , could you please post your error message please? Let's try to investigate the issue together.

I had workaround the problem but seems like cvat serverless+nuclio has an issue with all kernel versions running on hyper-v , I had solved it by installing on a proxmox cpu type “host” so that the virtual machine will use all the cpu assets. I was very hopefull from hyper-v since I was going to use cvat on a double core 24processor with 32gb ram , insisted trying. I am preparing all the errors now , also shared on https://github.com/openvinotoolkit/cvat/issues/3109

kiteklan commented 3 years ago

@kiteklan , could you please post your error message please? Let's try to investigate the issue together.

I had workaround the problem but seems like cvat serverless+nuclio has an issue with all kernel versions running on hyper-v , I had solved it by installing on a proxmox cpu type “host” so that the virtual machine will use all the cpu assets. I was very hopefull from hyper-v since I was going to use cvat on a double core 24processor with 32gb ram , insisted trying. I am preparing all the errors now , also shared on #3109

https://github.com/nuclio/nuclio/issues/2162

kiteklan commented 3 years ago

nuctl deploy --project-name cvat --path serverless/openvino/omz/public/yolo-v3-tf/nuclio --volume pwd/serverless/common:/opt/nuclio/common --platform local

tryinh to install the following function to nuclio with 1.5.6 and always breaking up with the command

========= Converting yolo-v3-tf to IR (FP32) Conversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/FP32 --model_name=yolo-v3-tf '--input_shape=[1,416,416,3]' --input=input_1 '--scale_values=input_1[255]' --reverse_input_channels --transformations_config=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.json --input_model=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.pb

FAILED: yolo-v3-tf .......

Error - exit status 1 /nuclio/pkg/cmdrunner/shellrunner.go:96

Call stack: stdout: Sending build context to Docker daemon 44.69MB

Step 1/16 : FROM openvino/ubuntu18_dev:2020.2 ---> bf7a4dff2d97 Step 2/16 : ARG NUCLIO_LABEL ---> Using cache ---> c9a10c5b4f05 Step 3/16 : ARG NUCLIO_ARCH ---> Using cache ---> 8ed29e90bf42 Step 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR ---> Using cache ---> fc2cdf7bdff6 Step 5/16 : USER root ---> Using cache ---> 44197e8724ab Step 6/16 : WORKDIR /opt/nuclio ---> Using cache ---> 4b2adc3eb6f5 Step 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip ---> Using cache ---> 11dc453d1c27 Step 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name yolo-v3-tf -o /opt/nuclio/open_model_zoo ---> Using cache ---> f6ec46de8b52 Step 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo ---> Running in 762f8276d957 �[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo �[0m========= Converting yolo-v3-tf to IR (FP32) Conversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/FP32 --model_name=yolo-v3-tf '--input_shape=[1,416,416,3]' --input=input_1 '--scale_values=input_1[255]' --reverse_input_channels --transformations_config=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.json --input_model=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.pb

FAILED: yolo-v3-tf Removing intermediate container 762f8276d957

stderr: The command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1

/nuclio/pkg/cmdrunner/shellrunner.go:96

Failed to build /nuclio/pkg/dockerclient/shell.go:118 Failed to build docker image .../pkg/containerimagebuilderpusher/docker.go:53 Failed to build processor image /nuclio/pkg/processor/build/builder.go:250

On Fri, Apr 23, 2021 at 12:23 PM Nikita Manovich @.***> wrote:

@kiteklan https://github.com/kiteklan , could you please post your error message please? Let's try to investigate the issue together.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openvinotoolkit/cvat/issues/2067#issuecomment-825526628, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASI2NWZKRXOFVJ5TWO4MZM3TKE37ZANCNFSM4QI4JQ5Q .

kiteklan commented 3 years ago

Pls let me know what else I can send to help. I really like the cvat .I still keep the problematic hyper-v version with all stages’ snapshots of you need share the ssh. Deploy_cpu script can install some functions , some not but I really needed yolov3. Others give similar errors

kiteklan commented 3 years ago

here is the full log.

On Fri, Apr 23, 2021 at 12:23 PM Nikita Manovich @.***> wrote:

@kiteklan https://github.com/kiteklan , could you please post your error message please? Let's try to investigate the issue together.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openvinotoolkit/cvat/issues/2067#issuecomment-825526628, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASI2NWZKRXOFVJ5TWO4MZM3TKE37ZANCNFSM4QI4JQ5Q .

Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-142-generic x86_64)

8 packages can be updated. 0 of these updates are security updates. To see these additional updates run: apt list --upgradable

New release '20.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Apr 23 11:48:47 2021 . . .

@.***:~$ ls cvat

@.:~/cvat# cd serverless @.:~/cvat/serverless# ls common deploy_cpu.sh deploy_gpu.sh openvino pytorch tensorflow @.***:~/cvat/serverless# ./deploy_cpu.sh Deploying /root/cvat/serverless/openvino/dextr function... 21.04.23 12:20:46.925 nuctl (I) Deploying function {"name": ""} 21.04.23 12:20:46.925 nuctl (I) Building {"versionInfo": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""} 21.04.23 12:20:47.285 nuctl (I) Cleaning up before deployment {"functionName": "openvino-dextr"} 21.04.23 12:20:47.398 nuctl (I) Function already exists, deleting function containers {"functionName": "openvino-dextr"} 21.04.23 12:20:47.843 nuctl (I) Staging files and preparing base images 21.04.23 12:20:47.844 nuctl (I) Building processor image {"imageName": "cvat/openvino.dextr:latest"} 21.04.23 12:20:47.844 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.5.16-amd64"} 21.04.23 12:20:57.969 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"} 21.04.23 12:21:09.235 nuctl.platform (I) Building docker image {"image": "cvat/openvino.dextr:latest"} 21.04.23 12:21:09.875 nuctl.platform (I) Pushing docker image into registry {"image": "cvat/openvino.dextr:latest", "registry": ""} 21.04.23 12:21:09.875 nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/openvino.dextr:latest"} 21.04.23 12:21:09.875 nuctl (I) Build complete {"result": {"Image":"cvat/openvino.dextr:latest","UpdatedFunctionConfig":{"metadata":{"name":"openvino-dextr","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"openvino","min_pos_points":"4","name":"DEXTR","spec":"","type":"interactor"}},"spec":{"description":"Deep Extreme Cut","handler":"main:handler","runtime":"python:3.6","env":[{"name":"NUCLIO_PYTHON_EXE_PATH","value":"/opt/nuclio/common/openvino/python3"}],"resources":{},"image":"cvat/openvino.dextr:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"myHttpTrigger","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/root/cvat/serverless/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat/openvino.dextr","baseImage":"openvino/ubuntu18_runtime:2020.2","directives":{"postCopy":[{"kind":"RUN","value":"curl -O https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip"},{"kind":"RUN","value":"unzip dextr_model_v1.zip"},{"kind":"RUN","value":"pip3 install Pillow"}],"preCopy":[{"kind":"USER","value":"root"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/bin/pip"}]},"codeEntryType":"image"},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"securityContext":{},"eventTimeout":"30s"}}}} 21.04.23 12:21:14.833 nuctl.platform.docker (W) Failed to run container {"err": "stdout:\na7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35\ndocker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.\n\nstderr:\n", "errVerbose": "\nError - exit status 125\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\na7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35\ndocker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.\n\nstderr:\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nstdout:\na7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35\ndocker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.\n\nstderr:\n", "errCauses": [{"error": "exit status 125"}], "stdout": "a7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35\ndocker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.\n", "stderr": ""} 21.04.23 12:21:14.833 nuctl (W) Failed to create a function; setting the function status {"err": "Failed to run a Docker container", "errVerbose": "\nError - exit status 125\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\na7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35\ndocker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.\n\nstderr:\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to run a Docker container\n /nuclio/pkg/platform/local/platform.go:653\nFailed to run a Docker container", "errCauses": [{"error": "stdout:\na7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35\ndocker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.\n\nstderr:\n", "errorVerbose": "\nError - exit status 125\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\na7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35\ndocker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.\n\nstderr:\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nstdout:\na7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35\ndocker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.\n\nstderr:\n", "errorCauses": [{"error": "exit status 125"}]}]}

Error - exit status 125 /nuclio/pkg/cmdrunner/shellrunner.go:96

Call stack: stdout: a7e7f135a629ad788e0fc14259f8ade1f0898a4fb8f73c18fdd3669e07ff8b35 docker: Error response from daemon: driver failed programming external connectivity on endpoint nuclio-nuclio-openvino-dextr (57cff45a36550095dd802059ec3b0a38ea20f189b941054af837962b9ccdeba7): Bind for 0.0.0.0:49154 failed: port is already allocated.

stderr:

/nuclio/pkg/cmdrunner/shellrunner.go:96

Failed to run a Docker container /nuclio/pkg/platform/local/platform.go:653 Failed to deploy function ...//nuclio/pkg/platform/abstract/platform.go:182 Deploying /root/cvat/serverless/openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco function... 21.04.23 12:21:15.740 nuctl (I) Deploying function {"name": ""} 21.04.23 12:21:15.740 nuctl (I) Building {"versionInfo": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""} 21.04.23 12:21:16.093 nuctl (I) Cleaning up before deployment {"functionName": "openvino-mask-rcnn-inception-resnet-v2-atrous-coco"} 21.04.23 12:21:16.158 nuctl (I) Staging files and preparing base images 21.04.23 12:21:16.159 nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest"} 21.04.23 12:21:16.159 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.5.16-amd64"} 21.04.23 12:21:24.066 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"} 21.04.23 12:21:35.221 nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest"} 21.04.23 12:21:45.009 nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-518354243/staging", "cmd": "docker build --network host --force-rm -t cvat/openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest -f /tmp/nuclio-build-518354243/staging/Dockerfile.processor --build-arg NUCLIO_LABEL=1.5.16 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler .", "stderr": "The command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n"} 21.04.23 12:21:45.016 nuctl (W) Failed to create a function; setting the function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/18 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/18 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/18 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/18 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/18 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/18 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/18 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name mask_rcnn_inception_resnet_v2_atrous_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 608d937df434\nStep 9/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in 763a740471f6\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting mask_rcnn_inception_resnet_v2_atrous_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/FP32 --model_name=mask_rcnn_inception_resnet_v2_atrous_coco --reverse_input_channels '--input_shape=[1,800,1365,3]' --input=image_tensor --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nmask_rcnn_inception_resnet_v2_atrous_coco\nRemoving intermediate container 763a740471f6\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build docker image\n .../pkg/containerimagebuilderpusher/docker.go:53\nFailed to build processor image\n /nuclio/pkg/processor/build/builder.go:250\nFailed to build processor image", "errCauses": [{"error": "Failed to build docker image", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/18 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/18 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/18 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/18 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/18 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/18 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/18 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name mask_rcnn_inception_resnet_v2_atrous_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 608d937df434\nStep 9/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in 763a740471f6\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting mask_rcnn_inception_resnet_v2_atrous_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/FP32 --model_name=mask_rcnn_inception_resnet_v2_atrous_coco --reverse_input_channels '--input_shape=[1,800,1365,3]' --input=image_tensor --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nmask_rcnn_inception_resnet_v2_atrous_coco\nRemoving intermediate container 763a740471f6\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build docker image\n .../pkg/containerimagebuilderpusher/docker.go:53\nFailed to build docker image", "errorCauses": [{"error": "Failed to build", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/18 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/18 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/18 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/18 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/18 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/18 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/18 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name mask_rcnn_inception_resnet_v2_atrous_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 608d937df434\nStep 9/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in 763a740471f6\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting mask_rcnn_inception_resnet_v2_atrous_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/FP32 --model_name=mask_rcnn_inception_resnet_v2_atrous_coco --reverse_input_channels '--input_shape=[1,800,1365,3]' --input=image_tensor --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nmask_rcnn_inception_resnet_v2_atrous_coco\nRemoving intermediate container 763a740471f6\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build", "errorCauses": [{"error": "stdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/18 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/18 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/18 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/18 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/18 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/18 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/18 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name mask_rcnn_inception_resnet_v2_atrous_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 608d937df434\nStep 9/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in 763a740471f6\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting mask_rcnn_inception_resnet_v2_atrous_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/FP32 --model_name=mask_rcnn_inception_resnet_v2_atrous_coco --reverse_input_channels '--input_shape=[1,800,1365,3]' --input=image_tensor --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nmask_rcnn_inception_resnet_v2_atrous_coco\nRemoving intermediate container 763a740471f6\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/18 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/18 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/18 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/18 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/18 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/18 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/18 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name mask_rcnn_inception_resnet_v2_atrous_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 608d937df434\nStep 9/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in 763a740471f6\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting mask_rcnn_inception_resnet_v2_atrous_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/FP32 --model_name=mask_rcnn_inception_resnet_v2_atrous_coco --reverse_input_channels '--input_shape=[1,800,1365,3]' --input=image_tensor --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nmask_rcnn_inception_resnet_v2_atrous_coco\nRemoving intermediate container 763a740471f6\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/18 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/18 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/18 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/18 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/18 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/18 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/18 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name mask_rcnn_inception_resnet_v2_atrous_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 608d937df434\nStep 9/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in 763a740471f6\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting mask_rcnn_inception_resnet_v2_atrous_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/FP32 --model_name=mask_rcnn_inception_resnet_v2_atrous_coco --reverse_input_channels '--input_shape=[1,800,1365,3]' --input=image_tensor --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/mask_rcnn_inception_resnet_v2_atrous_coco/mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nmask_rcnn_inception_resnet_v2_atrous_coco\nRemoving intermediate container 763a740471f6\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n", "errorCauses": [{"error": "exit status 1"}]}]}]}]}

Error - exit status 1 /nuclio/pkg/cmdrunner/shellrunner.go:96

Call stack: stdout: Sending build context to Docker daemon 44.69MB Step 1/18 : FROM openvino/ubuntu18_dev:2020.2 ---> bf7a4dff2d97 Step 2/18 : ARG NUCLIO_LABEL ---> Using cache ---> c9a10c5b4f05 Step 3/18 : ARG NUCLIO_ARCH ---> Using cache ---> 8ed29e90bf42 Step 4/18 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR ---> Using cache ---> fc2cdf7bdff6 Step 5/18 : USER root ---> Using cache ---> 44197e8724ab Step 6/18 : WORKDIR /opt/nuclio ---> Using cache ---> 4b2adc3eb6f5 Step 7/18 : RUN ln -s /usr/bin/pip3 /usr/bin/pip ---> Using cache ---> 11dc453d1c27 Step 8/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name mask_rcnn_inception_resnet_v2_atrous_coco -o /opt/nuclio/open_model_zoo ---> Using cache ---> 608d937df434 Step 9/18 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo ---> Running in 763a740471f6

FAILED: mask_rcnn_inception_resnet_v2_atrous_coco Removing intermediate container 763a740471f6

stderr: The command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name mask_rcnn_inception_resnet_v2_atrous_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1

/nuclio/pkg/cmdrunner/shellrunner.go:96

Failed to build /nuclio/pkg/dockerclient/shell.go:118 Failed to build docker image .../pkg/containerimagebuilderpusher/docker.go:53 Failed to build processor image /nuclio/pkg/processor/build/builder.go:250 Failed to deploy function ...//nuclio/pkg/platform/abstract/platform.go:182 Deploying /root/cvat/serverless/openvino/omz/public/faster_rcnn_inception_v2_coco function... 21.04.23 12:21:45.862 nuctl (I) Deploying function {"name": ""} 21.04.23 12:21:45.862 nuctl (I) Building {"versionInfo": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""} 21.04.23 12:21:46.262 nuctl (I) Cleaning up before deployment {"functionName": "openvino-omz-public-faster_rcnn_inception_v2_coco"} 21.04.23 12:21:46.309 nuctl (I) Staging files and preparing base images 21.04.23 12:21:46.310 nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"} 21.04.23 12:21:46.310 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.5.16-amd64"} 21.04.23 12:21:54.597 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"} 21.04.23 12:22:05.063 nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"} 21.04.23 12:22:11.545 nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-008327586/staging", "cmd": "docker build --network host --force-rm -t cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest -f /tmp/nuclio-build-008327586/staging/Dockerfile.processor --build-arg NUCLIO_LABEL=1.5.16 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler .", "stderr": "The command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n"} 21.04.23 12:22:11.552 nuctl (W) Failed to create a function; setting the function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 21d848ab1b94\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in df783794e387\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting faster_rcnn_inception_v2_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/FP32 --model_name=faster_rcnn_inception_v2_coco --reverse_input_channels '--input_shape=[1,600,1024,3]' --input=image_tensor --output=detection_scores,detection_boxes,num_detections --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nfaster_rcnn_inception_v2_coco\nRemoving intermediate container df783794e387\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build docker image\n .../pkg/containerimagebuilderpusher/docker.go:53\nFailed to build processor image\n /nuclio/pkg/processor/build/builder.go:250\nFailed to build processor image", "errCauses": [{"error": "Failed to build docker image", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 21d848ab1b94\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in df783794e387\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting faster_rcnn_inception_v2_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/FP32 --model_name=faster_rcnn_inception_v2_coco --reverse_input_channels '--input_shape=[1,600,1024,3]' --input=image_tensor --output=detection_scores,detection_boxes,num_detections --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nfaster_rcnn_inception_v2_coco\nRemoving intermediate container df783794e387\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build docker image\n .../pkg/containerimagebuilderpusher/docker.go:53\nFailed to build docker image", "errorCauses": [{"error": "Failed to build", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 21d848ab1b94\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in df783794e387\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting faster_rcnn_inception_v2_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/FP32 --model_name=faster_rcnn_inception_v2_coco --reverse_input_channels '--input_shape=[1,600,1024,3]' --input=image_tensor --output=detection_scores,detection_boxes,num_detections --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nfaster_rcnn_inception_v2_coco\nRemoving intermediate container df783794e387\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build", "errorCauses": [{"error": "stdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 21d848ab1b94\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in df783794e387\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting faster_rcnn_inception_v2_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/FP32 --model_name=faster_rcnn_inception_v2_coco --reverse_input_channels '--input_shape=[1,600,1024,3]' --input=image_tensor --output=detection_scores,detection_boxes,num_detections --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nfaster_rcnn_inception_v2_coco\nRemoving intermediate container df783794e387\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 21d848ab1b94\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in df783794e387\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting faster_rcnn_inception_v2_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/FP32 --model_name=faster_rcnn_inception_v2_coco --reverse_input_channels '--input_shape=[1,600,1024,3]' --input=image_tensor --output=detection_scores,detection_boxes,num_detections --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nfaster_rcnn_inception_v2_coco\nRemoving intermediate container df783794e387\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> 21d848ab1b94\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in df783794e387\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting faster_rcnn_inception_v2_coco to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/FP32 --model_name=faster_rcnn_inception_v2_coco --reverse_input_channels '--input_shape=[1,600,1024,3]' --input=image_tensor --output=detection_scores,detection_boxes,num_detections --transformations_config=/opt/intel/openvino/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support.json --tensorflow_object_detection_api_pipeline_config=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/pipeline.config --input_model=/opt/nuclio/open_model_zoo/public/faster_rcnn_inception_v2_coco/faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb\n\nFAILED:\nfaster_rcnn_inception_v2_coco\nRemoving intermediate container df783794e387\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n", "errorCauses": [{"error": "exit status 1"}]}]}]}]}

Error - exit status 1 /nuclio/pkg/cmdrunner/shellrunner.go:96

Call stack: stdout: Sending build context to Docker daemon 44.69MB Step 1/16 : FROM openvino/ubuntu18_dev:2020.2 ---> bf7a4dff2d97 Step 2/16 : ARG NUCLIO_LABEL ---> Using cache ---> c9a10c5b4f05 Step 3/16 : ARG NUCLIO_ARCH ---> Using cache ---> 8ed29e90bf42 Step 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR ---> Using cache ---> fc2cdf7bdff6 Step 5/16 : USER root ---> Using cache ---> 44197e8724ab Step 6/16 : WORKDIR /opt/nuclio ---> Using cache ---> 4b2adc3eb6f5 Step 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip ---> Using cache ---> 11dc453d1c27 Step 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo ---> Using cache ---> 21d848ab1b94 Step 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo ---> Running in df783794e387

FAILED: faster_rcnn_inception_v2_coco Removing intermediate container df783794e387

stderr: The command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1

/nuclio/pkg/cmdrunner/shellrunner.go:96

Failed to build /nuclio/pkg/dockerclient/shell.go:118 Failed to build docker image .../pkg/containerimagebuilderpusher/docker.go:53 Failed to build processor image /nuclio/pkg/processor/build/builder.go:250 Failed to deploy function ...//nuclio/pkg/platform/abstract/platform.go:182 Deploying /root/cvat/serverless/openvino/omz/public/yolo-v3-tf function... 21.04.23 12:22:12.416 nuctl (I) Deploying function {"name": ""} 21.04.23 12:22:12.417 nuctl (I) Building {"versionInfo": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""} 21.04.23 12:22:12.771 nuctl (I) Cleaning up before deployment {"functionName": "openvino-omz-public-yolo-v3-tf"} 21.04.23 12:22:12.821 nuctl (I) Staging files and preparing base images 21.04.23 12:22:12.822 nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.public.yolo-v3-tf:latest"} 21.04.23 12:22:12.822 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.5.16-amd64"} 21.04.23 12:22:20.813 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"} 21.04.23 12:22:31.319 nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.public.yolo-v3-tf:latest"} 21.04.23 12:22:37.577 nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-422481459/staging", "cmd": "docker build --network host --force-rm -t cvat/openvino.omz.public.yolo-v3-tf:latest -f /tmp/nuclio-build-422481459/staging/Dockerfile.processor --build-arg NUCLIO_LABEL=1.5.16 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler .", "stderr": "The command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n"} 21.04.23 12:22:37.583 nuctl (W) Failed to create a function; setting the function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name yolo-v3-tf -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> f6ec46de8b52\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in c3ff545e7063\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting yolo-v3-tf to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/FP32 --model_name=yolo-v3-tf '--input_shape=[1,416,416,3]' --input=input_1 '--scale_values=input_1[255]' --reverse_input_channels --transformations_config=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.json --input_model=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.pb\n\nFAILED:\nyolo-v3-tf\nRemoving intermediate container c3ff545e7063\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build docker image\n .../pkg/containerimagebuilderpusher/docker.go:53\nFailed to build processor image\n /nuclio/pkg/processor/build/builder.go:250\nFailed to build processor image", "errCauses": [{"error": "Failed to build docker image", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name yolo-v3-tf -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> f6ec46de8b52\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in c3ff545e7063\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting yolo-v3-tf to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/FP32 --model_name=yolo-v3-tf '--input_shape=[1,416,416,3]' --input=input_1 '--scale_values=input_1[255]' --reverse_input_channels --transformations_config=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.json --input_model=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.pb\n\nFAILED:\nyolo-v3-tf\nRemoving intermediate container c3ff545e7063\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build docker image\n .../pkg/containerimagebuilderpusher/docker.go:53\nFailed to build docker image", "errorCauses": [{"error": "Failed to build", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name yolo-v3-tf -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> f6ec46de8b52\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in c3ff545e7063\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting yolo-v3-tf to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/FP32 --model_name=yolo-v3-tf '--input_shape=[1,416,416,3]' --input=input_1 '--scale_values=input_1[255]' --reverse_input_channels --transformations_config=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.json --input_model=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.pb\n\nFAILED:\nyolo-v3-tf\nRemoving intermediate container c3ff545e7063\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n\n /nuclio/pkg/cmdrunner/shellrunner.go:96\nFailed to build\n /nuclio/pkg/dockerclient/shell.go:118\nFailed to build", "errorCauses": [{"error": "stdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name yolo-v3-tf -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> f6ec46de8b52\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in c3ff545e7063\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting yolo-v3-tf to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/FP32 --model_name=yolo-v3-tf '--input_shape=[1,416,416,3]' --input=input_1 '--scale_values=input_1[255]' --reverse_input_channels --transformations_config=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.json --input_model=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.pb\n\nFAILED:\nyolo-v3-tf\nRemoving intermediate container c3ff545e7063\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo' returned a non-zero code: 1\n", "errorVerbose": "\nError - exit status 1\n /nuclio/pkg/cmdrunner/shellrunner.go:96\n\nCall stack:\nstdout:\nSending build context to Docker daemon 44.69MB\r\r\nStep 1/16 : FROM openvino/ubuntu18_dev:2020.2\n ---> bf7a4dff2d97\nStep 2/16 : ARG NUCLIO_LABEL\n ---> Using cache\n ---> c9a10c5b4f05\nStep 3/16 : ARG NUCLIO_ARCH\n ---> Using cache\n ---> 8ed29e90bf42\nStep 4/16 : ARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n ---> Using cache\n ---> fc2cdf7bdff6\nStep 5/16 : USER root\n ---> Using cache\n ---> 44197e8724ab\nStep 6/16 : WORKDIR /opt/nuclio\n ---> Using cache\n ---> 4b2adc3eb6f5\nStep 7/16 : RUN ln -s /usr/bin/pip3 /usr/bin/pip\n ---> Using cache\n ---> 11dc453d1c27\nStep 8/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name yolo-v3-tf -o /opt/nuclio/open_model_zoo\n ---> Using cache\n ---> f6ec46de8b52\nStep 9/16 : RUN /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n ---> Running in c3ff545e7063\n\u001b[91m+ /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo\n\u001b[0m========= Converting yolo-v3-tf to IR (FP32)\nConversion command: /usr/bin/python3 -- /opt/intel/openvino/deployment_tools/model_optimizer/mo.py --framework=tf --data_type=FP32 --output_dir=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/FP32 --model_name=yolo-v3-tf '--input_shape=[1,416,416,3]' --input=input_1 '--scale_values=input_1[255]' --reverse_input_channels --transformations_config=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.json --input_model=/opt/nuclio/open_model_zoo/public/yolo-v3-tf/yolo-v3.pb\n\nFAILED:\nyolo-v3-tf\nRemoving intermediate container c3ff545e7063\n\nstderr:\nThe command '/bin/bash -xo pipefail -c /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name yolo-v3-tf --precisions

kiteklan commented 3 years ago

I have a proxmox-KVM system with a newer cpu and have no issues. I beleive the problem is resulted from old cpu without required features and HyperV virtualization platform.

joaoherrera commented 1 year ago

BTW, I see a similar error message for this function (in error state) in the nuclio dashboard

Error - exit status 1
    /nuclio/pkg/cmdrunner/cmdrunner.go:124

Call stack:
stdout:

stderr:
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-033503775: no such file or directory

    /nuclio/pkg/cmdrunner/cmdrunner.go:124
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:233
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:194
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:51
Failed to build processor image
    /nuclio/pkg/processor/build/builder.go:250

I had the same issue. Maybe something went wrong when deploying a new function. To solve it, remove the directory that was created at /etc/nuclio/store/functions/nuclio in the nuclio-local-storage-reader container.