search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.57k stars 3k forks source link

Issue loading openvino models for semi-automatic and automatic annotation #1977

Closed creativesiva closed 4 years ago

creativesiva commented 4 years ago

I am getting error while trying to deploy openvino and pytorch models to nuclio (serverless functions), I have used 'bash serverless/deploy.sh to deploy models. Below is the log error for semantic-segmentation-adas-0001

Error - Container wasn't healthy in time /nuclio/pkg/dockerclient/shell.go:429

Call stack: Container wasn't healthy in time /nuclio/pkg/dockerclient/shell.go:429 Function wasn't ready in time. Logs:

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

/nuclio/pkg/platform/local/platform.go:628

As a end results, it is installing i am able to deploy tensorflow models, below is the error message for rest of the models. NAMESPACE | NAME | PROJECT | STATE | NODE PORT | REPLICAS
nuclio | openvino.dextr | cvat | error | 0 | 1/1
nuclio | openvino.omz.intel.person-reidentification-retail-0300 | cvat | error | 0 | 1/1
nuclio | openvino.omz.intel.text-detection-0004 | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.faster_rcnn_inception_v2_coco | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.yolo-v3-tf | cvat | error | 0 | 1/1
nuclio | openvino.omz.semantic-segmentation-adas-0001 | cvat | error | 0 | 1/1
nuclio | tf.faster_rcnn_inception_v2_coco | cvat | ready | 39731 | 1/1
nuclio | tf.matterport.mask_rcnn | cvat | ready | 37237 | 1/1

docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ce454f5ed371 cvat/tf.faster_rcnn_inception_v2_coco:latest "processor" 44 minutes ago Up 44 minutes (healthy) 0.0.0.0:39731->8080/tcp nuclio-nuclio-tf.faster_rcnn_inception_v2_coco dde7ad667e3b cvat/tf.matterport.mask_rcnn:latest "processor" 45 minutes ago Up 45 minutes (healthy) 0.0.0.0:37237->8080/tcp nuclio-nuclio-tf.matterport.mask_rcnn 2a9b80907c35 cvat/openvino.dextr:latest "processor" 46 minutes ago Restarting (1) 44 seconds ago nuclio-nuclio-openvino.dextr 21170f4a7916 cvat/openvino.omz.intel.person-reidentification-retail-0300:latest "processor" 48 minutes ago Restarting (1) 3 seconds ago nuclio-nuclio-openvino.omz.intel.person-reidentification-retail-0300 13b9d8996523 cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest "processor" 49 minutes ago Restarting (1) 28 seconds ago nuclio-nuclio-openvino.omz.semantic-segmentation-adas-0001 81c6d6abd728 cvat/openvino.omz.intel.text-detection-0004:latest "processor" 50 minutes ago Restarting (1) 57 seconds ago nuclio-nuclio-openvino.omz.intel.text-detection-0004 13f30e51675d cvat/openvino.omz.public.yolo-v3-tf:latest "processor" 52 minutes ago Restarting (1) 20 seconds ago nuclio-nuclio-openvino.omz.public.yolo-v3-tf 4e454024a67b cvat/openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco:latest "processor" 53 minutes ago Restarting (1) 41 seconds ago nuclio-nuclio-openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco 86900a751357 cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest "processor" 55 minutes ago Restarting (1) 6 seconds ago nuclio-nuclio-openvino.omz.public.faster_rcnn_inception_v2_coco c096e1d34a9f alpine:3.11 "/bin/sh -c '/bin/sl…" 3 hours ago Up 3 hours nuclio-local-storage-reader 10761a37f44d nginx:stable-alpine "/docker-entrypoint.…" 9 hours ago Up 9 hours 0.0.0.0:8080->80/tcp cvat_proxy 888326330e63 cvat/server "/usr/bin/supervisord" 9 hours ago Up 9 hours 8080/tcp, 8443/tcp cvat a82bd68fdccd cvat_logstash "/usr/local/bin/dock…" 9 hours ago Up 9 hours 5000/tcp, 5044/tcp, 9600/tcp cvat_logstash a89ce4f4ad30 cvat_kibana "/usr/local/bin/kiba…" 9 hours ago Up 9 hours 5601/tcp cvat_kibana 1f55c7d21943 cvat_elasticsearch "/usr/local/bin/dock…" 9 hours ago Up 9 hours 9200/tcp, 9300/tcp cvat_elasticsearch 4117c788c958 cvat/ui "/docker-entrypoint.…" 9 hours ago Up 9 hours 80/tcp cvat_ui 0c50074a51c5 quay.io/nuclio/dashboard:1.4.8-amd64 "sh -c ./runner.sh" 9 hours ago Up 9 hours 80/tcp, 0.0.0.0:8070->8070/tcp nuclio

nuctl version nuctl-1.4.14-linux-amd64

nmanovic commented 4 years ago

@creativesiva , could you please provide the full deployment log for functions? Something like I did below:

``` 20.08.03 16:49:26.992 nuctl (I) Deploying function {"name": ""} 20.08.03 16:49:26.993 nuctl (I) Building {"versionInfo": "Label: 1.4.8, Git commit: 238d4539ac7783896d6c414535d0462b5f4cbcf1, OS: darwin, Arch: amd64, Go version: go1.14.3", "name": ""} 20.08.03 16:49:27.199 nuctl (I) Cleaning up before deployment 20.08.03 16:49:27.284 nuctl (I) Function already exists, deleting 20.08.03 16:49:29.836 nuctl (I) Staging files and preparing base images 20.08.03 16:49:29.840 nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"} 20.08.03 16:49:29.840 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.8-amd64"} 20.08.03 16:49:35.541 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"} 20.08.03 16:49:39.253 nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"} 20.08.03 16:49:40.483 nuctl.platform (I) Pushing docker image into registry {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest", "registry": ""} 20.08.03 16:49:40.483 nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"} 20.08.03 16:49:40.483 nuctl (I) Build complete {"result": {"Image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest","UpdatedFunctionConfig":{"metadata":{"name":"openvino.omz.public.faster_rcnn_inception_v2_coco","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"openvino","name":"Faster RCNN","spec":"[\n { \"id\": 1, \"name\": \"person\" },\n { \"id\": 2, \"name\": \"bicycle\" },\n { \"id\": 3, \"name\": \"car\" },\n { \"id\": 4, \"name\": \"motorcycle\" },\n { \"id\": 5, \"name\": \"airplane\" },\n { \"id\": 6, \"name\": \"bus\" },\n { \"id\": 7, \"name\": \"train\" },\n { \"id\": 8, \"name\": \"truck\" },\n { \"id\": 9, \"name\": \"boat\" },\n { \"id\":10, \"name\": \"traffic_light\" },\n { \"id\":11, \"name\": \"fire_hydrant\" },\n { \"id\":13, \"name\": \"stop_sign\" },\n { \"id\":14, \"name\": \"parking_meter\" },\n { \"id\":15, \"name\": \"bench\" },\n { \"id\":16, \"name\": \"bird\" },\n { \"id\":17, \"name\": \"cat\" },\n { \"id\":18, \"name\": \"dog\" },\n { \"id\":19, \"name\": \"horse\" },\n { \"id\":20, \"name\": \"sheep\" },\n { \"id\":21, \"name\": \"cow\" },\n { \"id\":22, \"name\": \"elephant\" },\n { \"id\":23, \"name\": \"bear\" },\n { \"id\":24, \"name\": \"zebra\" },\n { \"id\":25, \"name\": \"giraffe\" },\n { \"id\":27, \"name\": \"backpack\" },\n { \"id\":28, \"name\": \"umbrella\" },\n { \"id\":31, \"name\": \"handbag\" },\n { \"id\":32, \"name\": \"tie\" },\n { \"id\":33, \"name\": \"suitcase\" },\n { \"id\":34, \"name\": \"frisbee\" },\n { \"id\":35, \"name\": \"skis\" },\n { \"id\":36, \"name\": \"snowboard\" },\n { \"id\":37, \"name\": \"sports_ball\" },\n { \"id\":38, \"name\": \"kite\" },\n { \"id\":39, \"name\": \"baseball_bat\" },\n { \"id\":40, \"name\": \"baseball_glove\" },\n { \"id\":41, \"name\": \"skateboard\" },\n { \"id\":42, \"name\": \"surfboard\" },\n { \"id\":43, \"name\": \"tennis_racket\" },\n { \"id\":44, \"name\": \"bottle\" },\n { \"id\":46, \"name\": \"wine_glass\" },\n { \"id\":47, \"name\": \"cup\" },\n { \"id\":48, \"name\": \"fork\" },\n { \"id\":49, \"name\": \"knife\" },\n { \"id\":50, \"name\": \"spoon\" },\n { \"id\":51, \"name\": \"bowl\" },\n { \"id\":52, \"name\": \"banana\" },\n { \"id\":53, \"name\": \"apple\" },\n { \"id\":54, \"name\": \"sandwich\" },\n { \"id\":55, \"name\": \"orange\" },\n { \"id\":56, \"name\": \"broccoli\" },\n { \"id\":57, \"name\": \"carrot\" },\n { \"id\":58, \"name\": \"hot_dog\" },\n { \"id\":59, \"name\": \"pizza\" },\n { \"id\":60, \"name\": \"donut\" },\n { \"id\":61, \"name\": \"cake\" },\n { \"id\":62, \"name\": \"chair\" },\n { \"id\":63, \"name\": \"couch\" },\n { \"id\":64, \"name\": \"potted_plant\" },\n { \"id\":65, \"name\": \"bed\" },\n { \"id\":67, \"name\": \"dining_table\" },\n { \"id\":70, \"name\": \"toilet\" },\n { \"id\":72, \"name\": \"tv\" },\n { \"id\":73, \"name\": \"laptop\" },\n { \"id\":74, \"name\": \"mouse\" },\n { \"id\":75, \"name\": \"remote\" },\n { \"id\":76, \"name\": \"keyboard\" },\n { \"id\":77, \"name\": \"cell_phone\" },\n { \"id\":78, \"name\": \"microwave\" },\n { \"id\":79, \"name\": \"oven\" },\n { \"id\":80, \"name\": \"toaster\" },\n { \"id\":81, \"name\": \"sink\" },\n { \"id\":83, \"name\": \"refrigerator\" },\n { \"id\":84, \"name\": \"book\" },\n { \"id\":85, \"name\": \"clock\" },\n { \"id\":86, \"name\": \"vase\" },\n { \"id\":87, \"name\": \"scissors\" },\n { \"id\":88, \"name\": \"teddy_bear\" },\n { \"id\":89, \"name\": \"hair_drier\" },\n { \"id\":90, \"name\": \"toothbrush\" }\n]\n","type":"detector"}},"spec":{"description":"Faster RCNN inception v2 COCO via Intel OpenVINO toolkit","handler":"main:handler","runtime":"python:3.6","env":[{"name":"NUCLIO_PYTHON_EXE_PATH","value":"/opt/nuclio/common/python3"}],"resources":{},"image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/Users/nmanovic/Workspace/cvat/serverless/openvino/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco","baseImage":"openvino/ubuntu18_dev:2020.2","directives":{"postCopy":[{"kind":"USER","value":"openvino"}],"preCopy":[{"kind":"USER","value":"root"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/bin/pip"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo"}]},"codeEntryType":"image"},"platform":{"attributes":{"restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"eventTimeout":"30s"}}}} 20.08.03 16:49:40.908 nuctl.platform (I) Waiting for function to be ready {"timeout": 60} 20.08.03 16:49:43.053 nuctl (I) Function deploy complete {"httpPort": 54616} NAMESPACE | NAME | PROJECT | STATE | NODE PORT | REPLICAS nuclio | openvino.omz.intel.text-detection-0004 | cvat | ready | 54900 | 1/1 nuclio | openvino.omz.public.faster_rcnn_inception_v2_coco | cvat | ready | 54616 | 1/1 nuclio | openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco | cvat | ready | 54716 | 1/1 nuclio | openvino.omz.public.yolo-v3-tf | cvat | error | 57308 | 1/1 nuclio | openvino.omz.semantic-segmentation-adas-0001 | cvat | ready | 54992 | 1/1 nuclio | tf.faster_rcnn_inception_v2_coco | cvat | ready | 55372 | 1/1 ```
creativesiva commented 4 years ago

@nmanovic , Below is the deployment log.

error_log.txt

20.08.03 22:26:11.277 nuctl.platform (I) Waiting for function to be ready {"timeout": 60} 20.08.03 22:26:12.568 nuctl (I) Function deploy complete {"functionName": "tf.matterport.mask_rcnn", "httpPort": 37237} 20.08.03 22:26:13.531 nuctl (I) Deploying function {"name": ""} 20.08.03 22:26:13.532 nuctl (I) Building {"versionInfo": "Label: 1.4.14, Git commit: e10e9fb123caafbe4f95310a0e2ccfc71368ba87, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""} 20.08.03 22:26:13.767 nuctl (I) Cleaning up before deployment 20.08.03 22:26:13.880 nuctl (I) Function already exists, deleting 20.08.03 22:26:15.712 nuctl (I) Staging files and preparing base images 20.08.03 22:26:15.714 nuctl (I) Building processor image {"imageName": "cvat/tf.faster_rcnn_inception_v2_coco:latest"} 20.08.03 22:26:15.714 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.14-amd64"} 20.08.03 22:26:22.932 nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"} 20.08.03 22:26:34.942 nuctl.platform (I) Building docker image {"image": "cvat/tf.faster_rcnn_inception_v2_coco:latest"} 20.08.03 22:26:35.301 nuctl.platform (I) Pushing docker image into registry {"image": "cvat/tf.faster_rcnn_inception_v2_coco:latest", "registry": ""} 20.08.03 22:26:35.301 nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/tf.faster_rcnn_inception_v2_coco:latest"} 20.08.03 22:26:35.301 nuctl (I) Build complete {"result": {"Image":"cvat/tf.faster_rcnn_inception_v2_coco:latest","UpdatedFunctionConfig":{"metadata":{"name":"tf.faster_rcnn_inception_v2_coco","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"tensorflow","name":"Faster RCNN via Tensorflow","spec":"[\n { \"id\": 1, \"name\": \"person\" },\n { \"id\": 2, \"name\": \"bicycle\" },\n { \"id\": 3, \"name\": \"car\" },\n { \"id\": 4, \"name\": \"motorcycle\" },\n { \"id\": 5, \"name\": \"airplane\" },\n { \"id\": 6, \"name\": \"bus\" },\n { \"id\": 7, \"name\": \"train\" },\n { \"id\": 8, \"name\": \"truck\" },\n { \"id\": 9, \"name\": \"boat\" },\n { \"id\":10, \"name\": \"traffic_light\" },\n { \"id\":11, \"name\": \"fire_hydrant\" },\n { \"id\":13, \"name\": \"stop_sign\" },\n { \"id\":14, \"name\": \"parking_meter\" },\n { \"id\":15, \"name\": \"bench\" },\n { \"id\":16, \"name\": \"bird\" },\n { \"id\":17, \"name\": \"cat\" },\n { \"id\":18, \"name\": \"dog\" },\n { \"id\":19, \"name\": \"horse\" },\n { \"id\":20, \"name\": \"sheep\" },\n { \"id\":21, \"name\": \"cow\" },\n { \"id\":22, \"name\": \"elephant\" },\n { \"id\":23, \"name\": \"bear\" },\n { \"id\":24, \"name\": \"zebra\" },\n { \"id\":25, \"name\": \"giraffe\" },\n { \"id\":27, \"name\": \"backpack\" },\n { \"id\":28, \"name\": \"umbrella\" },\n { \"id\":31, \"name\": \"handbag\" },\n { \"id\":32, \"name\": \"tie\" },\n { \"id\":33, \"name\": \"suitcase\" },\n { \"id\":34, \"name\": \"frisbee\" },\n { \"id\":35, \"name\": \"skis\" },\n { \"id\":36, \"name\": \"snowboard\" },\n { \"id\":37, \"name\": \"sports_ball\" },\n { \"id\":38, \"name\": \"kite\" },\n { \"id\":39, \"name\": \"baseball_bat\" },\n { \"id\":40, \"name\": \"baseball_glove\" },\n { \"id\":41, \"name\": \"skateboard\" },\n { \"id\":42, \"name\": \"surfboard\" },\n { \"id\":43, \"name\": \"tennis_racket\" },\n { \"id\":44, \"name\": \"bottle\" },\n { \"id\":46, \"name\": \"wine_glass\" },\n { \"id\":47, \"name\": \"cup\" },\n { \"id\":48, \"name\": \"fork\" },\n { \"id\":49, \"name\": \"knife\" },\n { \"id\":50, \"name\": \"spoon\" },\n { \"id\":51, \"name\": \"bowl\" },\n { \"id\":52, \"name\": \"banana\" },\n { \"id\":53, \"name\": \"apple\" },\n { \"id\":54, \"name\": \"sandwich\" },\n { \"id\":55, \"name\": \"orange\" },\n { \"id\":56, \"name\": \"broccoli\" },\n { \"id\":57, \"name\": \"carrot\" },\n { \"id\":58, \"name\": \"hot_dog\" },\n { \"id\":59, \"name\": \"pizza\" },\n { \"id\":60, \"name\": \"donut\" },\n { \"id\":61, \"name\": \"cake\" },\n { \"id\":62, \"name\": \"chair\" },\n { \"id\":63, \"name\": \"couch\" },\n { \"id\":64, \"name\": \"potted_plant\" },\n { \"id\":65, \"name\": \"bed\" },\n { \"id\":67, \"name\": \"dining_table\" },\n { \"id\":70, \"name\": \"toilet\" },\n { \"id\":72, \"name\": \"tv\" },\n { \"id\":73, \"name\": \"laptop\" },\n { \"id\":74, \"name\": \"mouse\" },\n { \"id\":75, \"name\": \"remote\" },\n { \"id\":76, \"name\": \"keyboard\" },\n { \"id\":77, \"name\": \"cell_phone\" },\n { \"id\":78, \"name\": \"microwave\" },\n { \"id\":79, \"name\": \"oven\" },\n { \"id\":80, \"name\": \"toaster\" },\n { \"id\":81, \"name\": \"sink\" },\n { \"id\":83, \"name\": \"refrigerator\" },\n { \"id\":84, \"name\": \"book\" },\n { \"id\":85, \"name\": \"clock\" },\n { \"id\":86, \"name\": \"vase\" },\n { \"id\":87, \"name\": \"scissors\" },\n { \"id\":88, \"name\": \"teddy_bear\" },\n { \"id\":89, \"name\": \"hair_drier\" },\n { \"id\":90, \"name\": \"toothbrush\" }\n]\n","type":"detector"}},"spec":{"description":"Faster RCNN from Tensorflow Object Detection API","handler":"main:handler","runtime":"python:3.6","resources":{},"image":"cvat/tf.faster_rcnn_inception_v2_coco:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"build":{"image":"cvat/tf.faster_rcnn_inception_v2_coco","baseImage":"tensorflow/tensorflow:2.1.1","directives":{"postCopy":[{"kind":"RUN","value":"curl -O http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_v2_coco_2018_01_28.tar.gz"},{"kind":"RUN","value":"tar -xzf faster_rcnn_inception_v2_coco_2018_01_28.tar.gz \u0026\u0026 rm faster_rcnn_inception_v2_coco_2018_01_28.tar.gz"},{"kind":"RUN","value":"ln -s faster_rcnn_inception_v2_coco_2018_01_28 faster_rcnn"},{"kind":"RUN","value":"pip install pillow pyyaml"}],"preCopy":[{"kind":"RUN","value":"apt install curl"},{"kind":"WORKDIR","value":"/opt/nuclio"}]},"codeEntryType":"image"},"platform":{"attributes":{"restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"eventTimeout":"30s"}}}} 20.08.03 22:26:37.812 nuctl.platform (I) Waiting for function to be ready {"timeout": 60} 20.08.03 22:26:39.205 nuctl (I) Function deploy complete {"functionName": "tf.faster_rcnn_inception_v2_coco", "httpPort": 39731} NAMESPACE | NAME | PROJECT | STATE | NODE PORT | REPLICAS
nuclio | openvino.dextr | cvat | error | 0 | 1/1
nuclio | openvino.omz.intel.person-reidentification-retail-0300 | cvat | error | 0 | 1/1
nuclio | openvino.omz.intel.text-detection-0004 | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.faster_rcnn_inception_v2_coco | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco | cvat | error | 0 | 1/1
nuclio | openvino.omz.public.yolo-v3-tf | cvat | error | 0 | 1/1
nuclio | openvino.omz.semantic-segmentation-adas-0001 | cvat | error | 0 | 1/1
nuclio | tf.faster_rcnn_inception_v2_coco | cvat | ready | 39731 | 1/1
nuclio | tf.matterport.mask_rcnn | cvat | ready | 37237 | 1/1

nmanovic commented 4 years ago

@creativesiva , I have updated the documentation. Could you please add --platform local command line argument when you deploy a function (deploy.sh already has the additional argument)? Please notify me if it solves the problem.

For further investigation please add --verbose flag to a nuclio deploy command. It will provide additional information.

Also it looks like need more information about your system. Which OS do you use? Please provide as much details about your system as possible. For example, docker version.

AJ-RR commented 4 years ago

I have the same issue. I am using Ubuntu 18.04.1 and docker version 19.03.6. Adding --platform local command line argument does not solve the issue.

nmanovic commented 4 years ago

@AJ-RR , could you please also publish your deployment log?

creativesiva commented 4 years ago

@creativesiva , I have updated the documentation. Could you please add --platform local command line argument when you deploy a function (deploy.sh already has the additional argument)? Please notify me if it solves the problem. For further investigation please add --verbose flag to a nuclio deploy command. It will provide additional information.

Also it looks like need more information about your system. Which OS do you use? Please provide as much details about your system as possible. For example, docker version.

nuctl_deploy_verbose_log.txt

I have tried with --platfom local, Hard Luck. OS:Ubuntu 18.04.4, Docker-version: 19.03.12

AJ-RR commented 4 years ago

deployment_log.txt @nmanovic, Here it is Error - Container wasn't healthy in time /nuclio/pkg/dockerclient/shell.go:429

Call stack: Container wasn't healthy in time /nuclio/pkg/dockerclient/shell.go:429 Function wasn't ready in time. Logs:

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied /nuclio/cmd/processor/app/processor.go:265

Call stack: Failed to open configuration file /nuclio/cmd/processor/app/processor.go:265

/nuclio/pkg/platform/local/platform.go:628

Failed to deploy function ...//nuclio/pkg/platform/abstract/platform.go:171

nmanovic commented 4 years ago

@creativesiva , we are investigating the issue. I cannot reproduce it on my Linux and Mac machines. But an internal team also reported the problem. I know that nuclio guys also are investigating the issue. I hope to find a solution soon. If you can help and investigate on your end, it will be perfect.

nmanovic commented 4 years ago

@creativesiva , could you please try a workaround in PR #1996?

creativesiva commented 4 years ago

@creativesiva , could you please try a workaround in PR #1996?

@nmanovic , Please suggest how can i check this!!, I am not sure, what to do!

creativesiva commented 4 years ago

I understand that you have removed "- kind: USER value: openvino" from function.yml and did the same in my local repository. Now it is showing different error. my Command sudo ./nuctl deploy --project-name cvat --path serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio --volume serverless/openvino/common:/opt/nuclio/common --platform local Error Log

20.08.06 23:13:21.888                     nuctl (I) Deploying function {"name": ""}
20.08.06 23:13:21.888                     nuctl (I) Building {"versionInfo": "Label: 1.4.14, Git commit: e10e9fb123caafbe4f95310a0e2ccfc71368ba87, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
20.08.06 23:13:22.146                     nuctl (I) Cleaning up before deployment
20.08.06 23:13:22.209                     nuctl (I) Staging files and preparing base images
20.08.06 23:13:22.210                     nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest"}
20.08.06 23:13:22.210     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.14-amd64"}
20.08.06 23:13:33.632     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
20.08.06 23:13:46.855            nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest"}
20.08.06 23:13:47.649            nuctl.platform (I) Pushing docker image into registry {"image": "cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest", "registry": ""}
20.08.06 23:13:47.649            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest"}
20.08.06 23:13:47.649                     nuctl (I) Build complete {"result": {"Image":"cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest","UpdatedFunctionConfig":{"metadata":{"name":"openvino.omz.semantic-segmentation-adas-0001","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"openvino","name":"Semantic segmentation for ADAS","spec":"[\n  { \"id\": 0, \"name\": \"road\" },\n  { \"id\": 1, \"name\": \"sidewalk\" },\n  { \"id\": 2, \"name\": \"building\" },\n  { \"id\": 3, \"name\": \"wall\" },\n  { \"id\": 4, \"name\": \"fence\" },\n  { \"id\": 5, \"name\": \"pole\" },\n  { \"id\": 6, \"name\": \"traffic light\" },\n  { \"id\": 7, \"name\": \"traffic sign\" },\n  { \"id\": 8, \"name\": \"vegetation\" },\n  { \"id\": 9, \"name\": \"terrain\" },\n  { \"id\": 10, \"name\": \"sky\" },\n  { \"id\": 11, \"name\": \"person\" },\n  { \"id\": 12, \"name\": \"rider\" },\n  { \"id\": 13, \"name\": \"car\" },\n  { \"id\": 14, \"name\": \"truck\" },\n  { \"id\": 15, \"name\": \"bus\" },\n  { \"id\": 16, \"name\": \"train\" },\n  { \"id\": 17, \"name\": \"motorcycle\" },\n  { \"id\": 18, \"name\": \"bicycle\" },\n  { \"id\": 19, \"name\": \"ego-vehicle\" },\n  { \"id\": 20, \"name\": \"background\" }\n]\n","type":"detector"}},"spec":{"description":"Segmentation network to classify each pixel into typical 20 classes for ADAS","handler":"main:handler","runtime":"python:3.6","env":[{"name":"NUCLIO_PYTHON_EXE_PATH","value":"/opt/nuclio/common/python3"}],"resources":{},"image":"cvat/openvino.omz.intel.semantic-segmentation-adas-0001:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"serverless/openvino/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat/openvino.omz.intel.semantic-segmentation-adas-0001","baseImage":"openvino/ubuntu18_dev:2020.2","directives":{"postCopy":[{"kind":"RUN","value":"apt update \u0026\u0026 DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y python3-skimage"},{"kind":"RUN","value":"pip3 install \"numpy\u003c1.16.0\""}],"preCopy":[{"kind":"USER","value":"root"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/bin/pip"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name semantic-segmentation-adas-0001 -o /opt/nuclio/open_model_zoo"}]},"codeEntryType":"image"},"platform":{"attributes":{"restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"eventTimeout":"30s"}}}}
20.08.06 23:13:47.979     nuctl.platform.docker (W) Failed to run container {"err": "stdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n", "errVerbose": "\nError - exit status 125\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n", "errCauses": [{"error": "exit status 125"}], "stdout": "docker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n", "stderr": ""}
20.08.06 23:13:47.979                     nuctl (W) Create function failed, setting function status {"err": "Failed to run docker container", "errVerbose": "\nError - exit status 125\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to run docker container\n    /nuclio/pkg/platform/local/platform.go:605\nFailed to run docker container", "errCauses": [{"error": "stdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n", "errorVerbose": "\nError - exit status 125\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nstdout:\ndocker: Error response from daemon: create serverless/openvino/common: \"serverless/openvino/common\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path.\nSee 'docker run --help'.\n\nstderr:\n", "errorCauses": [{"error": "exit status 125"}]}]}

Error - exit status 125
    /nuclio/pkg/cmdrunner/cmdrunner.go:124

Call stack:
stdout:
docker: Error response from daemon: create serverless/openvino/common: "serverless/openvino/common" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.

stderr:

    /nuclio/pkg/cmdrunner/cmdrunner.go:124
Failed to run docker container
    /nuclio/pkg/platform/local/platform.go:605
Failed to deploy function
    ...//nuclio/pkg/platform/abstract/platform.go:171
nmanovic commented 4 years ago

@creativesiva , the mentioned deploy command isn't correct. The error message is clear here: If you intended to pass a host directory, use absolute path. For volumes you have to use absolute paths: ./nuctl deploy --project-name cvat --path serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio --volume <abspath>/serverless/openvino/common:/opt/nuclio/common --platform local

Also you don't need to use sudo if your docker setup is correct (you are in docker group).

creativesiva commented 4 years ago

@creativesiva , the mentioned deploy command isn't correct. The error message is clear here: If you intended to pass a host directory, use absolute path. For volumes you have to use absolute paths: ./nuctl deploy --project-name cvat --path serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio --volume <abspath>/serverless/openvino/common:/opt/nuclio/common --platform local

Also you don't need to use sudo if your docker setup is correct (you are in docker group).

@nmanovic, Thanks. The model deployment is successful now. (I have not made changes to text-detection function, that is why error, rest all functions are modified as per your suggestion)

Thanks a lot. 

  NAMESPACE |                             NAME                              | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino.dextr                                                | cvat    | ready |     46667 | 1/1       
  nuclio    | openvino.omz.intel.person-reidentification-retail-0300        | cvat    | ready |     35367 | 1/1       
  nuclio    | openvino.omz.intel.text-detection-0004                        | cvat    | error |         0 | 1/1       
  nuclio    | openvino.omz.public.faster_rcnn_inception_v2_coco             | cvat    | ready |     34845 | 1/1       
  nuclio    | openvino.omz.public.mask_rcnn_inception_resnet_v2_atrous_coco | cvat    | ready |     36551 | 1/1       
  nuclio    | openvino.omz.public.yolo-v3-tf                                | cvat    | ready |     37233 | 1/1       
  nuclio    | openvino.omz.semantic-segmentation-adas-0001                  | cvat    | ready |     33293 | 1/1       
  nuclio    | tf.faster_rcnn_inception_v2_coco                              | cvat    | ready |     39731 | 1/1       
  nuclio    | tf.matterport.mask_rcnn                                       | cvat    | ready |     37237 | 1/1
nmanovic commented 4 years ago

@creativesiva , could you please help me to test a fix in nuclio? Below I have attached nuctl for linux with a fix which just adds necessary permissions to /tmp/processor-config-* file (https://github.com/liranbg/nuclio/pull/22). Please revert all changes inside CVAT and try the binary to deploy functions. Does it work? I cannot check on my end unfortunately.

nuctl-latest-linux-amd64.zip

creativesiva commented 4 years ago

@nmanovic , i have reverted function.yml changes and used nuctl-latest-linux-amd64 to run the deploy.sh. It is throwing error. Error Log

20.08.07 21:18:49.114                     nuctl (I) Deploying function {"name": ""}
20.08.07 21:18:49.114                     nuctl (I) Building {"versionInfo": "Label: 1.4.14, Git commit: e10e9fb123caafbe4f95310a0e2ccfc71368ba87, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
20.08.07 21:18:49.364                     nuctl (I) Cleaning up before deployment
20.08.07 21:18:49.475                     nuctl (I) Function already exists, deleting
20.08.07 21:18:50.691                     nuctl (I) Staging files and preparing base images
20.08.07 21:18:50.692                     nuctl (I) Building processor image {"imageName": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.07 21:18:50.692     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.14-amd64"}
20.08.07 21:18:57.453     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
20.08.07 21:19:10.244            nuctl.platform (I) Building docker image {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.07 21:19:27.725            nuctl.platform (I) Pushing docker image into registry {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest", "registry": ""}
20.08.07 21:19:27.725            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest"}
20.08.07 21:19:27.725                     nuctl (I) Build complete {"result": {"Image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest","UpdatedFunctionConfig":{"metadata":{"name":"openvino.omz.public.faster_rcnn_inception_v2_coco","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"framework":"openvino","name":"Faster RCNN","spec":"[\n  { \"id\": 1, \"name\": \"person\" },\n  { \"id\": 2, \"name\": \"bicycle\" },\n  { \"id\": 3, \"name\": \"car\" },\n  { \"id\": 4, \"name\": \"motorcycle\" },\n  { \"id\": 5, \"name\": \"airplane\" },\n  { \"id\": 6, \"name\": \"bus\" },\n  { \"id\": 7, \"name\": \"train\" },\n  { \"id\": 8, \"name\": \"truck\" },\n  { \"id\": 9, \"name\": \"boat\" },\n  { \"id\":10, \"name\": \"traffic_light\" },\n  { \"id\":11, \"name\": \"fire_hydrant\" },\n  { \"id\":13, \"name\": \"stop_sign\" },\n  { \"id\":14, \"name\": \"parking_meter\" },\n  { \"id\":15, \"name\": \"bench\" },\n  { \"id\":16, \"name\": \"bird\" },\n  { \"id\":17, \"name\": \"cat\" },\n  { \"id\":18, \"name\": \"dog\" },\n  { \"id\":19, \"name\": \"horse\" },\n  { \"id\":20, \"name\": \"sheep\" },\n  { \"id\":21, \"name\": \"cow\" },\n  { \"id\":22, \"name\": \"elephant\" },\n  { \"id\":23, \"name\": \"bear\" },\n  { \"id\":24, \"name\": \"zebra\" },\n  { \"id\":25, \"name\": \"giraffe\" },\n  { \"id\":27, \"name\": \"backpack\" },\n  { \"id\":28, \"name\": \"umbrella\" },\n  { \"id\":31, \"name\": \"handbag\" },\n  { \"id\":32, \"name\": \"tie\" },\n  { \"id\":33, \"name\": \"suitcase\" },\n  { \"id\":34, \"name\": \"frisbee\" },\n  { \"id\":35, \"name\": \"skis\" },\n  { \"id\":36, \"name\": \"snowboard\" },\n  { \"id\":37, \"name\": \"sports_ball\" },\n  { \"id\":38, \"name\": \"kite\" },\n  { \"id\":39, \"name\": \"baseball_bat\" },\n  { \"id\":40, \"name\": \"baseball_glove\" },\n  { \"id\":41, \"name\": \"skateboard\" },\n  { \"id\":42, \"name\": \"surfboard\" },\n  { \"id\":43, \"name\": \"tennis_racket\" },\n  { \"id\":44, \"name\": \"bottle\" },\n  { \"id\":46, \"name\": \"wine_glass\" },\n  { \"id\":47, \"name\": \"cup\" },\n  { \"id\":48, \"name\": \"fork\" },\n  { \"id\":49, \"name\": \"knife\" },\n  { \"id\":50, \"name\": \"spoon\" },\n  { \"id\":51, \"name\": \"bowl\" },\n  { \"id\":52, \"name\": \"banana\" },\n  { \"id\":53, \"name\": \"apple\" },\n  { \"id\":54, \"name\": \"sandwich\" },\n  { \"id\":55, \"name\": \"orange\" },\n  { \"id\":56, \"name\": \"broccoli\" },\n  { \"id\":57, \"name\": \"carrot\" },\n  { \"id\":58, \"name\": \"hot_dog\" },\n  { \"id\":59, \"name\": \"pizza\" },\n  { \"id\":60, \"name\": \"donut\" },\n  { \"id\":61, \"name\": \"cake\" },\n  { \"id\":62, \"name\": \"chair\" },\n  { \"id\":63, \"name\": \"couch\" },\n  { \"id\":64, \"name\": \"potted_plant\" },\n  { \"id\":65, \"name\": \"bed\" },\n  { \"id\":67, \"name\": \"dining_table\" },\n  { \"id\":70, \"name\": \"toilet\" },\n  { \"id\":72, \"name\": \"tv\" },\n  { \"id\":73, \"name\": \"laptop\" },\n  { \"id\":74, \"name\": \"mouse\" },\n  { \"id\":75, \"name\": \"remote\" },\n  { \"id\":76, \"name\": \"keyboard\" },\n  { \"id\":77, \"name\": \"cell_phone\" },\n  { \"id\":78, \"name\": \"microwave\" },\n  { \"id\":79, \"name\": \"oven\" },\n  { \"id\":80, \"name\": \"toaster\" },\n  { \"id\":81, \"name\": \"sink\" },\n  { \"id\":83, \"name\": \"refrigerator\" },\n  { \"id\":84, \"name\": \"book\" },\n  { \"id\":85, \"name\": \"clock\" },\n  { \"id\":86, \"name\": \"vase\" },\n  { \"id\":87, \"name\": \"scissors\" },\n  { \"id\":88, \"name\": \"teddy_bear\" },\n  { \"id\":89, \"name\": \"hair_drier\" },\n  { \"id\":90, \"name\": \"toothbrush\" }\n]\n","type":"detector"}},"spec":{"description":"Faster RCNN inception v2 COCO via Intel OpenVINO toolkit","handler":"main:handler","runtime":"python:3.6","env":[{"name":"NUCLIO_PYTHON_EXE_PATH","value":"/opt/nuclio/common/python3"}],"resources":{},"image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/home/aipl/cvat/serverless/openvino/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat/openvino.omz.public.faster_rcnn_inception_v2_coco","baseImage":"openvino/ubuntu18_dev:2020.2","directives":{"postCopy":[{"kind":"USER","value":"openvino"}],"preCopy":[{"kind":"USER","value":"root"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/bin/pip"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name faster_rcnn_inception_v2_coco -o /opt/nuclio/open_model_zoo"},{"kind":"RUN","value":"/opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/converter.py --name faster_rcnn_inception_v2_coco --precisions FP32 -d /opt/nuclio/open_model_zoo -o /opt/nuclio/open_model_zoo"}]},"codeEntryType":"image"},"platform":{"attributes":{"restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":60,"eventTimeout":"30s"}}}}
20.08.07 21:19:30.374            nuctl.platform (I) Waiting for function to be ready {"timeout": 60}
20.08.07 21:20:30.423     nuctl.platform.docker (W) Container wasn't healthy within timeout {"containerID": "8280740ef19cc90d2aedb2b962cf1d14c73b36fe42e0fb59578b5cef1cb3d5be", "timeout": "1m0s", "logs": "\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n"}
20.08.07 21:20:30.473                     nuctl (W) Create function failed, setting function status {"err": "Function wasn't ready in time. Logs:\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n", "errVerbose": "\nError - Container wasn't healthy in time\n    /nuclio/pkg/dockerclient/shell.go:429\n\nCall stack:\nContainer wasn't healthy in time\n    /nuclio/pkg/dockerclient/shell.go:429\nFunction wasn't ready in time. Logs:\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\n    /nuclio/pkg/platform/local/platform.go:628\nFunction wasn't ready in time. Logs:\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n\nError - open /etc/nuclio/config/processor/processor.yaml: permission denied\n    /nuclio/cmd/processor/app/processor.go:265\n\nCall stack:\nFailed to open configuration file\n    /nuclio/cmd/processor/app/processor.go:265\n", "errCauses": [{"error": "Container wasn't healthy in time", "errorVerbose": "\nError - Container wasn't healthy in time\n    /nuclio/pkg/dockerclient/shell.go:429\n\nCall stack:\nContainer wasn't healthy in time\n    /nuclio/pkg/dockerclient/shell.go:429\nContainer wasn't healthy in time", "errorCauses": [{}]}]}

Error - Container wasn't healthy in time
    /nuclio/pkg/dockerclient/shell.go:429

Call stack:
Container wasn't healthy in time
    /nuclio/pkg/dockerclient/shell.go:429
Function wasn't ready in time. Logs:

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

Error - open /etc/nuclio/config/processor/processor.yaml: permission denied
    /nuclio/cmd/processor/app/processor.go:265

Call stack:
Failed to open configuration file
    /nuclio/cmd/processor/app/processor.go:265

    /nuclio/pkg/platform/local/platform.go:628
Failed to deploy function
    ...//nuclio/pkg/platform/abstract/platform.go:171
nmanovic commented 4 years ago

@creativesiva , thanks for the info. I have merged the PR. Now it should work. I hope nuclio team will be able to reproduce the issue and fix in future releases.

nmanovic commented 4 years ago

@creativesiva , are you sure that you used the attached binary? At least Git commit e10e9fb123caafbe4f95310a0e2ccfc71368ba87 in your logs looks strange.

creativesiva commented 4 years ago

@nmanovic Sorry i did not noticed your question. Yes, i have used (nuctl-latest-linux-amd64.zip)

thangvip4321 commented 4 years ago

@nmanovic i have updated my code, but the error still persist. Here's the log:

I'm running on Ubuntu 18.04, docker version : 19.03.11 , nuctl: 1.4.16. log_cvat.txt

nmanovic commented 4 years ago

@thangvip4321 , indeed it is another issue: Error - read /etc/nuclio/config/processor/processor.yaml: is a directory

nmanovic commented 4 years ago

@thangvip4321 , let me check with nuctl 1.4.16 if I can reproduce the issue.

nmanovic commented 4 years ago

@thangvip4321 , I cannot reproduce the problem. Need your help.