Run AI tools Interactors like f-BRS, HRNet, Inside-Outside-Guidance on local installed CVAT

medphisiker commented 1 year ago

My actions before raising this issue

[ ] I install CVAT on my Linux Ubuntu 20.04 server by this instruction link to docs. Then create a supervisor user and login. CVAT works perfectly.
[ ] I read docs about CVAT AI Tools AI tools. I was very interested in Interactors for Semi-automatic segmentation like DEXTR, f-BRS, HRNet, Inside-Outside-Guidance.
[ ] I follow guide in this page guide how install and run AI tools. I successfully launched DEXTR and it worked. For DEXTR running we have this commands:

nuctl deploy --project-name cvat \
  --path serverless/openvino/dextr/nuclio \
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

It is upload a special docker image with DEXTR, run it, and then CVAT can interact with it.

But how I can run other interesting CVAT interactions like f-BRS, HRNet, Inside-Outside-Guidance ?

Expected Behaviour

I think it will be great to have commands like for DEXTR for running other CVAT Interactors (f-BRS, HRNet, Inside-Outside-Guidance). Probably it will be convenient to have such commands on docs page about AI Tools AI tools. And probably place these commands under each of the interactor, detector or tracker.

Current Behaviour

We have very good docs about DEXTR Interactor running, but it is not so clear how to run the rest (f-BRS, HRNet, Inside-Outside-Guidance).

Possible Solution

It seems to me it would be convenient to specify them as suggested in the Expected Behavior clause.

Steps to Reproduce (for bugs)

This is not a bug, but a question =)

Context

Everything is fine, but I really want to try the other integrators. After using the MiVOS project link to git repo for markup, I was very pleased with such tools. By the way, it would be very cool to add such an interactor. However, its complexity lies in the fact that it is both an interactive segmentation and a tracker at once.

Your Environment

I have everything working according to the instructions. I just wanted to ask how to run the rest of the interactors like f-BRS, HRNet, Inside-Outside-Guidance.

zhiltsov-max commented 1 year ago

Hi, the model configs are placed in other directories in the serverless/ directory. To run another model, you need to run the same command with a modified --path directory. For instance, f-BRS and HRNet in the serverless/pytorch/saic-vul/.

medphisiker commented 1 year ago

Thank you for fast reply )

I tried to run the models interactors on CPU according to your recommendations. The base commands pattern was taken from DEXTR:

 nuctl deploy --project-name cvat \
  --path <interartor_path>\
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

where for dextr == serverless/openvino/dextr/nuclio # it works but give error it the terminal for f-BRS == serverless/pytorch/saic-vul/fbrs/nuclio # it works but give error it the terminal for Inside-Outside-Guidance == serverless/pytorch/shiyinzhang/iog/nuclio # it works but give error it the terminal for HRnet == serverless/pytorch/saic-vul/hrnet/nuclio # it doesn't work

Error in the terminal presented here:

(base) admin-gpu@server-gpu:~/cvat$  nuctl deploy --project-name cvat \
>   --path serverless/pytorch/shiyinzhang/iog/nuclio \
>   --volume `pwd`/serverless/common:/opt/nuclio/common \
>   --platform local
22.12.04 09:44:25.555                     nuctl (I) Deploying function {"name": ""}
22.12.04 09:44:25.555                     nuctl (I) Building {"builderKind": "docker", "versionInfo": "Label: 1.8.14, Git commit: cbb0774230996a3eb4621c1a2079e2317578005b, OS: linux, Arch: amd64, Go version: go1.17.8", "name": ""}
22.12.04 09:44:25.691                     nuctl (I) Staging files and preparing base images
22.12.04 09:44:25.692                     nuctl (I) Building processor image {"registryURL": "", "imageName": "cvat.pth.shiyinzhang.iog:latest"}
22.12.04 09:44:25.692     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.8.14-amd64"}
22.12.04 09:44:29.653     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/uhttpc:0.0.1-amd64"}
22.12.04 09:44:36.825            nuctl.platform (I) Building docker image {"image": "cvat.pth.shiyinzhang.iog:latest"}
22.12.04 09:44:36.975            nuctl.platform (I) Pushing docker image into registry {"image": "cvat.pth.shiyinzhang.iog:latest", "registry": ""}
22.12.04 09:44:36.975            nuctl.platform (I) Docker image was successfully built and pushed into docker registry {"image": "cvat.pth.shiyinzhang.iog:latest"}
22.12.04 09:44:36.975                     nuctl (I) Build complete {"result": {"Image":"cvat.pth.shiyinzhang.iog:latest","UpdatedFunctionConfig":{"metadata":{"name":"pth.shiyinzhang.iog","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"animated_gif":"https://raw.githubusercontent.com/opencv/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/iog_example.gif","framework":"pytorch","help_message":"The interactor allows to get a mask of an object using its wrapping boundig box, positive, and negative points inside it","min_neg_points":"0","min_pos_points":"1","name":"IOG","spec":"","startswith_box":"true","type":"interactor","version":"2"}},"spec":{"description":"Interactive Object Segmentation with Inside-Outside Guidance","handler":"main:handler","runtime":"python:3.9","env":[{"name":"PYTHONPATH","value":"/opt/nuclio/iog"}],"resources":{"requests":{"cpu":"25m","memory":"1Mi"}},"image":"cvat.pth.shiyinzhang.iog:latest","targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"myHttpTrigger","maxWorkers":2,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432}}},"volumes":[{"volume":{"name":"volume-1","hostPath":{"path":"/home/admin-gpu/cvat/serverless/common"}},"volumeMount":{"name":"volume-1","mountPath":"/opt/nuclio/common"}}],"build":{"image":"cvat.pth.shiyinzhang.iog","baseImage":"continuumio/miniconda3","directives":{"preCopy":[{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"RUN","value":"conda create -y -n iog python=3.9"},{"kind":"SHELL","value":"[\"conda\", \"run\", \"-n\", \"iog\", \"/bin/bash\", \"-c\"]"},{"kind":"RUN","value":"conda install -y -c anaconda curl"},{"kind":"RUN","value":"conda install -y pytorch torchvision -c pytorch"},{"kind":"RUN","value":"apt update && apt install -y libgl1-mesa-glx"},{"kind":"RUN","value":"conda install -y -c conda-forge pycocotools scipy"},{"kind":"RUN","value":"pip install opencv-python"},{"kind":"RUN","value":"git clone https://github.com/shiyinzhang/Inside-Outside-Guidance.git iog"},{"kind":"WORKDIR","value":"/opt/nuclio/iog"},{"kind":"ENV","value":"fileid=1Lm1hhMhhjjnNwO4Pf7SC6tXLayH2iH0l"},{"kind":"ENV","value":"filename=IOG_PASCAL_SBD.pth"},{"kind":"RUN","value":"curl -L \"https://drive.google.com/uc?export=download&confirm=t&id=${fileid}\" -o ${filename}"},{"kind":"WORKDIR","value":"/opt/nuclio"},{"kind":"ENTRYPOINT","value":"[\"conda\", \"run\", \"-n\", \"iog\"]"}]},"codeEntryType":"image"},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":120,"securityContext":{},"eventTimeout":"30s"}}}}
22.12.04 09:44:36.979                     nuctl (I) Cleaning up before deployment {"functionName": "pth.shiyinzhang.iog"}
22.12.04 09:44:37.776            nuctl.platform (I) Waiting for function to be ready {"timeout": 120}
22.12.04 09:44:39.511                     nuctl (I) Function deploy complete {"functionName": "pth.shiyinzhang.iog", "httpPort": 49166, "internalInvocationURLs": ["172.17.0.4:8080"], "externalInvocationURLs": []}

Error - exit status 1
    /nuclio/pkg/cmdrunner/shellrunner.go:96

Call stack:
stdout:
cat: read error: Is a directory
eyJtZXRhZGF0YSI6eyJuYW1lIjoib3BlbnZpbm8tZGV4dHIiLCJuYW1lc3BhY2UiOiJudWNsaW8iLCJsYWJlbHMiOnsibnVjbGlvLmlvL3Byb2plY3QtbmFtZSI6ImN2YXQifSwiYW5ub3RhdGlvbnMiOnsiYW5pbWF0ZWRfZ2lmIjoiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL29wZW5jdi9jdmF0LzBmYmIxOWFlMzg0NmEwMTc4NTNkNTJlMTg3ZjBjZTE0OWFkY2VkN2Qvc2l0ZS9jb250ZW50L2VuL2ltYWdlcy9kZXh0cl9leGFtcGxlLmdpZiIsImZyYW1ld29yayI6Im9wZW52aW5vIiwiaGVscF9tZXNzYWdlIjoiVGhlIGludGVyYWN0b3IgYWxsb3dzIHRvIGdldCBhIG1hc2sgb2YgYW4gb2JqZWN0IHVzaW5nIGl0cyBleHRyZW1lIHBvaW50cyAobW9yZSBvciBlcXVhbCB0aGFuIDQpLiBZb3UgY2FuIGFkZCBhIHBvaW50IGxlZnQtY2xpY2tpbmcgdGhlIGltYWdlIiwibWluX3Bvc19wb2ludHMiOiI0IiwibmFtZSI6IkRFWFRSIiwic3BlYyI6IiIsInR5cGUiOiJpbnRlcmFjdG9yIiwidmVyc2lvbiI6IjIifX0sInNwZWMiOnsiZGVzY3JpcHRpb24iOiJEZWVwIEV4dHJlbWUgQ3V0IiwiaGFuZGxlciI6Im1haW46aGFuZGxlciIsInJ1bnRpbWUiOiJweXRob246My42IiwiZW52IjpbeyJuYW1lIjoiTlVDTElPX1BZVEhPTl9FWEVfUEFUSCIsInZhbHVlIjoiL29wdC9udWNsaW8vY29tbW9uL29wZW52aW5vL3B5dGhvbjMifV0sInJlc291cmNlcyI6eyJsaW1pdHMiOnsibnZpZGlhLmNvbS9ncHUiOiIxIn0sInJlcXVlc3RzIjp7ImNwdSI6IjI1bSIsIm1lbW9yeSI6IjFNaSJ9fSwiaW1hZ2UiOiJjdmF0L29wZW52aW5vLmRleHRyOmxhdGVzdCIsInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibWF4V29ya2VycyI6MSwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn19fSwidm9sdW1lcyI6W3sidm9sdW1lIjp7Im5hbWUiOiJ2b2x1bWUtMSIsImhvc3RQYXRoIjp7InBhdGgiOiIvaG9tZS9hZG1pbi1ncHUvY3ZhdC9zZXJ2ZXJsZXNzL2NvbW1vbiJ9fSwidm9sdW1lTW91bnQiOnsibmFtZSI6InZvbHVtZS0xIiwibW91bnRQYXRoIjoiL29wdC9udWNsaW8vY29tbW9uIn19XSwiYnVpbGQiOnsiaW1hZ2UiOiJjdmF0L29wZW52aW5vLmRleHRyIiwiYmFzZUltYWdlIjoib3BlbnZpbm8vdWJ1bnR1MThfcnVudGltZToyMDIwLjIiLCJkaXJlY3RpdmVzIjp7InBvc3RDb3B5IjpbeyJraW5kIjoiUlVOIiwidmFsdWUiOiJjdXJsIC1PIGh0dHBzOi8vZG93bmxvYWQuMDEub3JnL29wZW52aW5vdG9vbGtpdC9tb2RlbHNfY29udHJpYi9jdmF0L2RleHRyX21vZGVsX3YxLnppcCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoidW56aXAgZGV4dHJfbW9kZWxfdjEuemlwIn0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJwaXAzIGluc3RhbGwgLVUgcGlwIFx1MDAyNlx1MDAyNiBwaXAzIGluc3RhbGwgd2hlZWwgUGlsbG93In1dLCJwcmVDb3B5IjpbeyJraW5kIjoiVVNFUiIsInZhbHVlIjoicm9vdCJ9LHsia2luZCI6IldPUktESVIiLCJ2YWx1ZSI6Ii9vcHQvbnVjbGlvIn0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJsbiAtcyAvdXNyL2Jpbi9waXAzIC91c3IvYmluL3BpcCJ9XX0sImNvZGVFbnRyeVR5cGUiOiJpbWFnZSIsInRpbWVzdGFtcCI6MTY3MDE0NTY0Nn0sInBsYXRmb3JtIjp7ImF0dHJpYnV0ZXMiOnsibW91bnRNb2RlIjoidm9sdW1lIiwicmVzdGFydFBvbGljeSI6eyJtYXhpbXVtUmV0cnlDb3VudCI6MywibmFtZSI6ImFsd2F5cyJ9fX0sInJlYWRpbmVzc1RpbWVvdXRTZWNvbmRzIjoxMjAsInNlY3VyaXR5Q29udGV4dCI6e30sImV2ZW50VGltZW91dCI6IjMwcyJ9LCJzdGF0dXMiOnsic3RhdGUiOiJyZWFkeSIsImh0dHBQb3J0Ijo0OTE2NCwiaW50ZXJuYWxJbnZvY2F0aW9uVXJscyI6WyIxNzIuMTcuMC40OjgwODAiXX19
eyJtZXRhZGF0YSI6eyJuYW1lIjoicHRoLXNhaWMtdnVsLWZicnMiLCJuYW1lc3BhY2UiOiJudWNsaW8iLCJsYWJlbHMiOnsibnVjbGlvLmlvL3Byb2plY3QtbmFtZSI6ImN2YXQifSwiYW5ub3RhdGlvbnMiOnsiYW5pbWF0ZWRfZ2lmIjoiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL29wZW5jdi9jdmF0LzBmYmIxOWFlMzg0NmEwMTc4NTNkNTJlMTg3ZjBjZTE0OWFkY2VkN2Qvc2l0ZS9jb250ZW50L2VuL2ltYWdlcy9mYnJzX2V4YW1wbGUuZ2lmIiwiZnJhbWV3b3JrIjoicHl0b3JjaCIsImhlbHBfbWVzc2FnZSI6IlRoZSBpbnRlcmFjdG9yIGFsbG93cyB0byBnZXQgYSBtYXNrIGZvciBhbiBvYmplY3QgdXNpbmcgcG9zaXRpdmUgcG9pbnRzLCBhbmQgbmVnYXRpdmUgcG9pbnRzIiwibWluX25lZ19wb2ludHMiOiIwIiwibWluX3Bvc19wb2ludHMiOiIxIiwibmFtZSI6ImYtQlJTIiwic3BlYyI6IiIsInR5cGUiOiJpbnRlcmFjdG9yIiwidmVyc2lvbiI6IjIifX0sInNwZWMiOnsiZGVzY3JpcHRpb24iOiJmLUJSUyBpbnRlcmFjdGl2ZSBzZWdtZW50YXRpb24iLCJoYW5kbGVyIjoibWFpbjpoYW5kbGVyIiwicnVudGltZSI6InB5dGhvbjozLjYiLCJlbnYiOlt7Im5hbWUiOiJQWVRIT05QQVRIIiwidmFsdWUiOiIvb3B0L251Y2xpby9mYnJzIn1dLCJyZXNvdXJjZXMiOnsicmVxdWVzdHMiOnsiY3B1IjoiMjVtIiwibWVtb3J5IjoiMU1pIn19LCJpbWFnZSI6ImN2YXQucHRoLnNhaWMtdnVsLmZicnM6bGF0ZXN0IiwidGFyZ2V0Q1BVIjo3NSwidHJpZ2dlcnMiOnsibXlIdHRwVHJpZ2dlciI6eyJjbGFzcyI6IiIsImtpbmQiOiJodHRwIiwibmFtZSI6Im15SHR0cFRyaWdnZXIiLCJtYXhXb3JrZXJzIjoyLCJ3b3JrZXJBdmFpbGFiaWxpdHlUaW1lb3V0TWlsbGlzZWNvbmRzIjoxMDAwMCwiYXR0cmlidXRlcyI6eyJtYXhSZXF1ZXN0Qm9keVNpemUiOjMzNTU0NDMyfX19LCJ2b2x1bWVzIjpbeyJ2b2x1bWUiOnsibmFtZSI6InZvbHVtZS0xIiwiaG9zdFBhdGgiOnsicGF0aCI6Ii9ob21lL2FkbWluLWdwdS9jdmF0L3NlcnZlcmxlc3MvY29tbW9uIn19LCJ2b2x1bWVNb3VudCI6eyJuYW1lIjoidm9sdW1lLTEiLCJtb3VudFBhdGgiOiIvb3B0L251Y2xpby9jb21tb24ifX1dLCJidWlsZCI6eyJpbWFnZSI6ImN2YXQucHRoLnNhaWMtdnVsLmZicnMiLCJiYXNlSW1hZ2UiOiJweXRob246My42LjExIiwiZGlyZWN0aXZlcyI6eyJwcmVDb3B5IjpbeyJraW5kIjoiV09SS0RJUiIsInZhbHVlIjoiL29wdC9udWNsaW8ifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImdpdCBjbG9uZSBodHRwczovL2dpdGh1Yi5jb20vc2FpYy12dWwvZmJyc19pbnRlcmFjdGl2ZV9zZWdtZW50YXRpb24uZ2l0IGZicnMifSx7ImtpbmQiOiJXT1JLRElSIiwidmFsdWUiOiIvb3B0L251Y2xpby9mYnJzIn0seyJraW5kIjoiRU5WIiwidmFsdWUiOiJmaWxlaWQ9MVo5ZFF0cFdWVG9iRWRtVUJudHBVVTBwSmwtcEVYVXdSIn0seyJraW5kIjoiRU5WIiwidmFsdWUiOiJmaWxlbmFtZT1yZXNuZXQxMDFfZGgyNTZfc2JkLnB0aCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiY3VybCAtTCBcImh0dHBzOi8vZHJpdmUuZ29vZ2xlLmNvbS91Yz9leHBvcnQ9ZG93bmxvYWRcdTAwMjZjb25maXJtPXRcdTAwMjZpZD0ke2ZpbGVpZH1cIiAtbyAke2ZpbGVuYW1lfSJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiYXB0IHVwZGF0ZSBcdTAwMjZcdTAwMjYgYXB0IGluc3RhbGwgLXkgbGliZ2wxLW1lc2EtZ2x4In0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJwaXAzIGluc3RhbGwgLXIgcmVxdWlyZW1lbnRzLnR4dCJ9LHsia2luZCI6IldPUktESVIiLCJ2YWx1ZSI6Ii9vcHQvbnVjbGlvIn1dfSwiY29kZUVudHJ5VHlwZSI6ImltYWdlIiwidGltZXN0YW1wIjoxNjcwMTQ2OTY4fSwicGxhdGZvcm0iOnsiYXR0cmlidXRlcyI6eyJtb3VudE1vZGUiOiJ2b2x1bWUiLCJyZXN0YXJ0UG9saWN5Ijp7Im1heGltdW1SZXRyeUNvdW50IjozLCJuYW1lIjoiYWx3YXlzIn19fSwicmVhZGluZXNzVGltZW91dFNlY29uZHMiOjEyMCwic2VjdXJpdHlDb250ZXh0Ijp7fSwiZXZlbnRUaW1lb3V0IjoiMzBzIn0sInN0YXR1cyI6eyJzdGF0ZSI6InJlYWR5IiwiaHR0cFBvcnQiOjQ5MTY1LCJpbnRlcm5hbEludm9jYXRpb25VcmxzIjpbIjE3Mi4xNy4wLjQ6ODA4MCJdfX0=
eyJtZXRhZGF0YSI6eyJuYW1lIjoicHRoLnNoaXlpbnpoYW5nLmlvZyIsIm5hbWVzcGFjZSI6Im51Y2xpbyIsImxhYmVscyI6eyJudWNsaW8uaW8vcHJvamVjdC1uYW1lIjoiY3ZhdCJ9LCJhbm5vdGF0aW9ucyI6eyJhbmltYXRlZF9naWYiOiJodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vb3BlbmN2L2N2YXQvMGZiYjE5YWUzODQ2YTAxNzg1M2Q1MmUxODdmMGNlMTQ5YWRjZWQ3ZC9zaXRlL2NvbnRlbnQvZW4vaW1hZ2VzL2lvZ19leGFtcGxlLmdpZiIsImZyYW1ld29yayI6InB5dG9yY2giLCJoZWxwX21lc3NhZ2UiOiJUaGUgaW50ZXJhY3RvciBhbGxvd3MgdG8gZ2V0IGEgbWFzayBvZiBhbiBvYmplY3QgdXNpbmcgaXRzIHdyYXBwaW5nIGJvdW5kaWcgYm94LCBwb3NpdGl2ZSwgYW5kIG5lZ2F0aXZlIHBvaW50cyBpbnNpZGUgaXQiLCJtaW5fbmVnX3BvaW50cyI6IjAiLCJtaW5fcG9zX3BvaW50cyI6IjEiLCJuYW1lIjoiSU9HIiwic3BlYyI6IiIsInN0YXJ0c3dpdGhfYm94IjoidHJ1ZSIsInR5cGUiOiJpbnRlcmFjdG9yIiwidmVyc2lvbiI6IjIifX0sInNwZWMiOnsiZGVzY3JpcHRpb24iOiJJbnRlcmFjdGl2ZSBPYmplY3QgU2VnbWVudGF0aW9uIHdpdGggSW5zaWRlLU91dHNpZGUgR3VpZGFuY2UiLCJoYW5kbGVyIjoibWFpbjpoYW5kbGVyIiwicnVudGltZSI6InB5dGhvbjozLjkiLCJlbnYiOlt7Im5hbWUiOiJQWVRIT05QQVRIIiwidmFsdWUiOiIvb3B0L251Y2xpby9pb2cifV0sInJlc291cmNlcyI6eyJyZXF1ZXN0cyI6eyJjcHUiOiIyNW0iLCJtZW1vcnkiOiIxTWkifX0sImltYWdlIjoiY3ZhdC5wdGguc2hpeWluemhhbmcuaW9nOmxhdGVzdCIsInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibWF4V29ya2VycyI6Miwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn19fSwidm9sdW1lcyI6W3sidm9sdW1lIjp7Im5hbWUiOiJ2b2x1bWUtMSIsImhvc3RQYXRoIjp7InBhdGgiOiIvaG9tZS9hZG1pbi1ncHUvY3ZhdC9zZXJ2ZXJsZXNzL2NvbW1vbiJ9fSwidm9sdW1lTW91bnQiOnsibmFtZSI6InZvbHVtZS0xIiwibW91bnRQYXRoIjoiL29wdC9udWNsaW8vY29tbW9uIn19XSwiYnVpbGQiOnsiaW1hZ2UiOiJjdmF0LnB0aC5zaGl5aW56aGFuZy5pb2ciLCJiYXNlSW1hZ2UiOiJjb250aW51dW1pby9taW5pY29uZGEzIiwiZGlyZWN0aXZlcyI6eyJwcmVDb3B5IjpbeyJraW5kIjoiV09SS0RJUiIsInZhbHVlIjoiL29wdC9udWNsaW8ifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImNvbmRhIGNyZWF0ZSAteSAtbiBpb2cgcHl0aG9uPTMuOSJ9LHsia2luZCI6IlNIRUxMIiwidmFsdWUiOiJbXCJjb25kYVwiLCBcInJ1blwiLCBcIi1uXCIsIFwiaW9nXCIsIFwiL2Jpbi9iYXNoXCIsIFwiLWNcIl0ifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImNvbmRhIGluc3RhbGwgLXkgLWMgYW5hY29uZGEgY3VybCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiY29uZGEgaW5zdGFsbCAteSBweXRvcmNoIHRvcmNodmlzaW9uIC1jIHB5dG9yY2gifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImFwdCB1cGRhdGUgXHUwMDI2XHUwMDI2IGFwdCBpbnN0YWxsIC15IGxpYmdsMS1tZXNhLWdseCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiY29uZGEgaW5zdGFsbCAteSAtYyBjb25kYS1mb3JnZSBweWNvY290b29scyBzY2lweSJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoicGlwIGluc3RhbGwgb3BlbmN2LXB5dGhvbiJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiZ2l0IGNsb25lIGh0dHBzOi8vZ2l0aHViLmNvbS9zaGl5aW56aGFuZy9JbnNpZGUtT3V0c2lkZS1HdWlkYW5jZS5naXQgaW9nIn0seyJraW5kIjoiV09SS0RJUiIsInZhbHVlIjoiL29wdC9udWNsaW8vaW9nIn0seyJraW5kIjoiRU5WIiwidmFsdWUiOiJmaWxlaWQ9MUxtMWhoTWhoampuTndPNFBmN1NDNnRYTGF5SDJpSDBsIn0seyJraW5kIjoiRU5WIiwidmFsdWUiOiJmaWxlbmFtZT1JT0dfUEFTQ0FMX1NCRC5wdGgifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImN1cmwgLUwgXCJodHRwczovL2RyaXZlLmdvb2dsZS5jb20vdWM/ZXhwb3J0PWRvd25sb2FkXHUwMDI2Y29uZmlybT10XHUwMDI2aWQ9JHtmaWxlaWR9XCIgLW8gJHtmaWxlbmFtZX0ifSx7ImtpbmQiOiJXT1JLRElSIiwidmFsdWUiOiIvb3B0L251Y2xpbyJ9LHsia2luZCI6IkVOVFJZUE9JTlQiLCJ2YWx1ZSI6IltcImNvbmRhXCIsIFwicnVuXCIsIFwiLW5cIiwgXCJpb2dcIl0ifV19LCJjb2RlRW50cnlUeXBlIjoiaW1hZ2UiLCJ0aW1lc3RhbXAiOjE2NzAxNDcwNzZ9LCJwbGF0Zm9ybSI6eyJhdHRyaWJ1dGVzIjp7Im1vdW50TW9kZSI6InZvbHVtZSIsInJlc3RhcnRQb2xpY3kiOnsibWF4aW11bVJldHJ5Q291bnQiOjMsIm5hbWUiOiJhbHdheXMifX19LCJyZWFkaW5lc3NUaW1lb3V0U2Vjb25kcyI6MTIwLCJzZWN1cml0eUNvbnRleHQiOnt9LCJldmVudFRpbWVvdXQiOiIzMHMifSwic3RhdHVzIjp7InN0YXRlIjoicmVhZHkiLCJodHRwUG9ydCI6NDkxNjYsImludGVybmFsSW52b2NhdGlvblVybHMiOlsiMTcyLjE3LjAuNDo4MDgwIl19fQ==
eyJtZXRhZGF0YSI6eyJuYW1lIjoidGYtbWF0dGVycG9ydC1tYXNrLXJjbm4iLCJuYW1lc3BhY2UiOiJudWNsaW8iLCJsYWJlbHMiOnsibnVjbGlvLmlvL3Byb2plY3QtbmFtZSI6ImN2YXQifSwiYW5ub3RhdGlvbnMiOnsiZnJhbWV3b3JrIjoidGVuc29yZmxvdyIsIm5hbWUiOiJNYXNrIFJDTk4gdmlhIFRlbnNvcmZsb3ciLCJzcGVjIjoiW1xuICB7IFwiaWRcIjogMCwgXCJuYW1lXCI6IFwiQkdcIiB9LFxuICB7IFwiaWRcIjogMSwgXCJuYW1lXCI6IFwicGVyc29uXCIgfSxcbiAgeyBcImlkXCI6IDIsIFwibmFtZVwiOiBcImJpY3ljbGVcIiB9LFxuICB7IFwiaWRcIjogMywgXCJuYW1lXCI6IFwiY2FyXCIgfSxcbiAgeyBcImlkXCI6IDQsIFwibmFtZVwiOiBcIm1vdG9yY3ljbGVcIiB9LFxuICB7IFwiaWRcIjogNSwgXCJuYW1lXCI6IFwiYWlycGxhbmVcIiB9LFxuICB7IFwiaWRcIjogNiwgXCJuYW1lXCI6IFwiYnVzXCIgfSxcbiAgeyBcImlkXCI6IDcsIFwibmFtZVwiOiBcInRyYWluXCIgfSxcbiAgeyBcImlkXCI6IDgsIFwibmFtZVwiOiBcInRydWNrXCIgfSxcbiAgeyBcImlkXCI6IDksIFwibmFtZVwiOiBcImJvYXRcIiB9LFxuICB7IFwiaWRcIjogMTAsIFwibmFtZVwiOiBcInRyYWZmaWNfbGlnaHRcIiB9LFxuICB7IFwiaWRcIjogMTEsIFwibmFtZVwiOiBcImZpcmVfaHlkcmFudFwiIH0sXG4gIHsgXCJpZFwiOiAxMiwgXCJuYW1lXCI6IFwic3RvcF9zaWduXCIgfSxcbiAgeyBcImlkXCI6IDEzLCBcIm5hbWVcIjogXCJwYXJraW5nX21ldGVyXCIgfSxcbiAgeyBcImlkXCI6IDE0LCBcIm5hbWVcIjogXCJiZW5jaFwiIH0sXG4gIHsgXCJpZFwiOiAxNSwgXCJuYW1lXCI6IFwiYmlyZFwiIH0sXG4gIHsgXCJpZFwiOiAxNiwgXCJuYW1lXCI6IFwiY2F0XCIgfSxcbiAgeyBcImlkXCI6IDE3LCBcIm5hbWVcIjogXCJkb2dcIiB9LFxuICB7IFwiaWRcIjogMTgsIFwibmFtZVwiOiBcImhvcnNlXCIgfSxcbiAgeyBcImlkXCI6IDE5LCBcIm5hbWVcIjogXCJzaGVlcFwiIH0sXG4gIHsgXCJpZFwiOiAyMCwgXCJuYW1lXCI6IFwiY293XCIgfSxcbiAgeyBcImlkXCI6IDIxLCBcIm5hbWVcIjogXCJlbGVwaGFudFwiIH0sXG4gIHsgXCJpZFwiOiAyMiwgXCJuYW1lXCI6IFwiYmVhclwiIH0sXG4gIHsgXCJpZFwiOiAyMywgXCJuYW1lXCI6IFwiemVicmFcIiB9LFxuICB7IFwiaWRcIjogMjQsIFwibmFtZVwiOiBcImdpcmFmZmVcIiB9LFxuICB7IFwiaWRcIjogMjUsIFwibmFtZVwiOiBcImJhY2twYWNrXCIgfSxcbiAgeyBcImlkXCI6IDI2LCBcIm5hbWVcIjogXCJ1bWJyZWxsYVwiIH0sXG4gIHsgXCJpZFwiOiAyNywgXCJuYW1lXCI6IFwiaGFuZGJhZ1wiIH0sXG4gIHsgXCJpZFwiOiAyOCwgXCJuYW1lXCI6IFwidGllXCIgfSxcbiAgeyBcImlkXCI6IDI5LCBcIm5hbWVcIjogXCJzdWl0Y2FzZVwiIH0sXG4gIHsgXCJpZFwiOiAzMCwgXCJuYW1lXCI6IFwiZnJpc2JlZVwiIH0sXG4gIHsgXCJpZFwiOiAzMSwgXCJuYW1lXCI6IFwic2tpc1wiIH0sXG4gIHsgXCJpZFwiOiAzMiwgXCJuYW1lXCI6IFwic25vd2JvYXJkXCIgfSxcbiAgeyBcImlkXCI6IDMzLCBcIm5hbWVcIjogXCJzcG9ydHNfYmFsbFwiIH0sXG4gIHsgXCJpZFwiOiAzNCwgXCJuYW1lXCI6IFwia2l0ZVwiIH0sXG4gIHsgXCJpZFwiOiAzNSwgXCJuYW1lXCI6IFwiYmFzZWJhbGxfYmF0XCIgfSxcbiAgeyBcImlkXCI6IDM2LCBcIm5hbWVcIjogXCJiYXNlYmFsbF9nbG92ZVwiIH0sXG4gIHsgXCJpZFwiOiAzNywgXCJuYW1lXCI6IFwic2thdGVib2FyZFwiIH0sXG4gIHsgXCJpZFwiOiAzOCwgXCJuYW1lXCI6IFwic3VyZmJvYXJkXCIgfSxcbiAgeyBcImlkXCI6IDM5LCBcIm5hbWVcIjogXCJ0ZW5uaXNfcmFja2V0XCIgfSxcbiAgeyBcImlkXCI6IDQwLCBcIm5hbWVcIjogXCJib3R0bGVcIiB9LFxuICB7IFwiaWRcIjogNDEsIFwibmFtZVwiOiBcIndpbmVfZ2xhc3NcIiB9LFxuICB7IFwiaWRcIjogNDIsIFwibmFtZVwiOiBcImN1cFwiIH0sXG4gIHsgXCJpZFwiOiA0MywgXCJuYW1lXCI6IFwiZm9ya1wiIH0sXG4gIHsgXCJpZFwiOiA0NCwgXCJuYW1lXCI6IFwia25pZmVcIiB9LFxuICB7IFwiaWRcIjogNDUsIFwibmFtZVwiOiBcInNwb29uXCIgfSxcbiAgeyBcImlkXCI6IDQ2LCBcIm5hbWVcIjogXCJib3dsXCIgfSxcbiAgeyBcImlkXCI6IDQ3LCBcIm5hbWVcIjogXCJiYW5hbmFcIiB9LFxuICB7IFwiaWRcIjogNDgsIFwibmFtZVwiOiBcImFwcGxlXCIgfSxcbiAgeyBcImlkXCI6IDQ5LCBcIm5hbWVcIjogXCJzYW5kd2ljaFwiIH0sXG4gIHsgXCJpZFwiOiA1MCwgXCJuYW1lXCI6IFwib3JhbmdlXCIgfSxcbiAgeyBcImlkXCI6IDUxLCBcIm5hbWVcIjogXCJicm9jY29saVwiIH0sXG4gIHsgXCJpZFwiOiA1MiwgXCJuYW1lXCI6IFwiY2Fycm90XCIgfSxcbiAgeyBcImlkXCI6IDUzLCBcIm5hbWVcIjogXCJob3RfZG9nXCIgfSxcbiAgeyBcImlkXCI6IDU0LCBcIm5hbWVcIjogXCJwaXp6YVwiIH0sXG4gIHsgXCJpZFwiOiA1NSwgXCJuYW1lXCI6IFwiZG9udXRcIiB9LFxuICB7IFwiaWRcIjogNTYsIFwibmFtZVwiOiBcImNha2VcIiB9LFxuICB7IFwiaWRcIjogNTcsIFwibmFtZVwiOiBcImNoYWlyXCIgfSxcbiAgeyBcImlkXCI6IDU4LCBcIm5hbWVcIjogXCJjb3VjaFwiIH0sXG4gIHsgXCJpZFwiOiA1OSwgXCJuYW1lXCI6IFwicG90dGVkX3BsYW50XCIgfSxcbiAgeyBcImlkXCI6IDYwLCBcIm5hbWVcIjogXCJiZWRcIiB9LFxuICB7IFwiaWRcIjogNjEsIFwibmFtZVwiOiBcImRpbmluZ190YWJsZVwiIH0sXG4gIHsgXCJpZFwiOiA2MiwgXCJuYW1lXCI6IFwidG9pbGV0XCIgfSxcbiAgeyBcImlkXCI6IDYzLCBcIm5hbWVcIjogXCJ0dlwiIH0sXG4gIHsgXCJpZFwiOiA2NCwgXCJuYW1lXCI6IFwibGFwdG9wXCIgfSxcbiAgeyBcImlkXCI6IDY1LCBcIm5hbWVcIjogXCJtb3VzZVwiIH0sXG4gIHsgXCJpZFwiOiA2NiwgXCJuYW1lXCI6IFwicmVtb3RlXCIgfSxcbiAgeyBcImlkXCI6IDY3LCBcIm5hbWVcIjogXCJrZXlib2FyZFwiIH0sXG4gIHsgXCJpZFwiOiA2OCwgXCJuYW1lXCI6IFwiY2VsbF9waG9uZVwiIH0sXG4gIHsgXCJpZFwiOiA2OSwgXCJuYW1lXCI6IFwibWljcm93YXZlXCIgfSxcbiAgeyBcImlkXCI6IDcwLCBcIm5hbWVcIjogXCJvdmVuXCIgfSxcbiAgeyBcImlkXCI6IDcxLCBcIm5hbWVcIjogXCJ0b2FzdGVyXCIgfSxcbiAgeyBcImlkXCI6IDcyLCBcIm5hbWVcIjogXCJzaW5rXCIgfSxcbiAgeyBcImlkXCI6IDczLCBcIm5hbWVcIjogXCJyZWZyaWdlcmF0b3JcIiB9LFxuICB7IFwiaWRcIjogNzQsIFwibmFtZVwiOiBcImJvb2tcIiB9LFxuICB7IFwiaWRcIjogNzUsIFwibmFtZVwiOiBcImNsb2NrXCIgfSxcbiAgeyBcImlkXCI6IDc2LCBcIm5hbWVcIjogXCJ2YXNlXCIgfSxcbiAgeyBcImlkXCI6IDc3LCBcIm5hbWVcIjogXCJzY2lzc29yc1wiIH0sXG4gIHsgXCJpZFwiOiA3OCwgXCJuYW1lXCI6IFwidGVkZHlfYmVhclwiIH0sXG4gIHsgXCJpZFwiOiA3OSwgXCJuYW1lXCI6IFwiaGFpcl9kcmllclwiIH0sXG4gIHsgXCJpZFwiOiA4MCwgXCJuYW1lXCI6IFwidG9vdGhicnVzaFwiIH1cbl1cbiIsInR5cGUiOiJkZXRlY3RvciJ9fSwic3BlYyI6eyJkZXNjcmlwdGlvbiI6IkdQVSBiYXNlZCBpbXBsZW1lbnRhdGlvbiBvZiBNYXNrIFJDTk4gb24gUHl0aG9uIDMsIEtlcmFzLCBhbmQgVGVuc29yRmxvdy4iLCJoYW5kbGVyIjoibWFpbjpoYW5kbGVyIiwicnVudGltZSI6InB5dGhvbjozLjYiLCJlbnYiOlt7Im5hbWUiOiJNQVNLX1JDTk5fRElSIiwidmFsdWUiOiIvb3B0L251Y2xpby9NYXNrX1JDTk4ifV0sInJlc291cmNlcyI6eyJsaW1pdHMiOnsibnZpZGlhLmNvbS9ncHUiOiIxIn0sInJlcXVlc3RzIjp7ImNwdSI6IjI1bSIsIm1lbW9yeSI6IjFNaSJ9fSwidGFyZ2V0Q1BVIjo3NSwidHJpZ2dlcnMiOnsibXlIdHRwVHJpZ2dlciI6eyJjbGFzcyI6IiIsImtpbmQiOiJodHRwIiwibmFtZSI6Im15SHR0cFRyaWdnZXIiLCJtYXhXb3JrZXJzIjoxLCJ3b3JrZXJBdmFpbGFiaWxpdHlUaW1lb3V0TWlsbGlzZWNvbmRzIjoxMDAwMCwiYXR0cmlidXRlcyI6eyJtYXhSZXF1ZXN0Qm9keVNpemUiOjMzNTU0NDMyfX19LCJidWlsZCI6eyJpbWFnZSI6ImN2YXQvdGYubWF0dGVycG9ydC5tYXNrX3Jjbm5fZ3B1IiwiYmFzZUltYWdlIjoidGVuc29yZmxvdy90ZW5zb3JmbG93OjEuMTUuNS1ncHUtcHkzIiwiZGlyZWN0aXZlcyI6eyJwb3N0Q29weSI6W3sia2luZCI6IldPUktESVIiLCJ2YWx1ZSI6Ii9vcHQvbnVjbGlvIn0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJhcHQgdXBkYXRlIFx1MDAyNlx1MDAyNiBhcHQgaW5zdGFsbCAtLW5vLWluc3RhbGwtcmVjb21tZW5kcyAteSBnaXQgY3VybCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiZ2l0IGNsb25lIC0tZGVwdGggMSBodHRwczovL2dpdGh1Yi5jb20vbWF0dGVycG9ydC9NYXNrX1JDTk4uZ2l0In0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJjdXJsIC1MIGh0dHBzOi8vZ2l0aHViLmNvbS9tYXR0ZXJwb3J0L01hc2tfUkNOTi9yZWxlYXNlcy9kb3dubG9hZC92Mi4wL21hc2tfcmNubl9jb2NvLmg1IC1vIE1hc2tfUkNOTi9tYXNrX3Jjbm5fY29jby5oNSJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoicGlwMyBpbnN0YWxsIG51bXB5IGN5dGhvbiBweXlhbWwga2VyYXM9PTIuMS4wIHNjaWtpdC1pbWFnZSAnaW1hZ2Vpb1x1MDAzYz0yLjkuMCcgUGlsbG93In1dfSwiY29kZUVudHJ5VHlwZSI6ImltYWdlIn0sInBsYXRmb3JtIjp7ImF0dHJpYnV0ZXMiOnsibW91bnRNb2RlIjoidm9sdW1lIiwicmVzdGFydFBvbGljeSI6eyJtYXhpbXVtUmV0cnlDb3VudCI6MywibmFtZSI6ImFsd2F5cyJ9fX0sInJlYWRpbmVzc1RpbWVvdXRTZWNvbmRzIjoxMjAsInNlY3VyaXR5Q29udGV4dCI6e30sImV2ZW50VGltZW91dCI6IjMwcyJ9LCJzdGF0dXMiOnsic3RhdGUiOiJlcnJvciIsIm1lc3NhZ2UiOiJcbkVycm9yIC0gZXhpdCBzdGF0dXMgMTAwXG4gICAgL251Y2xpby9wa2cvY21kcnVubmVyL3NoZWxscnVubmVyLmdvOjk2XG5cbkNhbGwgc3RhY2s6XG5zdGRvdXQ6XG5TZW5kaW5nIGJ1aWxkIGNvbnRleHQgdG8gRG9ja2VyIGRhZW1vbiAgNTAuMDlNQlxyXHJcblN0ZXAgMS8xNyA6IEZST00gdGVuc29yZmxvdy90ZW5zb3JmbG93OjEuMTUuNS1ncHUtcHkzXG4xLjE1LjUtZ3B1LXB5MzogUHVsbGluZyBmcm9tIHRlbnNvcmZsb3cvdGVuc29yZmxvd1xuMTcxODU3YzQ5ZDBmOiBQdWxsaW5nIGZzIGxheWVyXG40MTk2NDA0NDdkMjY6IFB1bGxpbmcgZnMgbGF5ZXJcbjYxZTUyZjg2MjYxOTogUHVsbGluZyBmcyBsYXllclxuYzExOGRhZDdlMzdhOiBQdWxsaW5nIGZzIGxheWVyXG5mMzAxNWVmNjRiODQ6IFB1bGxpbmcgZnMgbGF5ZXJcbjRjOTdlZjIyNWY3MTogUHVsbGluZyBmcyBsYXllclxuYzRmNzEwNmM1NjJjOiBQdWxsaW5nIGZzIGxheWVyXG40MGMyNGFhMTcyYWM6IFB1bGxpbmcgZnMgbGF5ZXJcbjdmZWVhNDM2OWZjZDogUHVsbGluZyBmcyBsYXllclxuOTJiZDkyNDk2YTc0OiBQdWxsaW5nIGZzIGxheWVyXG41Yjg1NGNmOGYzYzI6IFB1bGxpbmcgZnMgbGF5ZXJcbmI4ZDk5NTgxN2Y4MjogUHVsbGluZyBmcyBsYXllclxuZjlkNDA3NzU0MTE2OiBQdWxsaW5nIGZzIGxheWVyXG5iMzdlNjFjNGZlYTI6IFB1bGxpbmcgZnMgbGF5ZXJcbmU4NjI1OTE2NGJkNjogUHVsbGluZyBmcyBsYXllclxuNGM5N2VmMjI1ZjcxOiBXYWl0aW5nXG40MGMyNGFhMTcyYWM6IFdhaXRpbmdcbmYzMDE1ZWY2NGI4NDogV2FpdGluZ1xuYzExOGRhZDdlMzdhOiBXYWl0aW5nXG43ZmVlYTQzNjlmY2Q6IFdhaXRpbmdcbmM0ZjcxMDZjNTYyYzogV2FpdGluZ1xuOTJiZDkyNDk2YTc0OiBXYWl0aW5nXG5iMzdlNjFjNGZlYTI6IFdhaXRpbmdcbmU4NjI1OTE2NGJkNjogV2FpdGluZ1xuYjhkOTk1ODE3ZjgyOiBXYWl0aW5nXG5mOWQ0MDc3NTQxMTY6IFdhaXRpbmdcbjYxZTUyZjg2MjYxOTogVmVyaWZ5aW5nIENoZWNrc3VtXG42MWU1MmY4NjI2MTk6IERvd25sb2FkIGNvbXBsZXRlXG40MTk2NDA0NDdkMjY6IERvd25sb2FkIGNvbXBsZXRlXG5jMTE4ZGFkN2UzN2E6IFZlcmlmeWluZyBDaGVja3N1bVxuYzExOGRhZDdlMzdhOiBEb3dubG9hZCBjb21wbGV0ZVxuMTcxODU3YzQ5ZDBmOiBWZXJpZnlpbmcgQ2hlY2tzdW1cbjE3MTg1N2M0OWQwZjogRG93bmxvYWQgY29tcGxldGVcbjE3MTg1N2M0OWQwZjogUHVsbCBjb21wbGV0ZVxuNDE5NjQwNDQ3ZDI2OiBQdWxsIGNvbXBsZXRlXG42MWU1MmY4NjI2MTk6IFB1bGwgY29tcGxldGVcbmMxMThkYWQ3ZTM3YTogUHVsbCBjb21wbGV0ZVxuNGM5N2VmMjI1ZjcxOiBEb3dubG9hZCBjb21wbGV0ZVxuZjMwMTVlZjY0Yjg0OiBWZXJpZnlpbmcgQ2hlY2tzdW1cbmYzMDE1ZWY2NGI4NDogRG93bmxvYWQgY29tcGxldGVcbmYzMDE1ZWY2NGI4NDogUHVsbCBjb21wbGV0ZVxuNGM5N2VmMjI1ZjcxOiBQdWxsIGNvbXBsZXRlXG43ZmVlYTQzNjlmY2Q6IERvd25sb2FkIGNvbXBsZXRlXG40MGMyNGFhMTcyYWM6IFZlcmlmeWluZyBDaGVja3N1bVxuNDBjMjRhYTE3MmFjOiBEb3dubG9hZCBjb21wbGV0ZVxuNWI4NTRjZjhmM2MyOiBWZXJpZnlpbmcgQ2hlY2tzdW1cbjViODU0Y2Y4ZjNjMjogRG93bmxvYWQgY29tcGxldGVcbmI4ZDk5NTgxN2Y4MjogVmVyaWZ5aW5nIENoZWNrc3VtXG5iOGQ5OTU4MTdmODI6IERvd25sb2FkIGNvbXBsZXRlXG45MmJkOTI0OTZhNzQ6IFZlcmlmeWluZyBDaGVja3N1bVxuOTJiZDkyNDk2YTc0OiBEb3dubG9hZCBjb21wbGV0ZVxuYjM3ZTYxYzRmZWEyOiBWZXJpZnlpbmcgQ2hlY2tzdW1cbmIzN2U2MWM0ZmVhMjogRG93bmxvYWQgY29tcGxldGVcbmU4NjI1OTE2NGJkNjogRG93bmxvYWQgY29tcGxldGVcbmM0ZjcxMDZjNTYyYzogVmVyaWZ5aW5nIENoZWNrc3VtXG5jNGY3MTA2YzU2MmM6IERvd25sb2FkIGNvbXBsZXRlXG5jNGY3MTA2YzU2MmM6IFB1bGwgY29tcGxldGVcbjQwYzI0YWExNzJhYzogUHVsbCBjb21wbGV0ZVxuN2ZlZWE0MzY5ZmNkOiBQdWxsIGNvbXBsZXRlXG45MmJkOTI0OTZhNzQ6IFB1bGwgY29tcGxldGVcbjViODU0Y2Y4ZjNjMjogUHVsbCBjb21wbGV0ZVxuYjhkOTk1ODE3ZjgyOiBQdWxsIGNvbXBsZXRlXG5mOWQ0MDc3NTQxMTY6IERvd25sb2FkIGNvbXBsZXRlXG5mOWQ0MDc3NTQxMTY6IFB1bGwgY29tcGxldGVcbmIzN2U2MWM0ZmVhMjogUHVsbCBjb21wbGV0ZVxuZTg2MjU5MTY0YmQ2OiBQdWxsIGNvbXBsZXRlXG5EaWdlc3Q6IHNoYTI1Njo3YWQ3NDJjYmIyYzc3YTQwZDA5OTZlYzA4MzQ1ZGZmNTRmZTI1ZjM5NDI4NDg2ZTVmNDFjNjlkYjA0YjZkMTdiXG5TdGF0dXM6IERvd25sb2FkZWQgbmV3ZXIgaW1hZ2UgZm9yIHRlbnNvcmZsb3cvdGVuc29yZmxvdzoxLjE1LjUtZ3B1LXB5M1xuIC0tLVx1MDAzZSA3M2JlMTEzNzM0OThcblN0ZXAgMi8xNyA6IEFSRyBOVUNMSU9fTEFCRUxcbiAtLS1cdTAwM2UgUnVubmluZyBpbiA0NWUyMzFjMGVlZDhcblJlbW92aW5nIGludGVybWVkaWF0ZSBjb250YWluZXIgNDVlMjMxYzBlZWQ4XG4gLS0tXHUwMDNlIGJlNTA5ZGY4YmYyZFxuU3RlcCAzLzE3IDogQVJHIE5VQ0xJT19BUkNIXG4gLS0tXHUwMDNlIFJ1bm5pbmcgaW4gOTNhZTIwNjZhZWI2XG5SZW1vdmluZyBpbnRlcm1lZGlhdGUgY29udGFpbmVyIDkzYWUyMDY2YWViNlxuIC0tLVx1MDAzZSA2MDFhZjVjMjU1NzhcblN0ZXAgNC8xNyA6IEFSRyBOVUNMSU9fQlVJTERfTE9DQUxfSEFORExFUl9ESVJcbiAtLS1cdTAwM2UgUnVubmluZyBpbiAyODk3Y2UxZWFlOGRcblJlbW92aW5nIGludGVybWVkaWF0ZSBjb250YWluZXIgMjg5N2NlMWVhZThkXG4gLS0tXHUwMDNlIDM4NWExOGZlZjRkY1xuU3RlcCA1LzE3IDogQ09QWSBhcnRpZmFjdHMvcHJvY2Vzc29yIC91c3IvbG9jYWwvYmluL3Byb2Nlc3NvclxuIC0tLVx1MDAzZSAxMzU2NTRlNTlkM2FcblN0ZXAgNi8xNyA6IENPUFkgYXJ0aWZhY3RzL3B5IC9vcHQvbnVjbGlvL1xuIC0tLVx1MDAzZSAyZWFiODJjZTM1ZmJcblN0ZXAgNy8xNyA6IENPUFkgYXJ0aWZhY3RzL3B5LXdobCAvb3B0L251Y2xpby93aGxcbiAtLS1cdTAwM2UgNDA5NjllMDdlMjE5XG5TdGVwIDgvMTcgOiBDT1BZIGFydGlmYWN0cy91aHR0cGMgL3Vzci9sb2NhbC9iaW4vdWh0dHBjXG4gLS0tXHUwMDNlIDkzNjc1ZGJlYWRhZVxuU3RlcCA5LzE3IDogQ09QWSBoYW5kbGVyIC9vcHQvbnVjbGlvXG4gLS0tXHUwMDNlIDgwYTYzYjgyZGYzNFxuU3RlcCAxMC8xNyA6IEhFQUxUSENIRUNLIC0taW50ZXJ2YWw9MXMgLS10aW1lb3V0PTNzIENNRCAvdXNyL2xvY2FsL2Jpbi91aHR0cGMgLS11cmwgaHR0cDovLzEyNy4wLjAuMTo4MDgyL3JlYWR5IHx8IGV4aXQgMVxuIC0tLVx1MDAzZSBSdW5uaW5nIGluIDRhZjRkMzc1MjBkMlxuUmVtb3ZpbmcgaW50ZXJtZWRpYXRlIGNvbnRhaW5lciA0YWY0ZDM3NTIwZDJcbiAtLS1cdTAwM2UgNDEyNTA0NjZhMDZlXG5TdGVwIDExLzE3IDogUlVOIHBpcCBpbnN0YWxsIG51Y2xpby1zZGsgbXNncGFjayAtLW5vLWluZGV4IC0tZmluZC1saW5rcyAvb3B0L251Y2xpby93aGxcbiAtLS1cdTAwM2UgUnVubmluZyBpbiA1YTg4ZTcwZGFkNGFcbkxvb2tpbmcgaW4gbGlua3M6IC9vcHQvbnVjbGlvL3dobFxuUHJvY2Vzc2luZyAvb3B0L251Y2xpby93aGwvbXNncGFjay0wLjYuMS50YXIuZ3pcblByb2Nlc3NpbmcgL29wdC9udWNsaW8vd2hsL251Y2xpb19zZGstMC40LjAtcHkyLnB5My1ub25lLWFueS53aGxcbkJ1aWxkaW5nIHdoZWVscyBmb3IgY29sbGVjdGVkIHBhY2thZ2VzOiBtc2dwYWNrXG4gIEJ1aWxkaW5nIHdoZWVsIGZvciBtc2dwYWNrIChzZXR1cC5weSk6IHN0YXJ0ZWRcbiAgQnVpbGRpbmcgd2hlZWwgZm9yIG1zZ3BhY2sgKHNldHVwLnB5KTogZmluaXNoZWQgd2l0aCBzdGF0dXMgJ2RvbmUnXG4gIENyZWF0ZWQgd2hlZWwgZm9yIG1zZ3BhY2s6IGZpbGVuYW1lPW1zZ3BhY2stMC42LjEtY3AzNi1jcDM2bS1saW51eF94ODZfNjQud2hsIHNpemU9MjI5ODkxIHNoYTI1Nj02NjIxYWE0NWI0ZWJlNWI3OGRiZGM2ZWJhOGNmYTU3ZTQ2MWIyMDZhOTcwMzE2YmZjMjIxMGU3YzkyODU1MDY5XG4gIFN0b3JlZCBpbiBkaXJlY3Rvcnk6IC9yb290Ly5jYWNoZS9waXAvd2hlZWxzLzE5L2RmL2MzLzA4NDI0ZmU1Mjg1NjY3YWZmOGE1OGE2OWM5OTVjNzIzZmZhODFjNDIwODEzOGQ2ZDBmXG5TdWNjZXNzZnVsbHkgYnVpbHQgbXNncGFja1xuSW5zdGFsbGluZyBjb2xsZWN0ZWQgcGFja2FnZXM6IG51Y2xpby1zZGssIG1zZ3BhY2tcblN1Y2Nlc3NmdWxseSBpbnN0YWxsZWQgbXNncGFjay0wLjYuMSBudWNsaW8tc2RrLTAuNC4wXG5SZW1vdmluZyBpbnRlcm1lZGlhdGUgY29udGFpbmVyIDVhODhlNzBkYWQ0YVxuIC0tLVx1MDAzZSBlYTQ0NjQ4MDIzZDBcblN0ZXAgMTIvMTcgOiBXT1JLRElSIC9vcHQvbnVjbGlvXG4gLS0tXHUwMDNlIFJ1bm5pbmcgaW4gNDY3OTQ4ZTM1MjI5XG5SZW1vdmluZyBpbnRlcm1lZGlhdGUgY29udGFpbmVyIDQ2Nzk0OGUzNTIyOVxuIC0tLVx1MDAzZSA1OTNkN2UyZTBjNjFcblN0ZXAgMTMvMTcgOiBSVU4gYXB0IHVwZGF0ZSBcdTAwMjZcdTAwMjYgYXB0IGluc3RhbGwgLS1uby1pbnN0YWxsLXJlY29tbWVuZHMgLXkgZ2l0IGN1cmxcbiAtLS1cdTAwM2UgUnVubmluZyBpbiAyYmZhODQwY2IzMmNcblx1MDAxYls5MW1cbldBUk5JTkc6IGFwdCBkb2VzIG5vdCBoYXZlIGEgc3RhYmxlIENMSSBpbnRlcmZhY2UuIFVzZSB3aXRoIGNhdXRpb24gaW4gc2NyaXB0cy5cblxuXHUwMDFiWzBtR2V0OjEgaHR0cDovL3NlY3VyaXR5LnVidW50dS5jb20vdWJ1bnR1IGJpb25pYy1zZWN1cml0eSBJblJlbGVhc2UgWzg4Ljcga0JdXG5IaXQ6MiBodHRwOi8vYXJjaGl2ZS51YnVudHUuY29tL3VidW50dSBiaW9uaWMgSW5SZWxlYXNlXG5HZXQ6MyBodHRwOi8vYXJjaGl2ZS51YnVudHUuY29tL3VidW50dSBiaW9uaWMtdXBkYXRlcyBJblJlbGVhc2UgWzg4Ljcga0JdXG5HZXQ6NCBodHRwczovL2RldmVsb3Blci5kb3dubG9hZC5udmlkaWEuY29tL2NvbXB1dGUvY3VkYS9yZXBvcy91YnVudHUxODA0L3g4Nl82NCAgSW5SZWxlYXNlIFsxNTgxIEJdXG5HZXQ6NSBodHRwOi8vc2VjdXJpdHkudWJ1bnR1LmNvbS91YnVudHUgYmlvbmljLXNlY3VyaXR5L211bHRpdmVyc2UgYW1kNjQgUGFja2FnZXMgWzIyLjkga0JdXG5FcnI6NCBodHRwczovL2RldmVsb3Blci5kb3dubG9hZC5udmlkaWEuY29tL2NvbXB1dGUvY3VkYS9yZXBvcy91YnVudHUxODA0L3g4Nl82NCAgSW5SZWxlYXNlXG4gIFRoZSBmb2xsb3dpbmcgc2lnbmF0dXJlcyBjb3VsZG4ndCBiZSB2ZXJpZmllZCBiZWNhdXNlIHRoZSBwdWJsaWMga2V5IGlzIG5vdCBhdmFpbGFibGU6IE5PX1BVQktFWSBBNEI0Njk5NjNCRjg2M0NDXG5HZXQ6NiBodHRwOi8vc2VjdXJpdHkudWJ1bnR1LmNvbS91YnVudHUgYmlvbmljLXNlY3VyaXR5L3VuaXZlcnNlIGFtZDY0IFBhY2thZ2VzIFsxNTY4IGtCXVxuSWduOjcgaHR0cHM6Ly9kZXZlbG9wZXIuZG93bmxvYWQubnZpZGlhLmNvbS9jb21wdXRlL21hY2hpbmUtbGVhcm5pbmcvcmVwb3MvdWJ1bnR1MTgwNC94ODZfNjQgIEluUmVsZWFzZVxuR2V0OjggaHR0cHM6Ly9kZXZlbG9wZXIuZG93bmxvYWQubnZpZGlhLmNvbS9jb21wdXRlL21hY2hpbmUtbGVhcm5pbmcvcmVwb3MvdWJ1bnR1MTgwNC94ODZfNjQgIFJlbGVhc2UgWzU2NCBCXVxuR2V0OjkgaHR0cDovL2FyY2hpdmUudWJ1bnR1LmNvbS91YnVudHUgYmlvbmljLWJhY2twb3J0cyBJblJlbGVhc2UgWzgzLjMga0JdXG5HZXQ6MTAgaHR0cHM6Ly9kZXZlbG9wZXIuZG93bmxvYWQubnZpZGlhLmNvbS9jb21wdXRlL21hY2hpbmUtbGVhcm5pbmcvcmVwb3MvdWJ1bnR1MTgwNC94ODZfNjQgIFJlbGVhc2UuZ3BnIFs4MzMgQl1cbkdldDoxMSBodHRwOi8vc2VjdXJpdHkudWJ1bnR1LmNvbS91YnVudHUgYmlvbmljLXNlY3VyaXR5L21haW4gYW1kNjQgUGFja2FnZXMgWzMwOTQga0JdXG5HZXQ6MTIgaHR0cHM6Ly9kZXZlbG9wZXIuZG93bmxvYWQubnZpZGlhLmNvbS9jb21wdXRlL21hY2hpbmUtbGVhcm5pbmcvcmVwb3MvdWJ1bnR1MTgwNC94ODZfNjQgIFBhY2thZ2VzIFs3My44IGtCXVxuR2V0OjEzIGh0dHA6Ly9zZWN1cml0eS51YnVudHUuY29tL3VidW50dSBiaW9uaWMtc2VjdXJpdHkvcmVzdHJpY3RlZCBhbWQ2NCBQYWNrYWdlcyBbMTMwNyBrQl1cbkdldDoxNCBodHRwOi8vYXJjaGl2ZS51YnVudHUuY29tL3VidW50dSBiaW9uaWMtdXBkYXRlcy9tYWluIGFtZDY0IFBhY2thZ2VzIFszNTE5IGtCXVxuR2V0OjE1IGh0dHA6Ly9hcmNoaXZlLnVidW50dS5jb20vdWJ1bnR1IGJpb25pYy11cGRhdGVzL3VuaXZlcnNlIGFtZDY0IFBhY2thZ2VzIFsyMzQyIGtCXVxuR2V0OjE2IGh0dHA6Ly9hcmNoaXZlLnVidW50dS5jb20vdWJ1bnR1IGJpb25pYy11cGRhdGVzL3Jlc3RyaWN0ZWQgYW1kNjQgUGFja2FnZXMgWzEzNDgga0JdXG5HZXQ6MTcgaHR0cDovL2FyY2hpdmUudWJ1bnR1LmNvbS91YnVudHUgYmlvbmljLXVwZGF0ZXMvbXVsdGl2ZXJzZSBhbWQ2NCBQYWNrYWdlcyBbMzAuMCBrQl1cbkdldDoxOCBodHRwOi8vYXJjaGl2ZS51YnVudHUuY29tL3VidW50dSBiaW9uaWMtYmFja3BvcnRzL21haW4gYW1kNjQgUGFja2FnZXMgWzY0LjAga0JdXG5HZXQ6MTkgaHR0cDovL2FyY2hpdmUudWJ1bnR1LmNvbS91YnVudHUgYmlvbmljLWJhY2twb3J0cy91bml2ZXJzZSBhbWQ2NCBQYWNrYWdlcyBbMjAuNSBrQl1cblJlYWRpbmcgcGFja2FnZSBsaXN0cy4uLlxuXHUwMDFiWzkxbVc6IEdQRyBlcnJvcjogaHR0cHM6Ly9kZXZlbG9wZXIuZG93bmxvYWQubnZpZGlhLmNvbS9jb21wdXRlL2N1ZGEvcmVwb3MvdWJ1bnR1MTgwNC94ODZfNjQgIEluUmVsZWFzZTogVGhlIGZvbGxvd2luZyBzaWduYXR1cmVzIGNvdWxkbid0IGJlIHZlcmlmaWVkIGJlY2F1c2UgdGhlIHB1YmxpYyBrZXkgaXMgbm90IGF2YWlsYWJsZTogTk9fUFVCS0VZIEE0QjQ2OTk2M0JGODYzQ0NcbkU6IFRoZSByZXBvc2l0b3J5ICdodHRwczovL2RldmVsb3Blci5kb3dubG9hZC5udmlkaWEuY29tL2NvbXB1dGUvY3VkYS9yZXBvcy91YnVudHUxODA0L3g4Nl82NCAgSW5SZWxlYXNlJyBpcyBubyBsb25nZXIgc2lnbmVkLlxuXHUwMDFiWzBtUmVtb3ZpbmcgaW50ZXJtZWRpYXRlIGNvbnRhaW5lciAyYmZhODQwY2IzMmNcblxuc3RkZXJyOlxuVGhlIGNvbW1hbmQgJy9iaW4vYmFzaCAtYyBhcHQgdXBkYXRlIFx1MDAyNlx1MDAyNiBhcHQgaW5zdGFsbCAtLW5vLWluc3RhbGwtcmVjb21tZW5kcyAteSBnaXQgY3VybCcgcmV0dXJuZWQgYSBub24temVybyBjb2RlOiAxMDBcblxuICAgIC9udWNsaW8vcGtnL2NtZHJ1bm5lci9zaGVsbHJ1bm5lci5nbzo5NlxuRmFpbGVkIHRvIGJ1aWxkXG4gICAgL251Y2xpby9wa2cvZG9ja2VyY2xpZW50L3NoZWxsLmdvOjExN1xuRmFpbGVkIHRvIGJ1aWxkIGRvY2tlciBpbWFnZVxuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzo1NFxuRmFpbGVkIHRvIGJ1aWxkIHByb2Nlc3NvciBpbWFnZVxuICAgIC9udWNsaW8vcGtnL3Byb2Nlc3Nvci9idWlsZC9idWlsZGVyLmdvOjI2M1xuIn19

stderr:

    /nuclio/pkg/cmdrunner/shellrunner.go:96
Failed to execute command: /bin/sh -c "/bin/cat /etc/nuclio/store/functions/nuclio/*"
    ...//nuclio/pkg/platform/local/client/store.go:413
Failed to run cat command
    ...//nuclio/pkg/platform/local/client/store.go:340
Failed to get functions
    ...//nuclio/pkg/platform/local/client/store.go:258
Failed to read functions from local store
    ...//nuclio/pkg/platform/local/client/store.go:222

The HRnet give that error in the terminal, it asks me to set the runtime. How to do it ?

22.12.04 09:45:41.159                     nuctl (I) Deploying function {"name": ""}
22.12.04 09:45:41.159                     nuctl (I) Building {"builderKind": "docker", "versionInfo": "Label: 1.8.14, Git commit: cbb0774230996a3eb4621c1a2079e2317578005b, OS: linux, Arch: amd64, Go version: go1.17.8", "name": ""}
22.12.04 09:45:41.159                     nuctl (W) Failed to create a function; setting the function status {"err": "Failed create runtime", "errVerbose": "\nError - Build path is directory - runtime must be specified\n    /nuclio/pkg/processor/build/builder.go:879\n\nCall stack:\nBuild path is directory - runtime must be specified\n    /nuclio/pkg/processor/build/builder.go:879\nFailed create runtime\n    /nuclio/pkg/processor/build/builder.go:228\nFailed create runtime"}

Error - Build path is directory - runtime must be specified
    /nuclio/pkg/processor/build/builder.go:879

Call stack:
Build path is directory - runtime must be specified
    /nuclio/pkg/processor/build/builder.go:879
Failed create runtime
    /nuclio/pkg/processor/build/builder.go:228
Failed to deploy function
    ...//nuclio/pkg/platform/abstract/platform.go:197

I also install nvidia-docker2 on my server. It it successfully passes verification from nvidia, it works.

I tried to run the models interactors on GPU according to your recommendations and documentation link. The base commands pattern was taken from DEXTR:

 nuctl deploy --project-name cvat \
  --path <interartor_path>\
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local \
  --triggers '{"myHttpTrigger": {"maxWorkers": 1}}' \
  --resource-limit nvidia.com/gpu=1

where for dextr == serverless/openvino/dextr/nuclio # it works but give error it the terminal for f-BRS == serverless/pytorch/saic-vul/fbrs/nuclio # it works but give error it the terminal for Inside-Outside-Guidance == serverless/pytorch/shiyinzhang/iog/nuclio # it works but give error it the terminal for HRnet == serverless/pytorch/saic-vul/hrnet/nuclio # it doesn't work also.

I am interested in the question how to set the runtime that HRNet asks? I think that runtime is CPU or GPU. As I understood, if we have lines:

  --triggers '{"myHttpTrigger": {"maxWorkers": 1}}' \
  --resource-limit nvidia.com/gpu=1

we choose GPU runtime, in case of their absence, the CPU instead.

There is an example in docs with running RCNN detector on GPU runtime link.

nuctl deploy --project-name cvat \
  --path serverless/tensorflow/matterport/mask_rcnn/nuclio \
  --platform local --base-image tensorflow/tensorflow:1.15.5-gpu-py3 \
  --desc "GPU based implementation of Mask RCNN on Python 3, Keras, and TensorFlow." \
  --image cvat/tf.matterport.mask_rcnn_gpu \
  --triggers '{"myHttpTrigger": {"maxWorkers": 1}}' \
  --resource-limit nvidia.com/gpu=1

In this example we also set image param "cvat/tf.matterport.mask_rcnn_gpu". I also go to path "serverless/tensorflow/matterport/mask_rcnn/nuclio" and I see that there are two config fails:

-serverless/tensorflow/faster_rcnn_inception_v2_coco |----function-gpu.yaml # contains baseImage: tensorflow/tensorflow:2.1.1-gpu |----function.yaml # baseImage: tensorflow/tensorflow:2.1.1

Does the presence of these files determine whether the model can be run on the CPU or GPU? For example, in serverless/openvino/dextr/nuclio we have only "function.yaml" in serverless/pytorch/saic-vul/fbrs/nuclio we have only "function.yaml" in serverless/pytorch/shiyinzhang/iog/nuclio we have only "function.yaml" but in serverless/pytorch/saic-vul/hrnet/nuclio we have only "function-gpu.yaml"

How does it all work?

medphisiker commented 1 year ago

I also looked at table "Deep learning serverless functions for automatic labeling" on github link. According to this table, it turns out that dextr, f-BRS, Inside-Outside-Guidance runs only on CPU, and HRNet runs only on GPU ?

But the documentation page recommended to run interactor models on GPU, if possible for f-BRS, HRNet, Inside-Outside-Guidance link.

zhiltsov-max commented 1 year ago

@azhavoro , could you please check?

medphisiker commented 1 year ago

This is strange. I stopped CVAT with a command from docs (link) docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml down

And then I manually stopped these docker containers:

cvat/openvine.dextr:latest
gcr.io/iguazio/alpine:3.15

Then I set CVAT_HOST environment variable by command: export CVAT_HOST=my_ip_address

Then I rerun CVAT by command: docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d

Then I create cvat project inside nuclio dashboard via command: nuctl create project cvat

And it was created: 22.12.13 02:33:22.307 nuctl (I) Project created {"Name": "cvat", "Namespace": "nuclio"}

And then I run dextr by command:

nuctl deploy --project-name cvat \
  --path serverless/openvino/dextr/nuclio \
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

There are all docker's containers in my system:

I have the same error in my terminal as in my message above (link).

But for now CVAT writes message:

Could not get models from the server
Error: Request failed with status code 500. "500 Server Error: Internal Server Error for url: http://nuclio:8070/api/functions".

And DL models are not available...

medphisiker commented 1 year ago

Hello,

I stop all containers, then remove all dockers images, volumes and other data. Then I reinstall cvat and reinstall and rerun DEXTR and f-BRS. After reinstall these DL model works perfectly on CPU. The error in my terminal indicated in my message above (link) disappears.

But I don't know the right command to run HRnet on the GPU. So I don't know if it works for me.

medphisiker commented 1 year ago

Hello,

I also read docs the "Serverless tutorial" link. There was another way to launch DL models by commands: serverless/deploy_cpu.sh <path_to_model> and serverless/deploy_gpu.sh <path_to_model>

where for dextr == serverless/openvino/dextr/nuclio for f-BRS == serverless/pytorch/saic-vul/fbrs/nuclio for Inside-Outside-Guidance == serverless/pytorch/shiyinzhang/iog/nuclio for HRnet == serverless/pytorch/saic-vul/hrnet/nuclio

I run dextr, f-BRS and Inside-Outside-Guidance by serverless/deploy_cpu.sh and they works well ) Then I try run dextr, f-BRS and Inside-Outside-Guidance by serverless/deploy_gpu.shand I got this error in terminal:

22.12.13 15:39:32.319                     nuctl (I) Project created {"Name": "cvat", "Namespace": "nuclio"}
Deploying serverless/pytorch/saic-vul/fbrs/nuclio function...

Error - stat serverless/pytorch/saic-vul/fbrs/nuclio/**/function-gpu.yaml: no such file or directory
    ...//nuclio/pkg/nuctl/command/common/helpers.go:36

Call stack:
File path `serverless/pytorch/saic-vul/fbrs/nuclio/**/function-gpu.yaml` does not exists
    ...//nuclio/pkg/nuctl/command/common/helpers.go:36
Failed opening function config file
    /nuclio/pkg/nuctl/command/deploy.go:122

When launched in this way, it turns out that they really should have a "function-gpu.yaml" file so that they can be launched on the GPU. But dextr, f-BRS and Inside-Outside-Guidance don't have them. They have only "function.yaml" for CPU runtime. But docs recommends to run f-BRS on GPU like HRnet link. It looks like we just need to fix this docs page for f-BRS for CPU only. HRnet and f-BRS used in a similar way and have an almost identical description. Perhaps the ability to run on the GPU in the description of f-BRS remained from HRnet.

And vice versa, HRnet has only the "function-gpu.yaml" file and apparently can only work on the GPU.

medphisiker commented 1 year ago

I also try to run HRnet interactor by command for launching it on GPU: serverless/deploy_gpu.sh serverless/pytorch/saic-vul/hrnet/nuclio

But I have these errors when it is building docker image:

2022-12-13 15:18:44 (4.17 MB/s) - 'coco_lvis_h18_itermask.pth' saved [40700707/40700707]

Removing intermediate container e0884f5c5089
 ---> 50c4d207c0df
Step 15/28 : RUN pip3 install setuptools
 ---> Running in b04488af87b1
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (45.2.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container b04488af87b1
 ---> 64c146a803d1
Step 16/28 : RUN pip3 install -r requirements.txt
 ---> Running in 1630c2f1f24d
Collecting scipy
  Downloading scipy-1.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 33.8/33.8 MB 3.9 MB/s eta 0:00:00
Collecting numpy
  Downloading numpy-1.23.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.1/17.1 MB 4.1 MB/s eta 0:00:00
Collecting Cython
  Downloading Cython-0.29.32-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 4.2 MB/s eta 0:00:00
Collecting scikit-image
  Downloading scikit_image-0.19.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 4.7 MB/s eta 0:00:00
Collecting opencv-python-headless
  Downloading opencv_python_headless-4.6.0.66-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (48.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 MB 4.8 MB/s eta 0:00:00
Collecting Pillow
  Downloading Pillow-9.3.0-cp38-cp38-manylinux_2_28_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 5.2 MB/s eta 0:00:00
Collecting matplotlib
  Downloading matplotlib-3.6.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/9.4 MB 5.1 MB/s eta 0:00:00
Collecting imgaug>=0.4
  Downloading imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 948.0/948.0 kB 5.1 MB/s eta 0:00:00
Collecting albumentations>0.5
  Downloading albumentations-1.3.0-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.5/123.5 kB 5.4 MB/s eta 0:00:00
Collecting graphviz
  Downloading graphviz-0.20.1-py3-none-any.whl (47 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.0/47.0 kB 3.3 MB/s eta 0:00:00
Collecting tqdm
  Downloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 kB 4.3 MB/s eta 0:00:00
Collecting pyyaml
  Downloading PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 701.2/701.2 kB 5.9 MB/s eta 0:00:00
Collecting easydict
  Downloading easydict-1.10.tar.gz (6.4 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting torch>=1.4.0
  Downloading torch-1.13.0-cp38-cp38-manylinux1_x86_64.whl (890.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 890.2/890.2 MB 3.0 MB/s eta 0:00:00
Collecting torchvision>=0.5.0
  Downloading torchvision-0.14.0-cp38-cp38-manylinux1_x86_64.whl (24.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.3/24.3 MB 5.8 MB/s eta 0:00:00
Collecting tensorboard
  Downloading tensorboard-2.11.0-py3-none-any.whl (6.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 6.7 MB/s eta 0:00:00
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 829.2/829.2 kB 8.2 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting cffi
  Downloading cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 442.7/442.7 kB 8.5 MB/s eta 0:00:00
Collecting ninja
  Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 146.0/146.0 kB 7.2 MB/s eta 0:00:00
Collecting tifffile>=2019.7.26
  Downloading tifffile-2022.10.10-py3-none-any.whl (210 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 210.3/210.3 kB 7.5 MB/s eta 0:00:00
Collecting imageio>=2.4.1
  Downloading imageio-2.22.4-py3-none-any.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 7.3 MB/s eta 0:00:00
Collecting PyWavelets>=1.1.1
  Downloading PyWavelets-1.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 6.3 MB/s eta 0:00:00
Collecting networkx>=2.2
  Downloading networkx-2.8.8-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 6.1 MB/s eta 0:00:00
Collecting packaging>=20.0
  Downloading packaging-22.0-py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 4.2 MB/s eta 0:00:00
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 5.9 MB/s eta 0:00:00
Collecting python-dateutil>=2.7
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 22.9 MB/s eta 0:00:00
Collecting contourpy>=1.0.1
  Downloading contourpy-1.0.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (295 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 296.0/296.0 kB 6.8 MB/s eta 0:00:00
Collecting fonttools>=4.22.0
  Downloading fonttools-4.38.0-py3-none-any.whl (965 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 965.4/965.4 kB 5.8 MB/s eta 0:00:00
Collecting pyparsing>=2.2.1
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 6.1 MB/s eta 0:00:00
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting Shapely
  Downloading shapely-2.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 6.4 MB/s eta 0:00:00
Collecting opencv-python
  Downloading opencv_python-4.6.0.66-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 MB 4.6 MB/s eta 0:00:00
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from imgaug>=0.4->-r requirements.txt (line 8)) (1.14.0)
Collecting qudida>=0.0.4
  Downloading qudida-0.0.4-py3-none-any.whl (3.5 kB)
Collecting nvidia-cudnn-cu11==8.5.0.96
  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 3.5 MB/s eta 0:00:00
Collecting typing-extensions
  Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting nvidia-cuda-nvrtc-cu11==11.7.99
  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 4.8 MB/s eta 0:00:00
Collecting nvidia-cuda-runtime-cu11==11.7.99
  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 57.2 MB/s eta 0:00:00
Collecting nvidia-cublas-cu11==11.10.3.66
  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 3.2 MB/s eta 0:00:00
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->-r requirements.txt (line 14)) (0.34.2)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->-r requirements.txt (line 14)) (45.2.0)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from torchvision>=0.5.0->-r requirements.txt (line 15)) (2.22.0)
Collecting markdown>=2.6.8
  Downloading Markdown-3.4.1-py3-none-any.whl (93 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.3/93.3 kB 4.3 MB/s eta 0:00:00
Collecting google-auth<3,>=1.6.3
  Downloading google_auth-2.15.0-py2.py3-none-any.whl (177 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.0/177.0 kB 4.4 MB/s eta 0:00:00
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 781.3/781.3 kB 4.1 MB/s eta 0:00:00
Collecting grpcio>=1.24.3
  Downloading grpcio-1.51.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 4.5 MB/s eta 0:00:00
Collecting protobuf<4,>=3.9.2
  Downloading protobuf-3.20.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 4.9 MB/s eta 0:00:00
Collecting absl-py>=0.4
  Downloading absl_py-1.3.0-py3-none-any.whl (124 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.6/124.6 kB 4.8 MB/s eta 0:00:00
Collecting werkzeug>=1.0.1
  Downloading Werkzeug-2.2.2-py3-none-any.whl (232 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 232.7/232.7 kB 4.9 MB/s eta 0:00:00
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.9/4.9 MB 5.3 MB/s eta 0:00:00
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 3.3 MB/s eta 0:00:00
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 5.9 MB/s eta 0:00:00
Collecting cachetools<6.0,>=2.0.0
  Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
  Downloading importlib_metadata-5.1.0-py3-none-any.whl (21 kB)
Collecting scikit-learn>=0.19.1
  Downloading scikit_learn-1.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 4.0 MB/s eta 0:00:00
Collecting MarkupSafe>=2.1.1
  Downloading MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting zipp>=0.5
  Downloading zipp-3.11.0-py3-none-any.whl (6.6 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 kB 4.5 MB/s eta 0:00:00
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 5.1 MB/s eta 0:00:00
Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting joblib>=1.1.1
  Downloading joblib-1.2.0-py3-none-any.whl (297 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 298.0/298.0 kB 5.6 MB/s eta 0:00:00
Building wheels for collected packages: easydict, future
  Building wheel for easydict (setup.py): started
  Building wheel for easydict (setup.py): finished with status 'done'
  Created wheel for easydict: filename=easydict-1.10-py3-none-any.whl size=6496 sha256=b76f70c5b91bd8202a262f1148fbff614c47ac4a50373eaa2f1838ba58630858
  Stored in directory: /root/.cache/pip/wheels/7b/d2/a1/c4e0795ef16ab3276fb4ef17d3df108a15654c70157cd69ed3
  Building wheel for future (setup.py): started
  Building wheel for future (setup.py): finished with status 'done'
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=cd3767555447ec4e046e6c9c0c185aa7be5d9d2be56c37a735d01bc39905987f
  Stored in directory: /root/.cache/pip/wheels/01/49/0c/4e0a697824c7bd6516afb22e1af9d51427ccd36c74b05a297e
Successfully built easydict future
Installing collected packages: tensorboard-plugin-wit, pyasn1, ninja, easydict, zipp, typing-extensions, tqdm, threadpoolctl, tensorboard-data-server, rsa, pyyaml, python-dateutil, pyparsing, pycparser, pyasn1-modules, protobuf, Pillow, packaging, oauthlib, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cublas-cu11, numpy, networkx, MarkupSafe, kiwisolver, joblib, grpcio, graphviz, future, fonttools, Cython, cycler, cachetools, absl-py, werkzeug, tifffile, Shapely, scipy, requests-oauthlib, PyWavelets, opencv-python-headless, opencv-python, nvidia-cudnn-cu11, importlib-metadata, imageio, google-auth, contourpy, cffi, torch, scikit-learn, scikit-image, matplotlib, markdown, google-auth-oauthlib, torchvision, tensorboard, qudida, imgaug, albumentations
Successfully installed Cython-0.29.32 MarkupSafe-2.1.1 Pillow-9.3.0 PyWavelets-1.4.1 Shapely-2.0.0 absl-py-1.3.0 albumentations-1.3.0 cachetools-5.2.0 cffi-1.15.1 contourpy-1.0.6 cycler-0.11.0 easydict-1.10 fonttools-4.38.0 future-0.18.2 google-auth-2.15.0 google-auth-oauthlib-0.4.6 graphviz-0.20.1 grpcio-1.51.1 imageio-2.22.4 imgaug-0.4.0 importlib-metadata-5.1.0 joblib-1.2.0 kiwisolver-1.4.4 markdown-3.4.1 matplotlib-3.6.2 networkx-2.8.8 ninja-1.11.1 numpy-1.23.5 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 oauthlib-3.2.2 opencv-python-4.6.0.66 opencv-python-headless-4.6.0.66 packaging-22.0 protobuf-3.20.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pyparsing-3.0.9 python-dateutil-2.8.2 pyyaml-6.0 qudida-0.0.4 requests-oauthlib-1.3.1 rsa-4.9 scikit-image-0.19.3 scikit-learn-1.2.0 scipy-1.9.3 tensorboard-2.11.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 threadpoolctl-3.1.0 tifffile-2022.10.10 torch-1.13.0 torchvision-0.14.0 tqdm-4.64.1 typing-extensions-4.4.0 werkzeug-2.2.2 zipp-3.11.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 1630c2f1f24d
 ---> 4dd1b707c0fa
Step 17/28 : RUN apt update && apt install -y libgl1-mesa-glx
 ---> Running in 7f852e4233de

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Hit:1 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists...
Building dependency tree...
Reading state information...
All packages are up to date.

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libdrm-amdgpu1 libdrm-common libdrm-intel1 libdrm-nouveau2 libdrm-radeon1
  libdrm2 libedit2 libgl1 libgl1-mesa-dri libglapi-mesa libglvnd0 libglx-mesa0
  libglx0 libllvm12 libpciaccess0 libsensors-config libsensors5 libvulkan1
  libwayland-client0 libx11-xcb1 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0
  libxcb-present0 libxcb-randr0 libxcb-shm0 libxcb-sync1 libxcb-xfixes0
  libxfixes3 libxshmfence1 libxxf86vm1 mesa-vulkan-drivers
Suggested packages:
  pciutils lm-sensors
The following NEW packages will be installed:
  libdrm-amdgpu1 libdrm-common libdrm-intel1 libdrm-nouveau2 libdrm-radeon1
  libdrm2 libedit2 libgl1 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa
  libglvnd0 libglx-mesa0 libglx0 libllvm12 libpciaccess0 libsensors-config
  libsensors5 libvulkan1 libwayland-client0 libx11-xcb1 libxcb-dri2-0
  libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0 libxcb-shm0
  libxcb-sync1 libxcb-xfixes0 libxfixes3 libxshmfence1 libxxf86vm1
  mesa-vulkan-drivers
0 upgraded, 33 newly installed, 0 to remove and 0 not upgraded.
Need to get 36.4 MB of archives.
After this operation, 525 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdrm-common all 2.4.107-8ubuntu1~20.04.2 [5396 B]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdrm2 amd64 2.4.107-8ubuntu1~20.04.2 [34.1 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libedit2 amd64 3.1-20191231-1 [87.0 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdrm-amdgpu1 amd64 2.4.107-8ubuntu1~20.04.2 [18.6 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libpciaccess0 amd64 0.16-0ubuntu1 [17.9 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdrm-intel1 amd64 2.4.107-8ubuntu1~20.04.2 [60.3 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdrm-nouveau2 amd64 2.4.107-8ubuntu1~20.04.2 [16.6 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdrm-radeon1 amd64 2.4.107-8ubuntu1~20.04.2 [19.7 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libglapi-mesa amd64 21.2.6-0ubuntu0.1~20.04.2 [27.4 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libllvm12 amd64 1:12.0.0-3ubuntu1~20.04.5 [18.8 MB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsensors-config all 1:3.6.0-2ubuntu1.1 [6052 B]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsensors5 amd64 1:3.6.0-2ubuntu1.1 [27.2 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 libvulkan1 amd64 1.2.131.2-1 [93.3 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgl1-mesa-dri amd64 21.2.6-0ubuntu0.1~20.04.2 [11.0 MB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libglvnd0 amd64 1.3.2-1~ubuntu0.20.04.2 [48.1 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libx11-xcb1 amd64 2:1.6.9-2ubuntu1.2 [9372 B]
Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-dri2-0 amd64 1.14-2 [6920 B]
Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-dri3-0 amd64 1.14-2 [6552 B]
Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-glx0 amd64 1.14-2 [22.1 kB]
Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-present0 amd64 1.14-2 [5560 B]
Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-shm0 amd64 1.14-2 [5584 B]
Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-sync1 amd64 1.14-2 [8884 B]
Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-xfixes0 amd64 1.14-2 [9296 B]
Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 libxfixes3 amd64 1:5.0.3-2 [10.9 kB]
Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 libxshmfence1 amd64 1.3-1 [5028 B]
Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 libxxf86vm1 amd64 1:1.1.4-1build1 [10.2 kB]
Get:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libglx-mesa0 amd64 21.2.6-0ubuntu0.1~20.04.2 [137 kB]
Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libglx0 amd64 1.3.2-1~ubuntu0.20.04.2 [32.5 kB]
Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgl1 amd64 1.3.2-1~ubuntu0.20.04.2 [85.8 kB]
Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgl1-mesa-glx amd64 21.2.6-0ubuntu0.1~20.04.2 [5536 B]
Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libwayland-client0 amd64 1.18.0-1ubuntu0.1 [23.9 kB]
Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-randr0 amd64 1.14-2 [16.3 kB]
Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 mesa-vulkan-drivers amd64 21.2.6-0ubuntu0.1~20.04.2 [5788 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 36.4 MB in 9s (3864 kB/s)
Selecting previously unselected package libdrm-common.
(Reading database ... 19422 files and directories currently installed.)
Preparing to unpack .../00-libdrm-common_2.4.107-8ubuntu1~20.04.2_all.deb ...
Unpacking libdrm-common (2.4.107-8ubuntu1~20.04.2) ...
Selecting previously unselected package libdrm2:amd64.
Preparing to unpack .../01-libdrm2_2.4.107-8ubuntu1~20.04.2_amd64.deb ...
Unpacking libdrm2:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Selecting previously unselected package libedit2:amd64.
Preparing to unpack .../02-libedit2_3.1-20191231-1_amd64.deb ...
Unpacking libedit2:amd64 (3.1-20191231-1) ...
Selecting previously unselected package libdrm-amdgpu1:amd64.
Preparing to unpack .../03-libdrm-amdgpu1_2.4.107-8ubuntu1~20.04.2_amd64.deb ...
Unpacking libdrm-amdgpu1:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Selecting previously unselected package libpciaccess0:amd64.
Preparing to unpack .../04-libpciaccess0_0.16-0ubuntu1_amd64.deb ...
Unpacking libpciaccess0:amd64 (0.16-0ubuntu1) ...
Selecting previously unselected package libdrm-intel1:amd64.
Preparing to unpack .../05-libdrm-intel1_2.4.107-8ubuntu1~20.04.2_amd64.deb ...
Unpacking libdrm-intel1:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Selecting previously unselected package libdrm-nouveau2:amd64.
Preparing to unpack .../06-libdrm-nouveau2_2.4.107-8ubuntu1~20.04.2_amd64.deb ...
Unpacking libdrm-nouveau2:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Selecting previously unselected package libdrm-radeon1:amd64.
Preparing to unpack .../07-libdrm-radeon1_2.4.107-8ubuntu1~20.04.2_amd64.deb ...
Unpacking libdrm-radeon1:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Selecting previously unselected package libglapi-mesa:amd64.
Preparing to unpack .../08-libglapi-mesa_21.2.6-0ubuntu0.1~20.04.2_amd64.deb ...
Unpacking libglapi-mesa:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Selecting previously unselected package libllvm12:amd64.
Preparing to unpack .../09-libllvm12_1%3a12.0.0-3ubuntu1~20.04.5_amd64.deb ...
Unpacking libllvm12:amd64 (1:12.0.0-3ubuntu1~20.04.5) ...
Selecting previously unselected package libsensors-config.
Preparing to unpack .../10-libsensors-config_1%3a3.6.0-2ubuntu1.1_all.deb ...
Unpacking libsensors-config (1:3.6.0-2ubuntu1.1) ...
Selecting previously unselected package libsensors5:amd64.
Preparing to unpack .../11-libsensors5_1%3a3.6.0-2ubuntu1.1_amd64.deb ...
Unpacking libsensors5:amd64 (1:3.6.0-2ubuntu1.1) ...
Selecting previously unselected package libvulkan1:amd64.
Preparing to unpack .../12-libvulkan1_1.2.131.2-1_amd64.deb ...
Unpacking libvulkan1:amd64 (1.2.131.2-1) ...
Selecting previously unselected package libgl1-mesa-dri:amd64.
Preparing to unpack .../13-libgl1-mesa-dri_21.2.6-0ubuntu0.1~20.04.2_amd64.deb ...
Unpacking libgl1-mesa-dri:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Selecting previously unselected package libglvnd0:amd64.
Preparing to unpack .../14-libglvnd0_1.3.2-1~ubuntu0.20.04.2_amd64.deb ...
Unpacking libglvnd0:amd64 (1.3.2-1~ubuntu0.20.04.2) ...
Selecting previously unselected package libx11-xcb1:amd64.
Preparing to unpack .../15-libx11-xcb1_2%3a1.6.9-2ubuntu1.2_amd64.deb ...
Unpacking libx11-xcb1:amd64 (2:1.6.9-2ubuntu1.2) ...
Selecting previously unselected package libxcb-dri2-0:amd64.
Preparing to unpack .../16-libxcb-dri2-0_1.14-2_amd64.deb ...
Unpacking libxcb-dri2-0:amd64 (1.14-2) ...
Selecting previously unselected package libxcb-dri3-0:amd64.
Preparing to unpack .../17-libxcb-dri3-0_1.14-2_amd64.deb ...
Unpacking libxcb-dri3-0:amd64 (1.14-2) ...
Selecting previously unselected package libxcb-glx0:amd64.
Preparing to unpack .../18-libxcb-glx0_1.14-2_amd64.deb ...
Unpacking libxcb-glx0:amd64 (1.14-2) ...
Selecting previously unselected package libxcb-present0:amd64.
Preparing to unpack .../19-libxcb-present0_1.14-2_amd64.deb ...
Unpacking libxcb-present0:amd64 (1.14-2) ...
Selecting previously unselected package libxcb-shm0:amd64.
Preparing to unpack .../20-libxcb-shm0_1.14-2_amd64.deb ...
Unpacking libxcb-shm0:amd64 (1.14-2) ...
Selecting previously unselected package libxcb-sync1:amd64.
Preparing to unpack .../21-libxcb-sync1_1.14-2_amd64.deb ...
Unpacking libxcb-sync1:amd64 (1.14-2) ...
Selecting previously unselected package libxcb-xfixes0:amd64.
Preparing to unpack .../22-libxcb-xfixes0_1.14-2_amd64.deb ...
Unpacking libxcb-xfixes0:amd64 (1.14-2) ...
Selecting previously unselected package libxfixes3:amd64.
Preparing to unpack .../23-libxfixes3_1%3a5.0.3-2_amd64.deb ...
Unpacking libxfixes3:amd64 (1:5.0.3-2) ...
Selecting previously unselected package libxshmfence1:amd64.
Preparing to unpack .../24-libxshmfence1_1.3-1_amd64.deb ...
Unpacking libxshmfence1:amd64 (1.3-1) ...
Selecting previously unselected package libxxf86vm1:amd64.
Preparing to unpack .../25-libxxf86vm1_1%3a1.1.4-1build1_amd64.deb ...
Unpacking libxxf86vm1:amd64 (1:1.1.4-1build1) ...
Selecting previously unselected package libglx-mesa0:amd64.
Preparing to unpack .../26-libglx-mesa0_21.2.6-0ubuntu0.1~20.04.2_amd64.deb ...
Unpacking libglx-mesa0:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Selecting previously unselected package libglx0:amd64.
Preparing to unpack .../27-libglx0_1.3.2-1~ubuntu0.20.04.2_amd64.deb ...
Unpacking libglx0:amd64 (1.3.2-1~ubuntu0.20.04.2) ...
Selecting previously unselected package libgl1:amd64.
Preparing to unpack .../28-libgl1_1.3.2-1~ubuntu0.20.04.2_amd64.deb ...
Unpacking libgl1:amd64 (1.3.2-1~ubuntu0.20.04.2) ...
Selecting previously unselected package libgl1-mesa-glx:amd64.
Preparing to unpack .../29-libgl1-mesa-glx_21.2.6-0ubuntu0.1~20.04.2_amd64.deb ...
Unpacking libgl1-mesa-glx:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Selecting previously unselected package libwayland-client0:amd64.
Preparing to unpack .../30-libwayland-client0_1.18.0-1ubuntu0.1_amd64.deb ...
Unpacking libwayland-client0:amd64 (1.18.0-1ubuntu0.1) ...
Selecting previously unselected package libxcb-randr0:amd64.
Preparing to unpack .../31-libxcb-randr0_1.14-2_amd64.deb ...
Unpacking libxcb-randr0:amd64 (1.14-2) ...
Selecting previously unselected package mesa-vulkan-drivers:amd64.
Preparing to unpack .../32-mesa-vulkan-drivers_21.2.6-0ubuntu0.1~20.04.2_amd64.deb ...
Unpacking mesa-vulkan-drivers:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Setting up libxcb-dri3-0:amd64 (1.14-2) ...
Setting up libx11-xcb1:amd64 (2:1.6.9-2ubuntu1.2) ...
Setting up libpciaccess0:amd64 (0.16-0ubuntu1) ...
Setting up libxcb-xfixes0:amd64 (1.14-2) ...
Setting up libglvnd0:amd64 (1.3.2-1~ubuntu0.20.04.2) ...
Setting up libxcb-glx0:amd64 (1.14-2) ...
Setting up libedit2:amd64 (3.1-20191231-1) ...
Setting up libsensors-config (1:3.6.0-2ubuntu1.1) ...
Setting up libxcb-shm0:amd64 (1.14-2) ...
Setting up libxxf86vm1:amd64 (1:1.1.4-1build1) ...
Setting up libxcb-present0:amd64 (1.14-2) ...
Setting up libxfixes3:amd64 (1:5.0.3-2) ...
Setting up libxcb-sync1:amd64 (1.14-2) ...
Setting up libllvm12:amd64 (1:12.0.0-3ubuntu1~20.04.5) ...
Setting up libsensors5:amd64 (1:3.6.0-2ubuntu1.1) ...
Setting up libglapi-mesa:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Setting up libvulkan1:amd64 (1.2.131.2-1) ...
Setting up libxcb-dri2-0:amd64 (1.14-2) ...
Setting up libxshmfence1:amd64 (1.3-1) ...
Setting up libxcb-randr0:amd64 (1.14-2) ...
Setting up libdrm-common (2.4.107-8ubuntu1~20.04.2) ...
Setting up libwayland-client0:amd64 (1.18.0-1ubuntu0.1) ...
Setting up libdrm2:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Setting up libdrm-amdgpu1:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Setting up mesa-vulkan-drivers:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Setting up libdrm-nouveau2:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Setting up libdrm-radeon1:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Setting up libdrm-intel1:amd64 (2.4.107-8ubuntu1~20.04.2) ...
Setting up libgl1-mesa-dri:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Setting up libglx-mesa0:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Setting up libglx0:amd64 (1.3.2-1~ubuntu0.20.04.2) ...
Setting up libgl1:amd64 (1.3.2-1~ubuntu0.20.04.2) ...
Setting up libgl1-mesa-glx:amd64 (21.2.6-0ubuntu0.1~20.04.2) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Removing intermediate container 7f852e4233de
 ---> bb26b7cc9564
Step 18/28 : RUN pip3 uninstall torch torch vision -y
 ---> Running in 28bf2387c1fa
Found existing installation: torch 1.13.0
Uninstalling torch-1.13.0:
  Successfully uninstalled torch-1.13.0
WARNING: Skipping vision as it is not installed.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 28bf2387c1fa
 ---> 8edb3147f633
Step 19/28 : RUN pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
 ---> Running in cc85eeb70a54
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.8.1+cu111
  Downloading https://download.pytorch.org/whl/cu111/torch-1.8.1%2Bcu111-cp38-cp38-linux_x86_64.whl (1982.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 GB 2.3 MB/s eta 0:00:00
Collecting torchvision==0.9.1+cu111
  Downloading https://download.pytorch.org/whl/cu111/torchvision-0.9.1%2Bcu111-cp38-cp38-linux_x86_64.whl (17.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.6/17.6 MB 5.2 MB/s eta 0:00:00
Collecting torchaudio==0.8.1
  Downloading torchaudio-0.8.1-cp38-cp38-manylinux1_x86_64.whl (1.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 2.4 MB/s eta 0:00:00
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from torch==1.8.1+cu111) (1.23.5)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch==1.8.1+cu111) (4.4.0)
Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.8/dist-packages (from torchvision==0.9.1+cu111) (9.3.0)
Installing collected packages: torch, torchvision, torchaudio
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.14.0
    Uninstalling torchvision-0.14.0:
      Successfully uninstalled torchvision-0.14.0
Successfully installed torch-1.8.1+cu111 torchaudio-0.8.1 torchvision-0.9.1+cu111
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container cc85eeb70a54
 ---> 7bb87a7f0e20
Step 20/28 : WORKDIR /opt/nuclio
 ---> Running in 42ee9c3cdead
Removing intermediate container 42ee9c3cdead
 ---> b7b6990c8529
Step 21/28 : COPY artifacts/processor /usr/local/bin/processor
 ---> d0cfb039f86f
Step 22/28 : COPY artifacts/py /opt/nuclio/
 ---> ffdb0b56a264
Step 23/28 : COPY artifacts/py3.8-whl /opt/nuclio/whl
 ---> 0d60dc4905f8
Step 24/28 : COPY artifacts/uhttpc /usr/local/bin/uhttpc
 ---> 68f5f00f562e
Step 25/28 : COPY handler /opt/nuclio
 ---> 29cd2875071b
Step 26/28 : HEALTHCHECK --interval=1s --timeout=3s CMD /usr/local/bin/uhttpc --url http://127.0.0.1:8082/ready || exit 1
 ---> Running in 78d523402ad5
Removing intermediate container 78d523402ad5
 ---> c9c4afb999cb
Step 27/28 : RUN python /opt/nuclio/whl/$(basename /opt/nuclio/whl/pip-*.whl)/pip install pip --no-index --find-links /opt/nuclio/whl && python -m pip install nuclio-sdk msgpack --no-index --find-links /opt/nuclio/whl
 ---> Running in dca7d796faf4
/bin/sh: 1: python: not found
Removing intermediate container dca7d796faf4

stderr:
The command '/bin/sh -c python /opt/nuclio/whl/$(basename /opt/nuclio/whl/pip-*.whl)/pip install pip --no-index --find-links /opt/nuclio/whl && python -m pip install nuclio-sdk msgpack --no-index --find-links /opt/nuclio/whl' returned a non-zero code: 127

    /nuclio/pkg/cmdrunner/shellrunner.go:96
Failed to build
    /nuclio/pkg/dockerclient/shell.go:117
Failed to build docker image
    .../pkg/containerimagebuilderpusher/docker.go:54
Failed to build processor image
    /nuclio/pkg/processor/build/builder.go:263
Failed to deploy function
    ...//nuclio/pkg/platform/abstract/platform.go:197
  NAMESPACE |        NAME        | PROJECT |  STATE   | REPLICAS | NODE PORT  
  nuclio    | openvino-dextr     | cvat    | ready    | 1/1      |     49154  
  nuclio    | pth-saic-vul-fbrs  | cvat    | ready    | 1/1      |     49157  
  nuclio    | pth-saic-vul-hrnet | cvat    | building | 1/1      |

It looks like python is missing(/bin/sh: 1: python: not found ) in the building docker image...

cvat-ai / cvat