search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.62k stars 3.01k forks source link

fbrs semi automatic annotation nuclio function starts unhealhy #4736

Closed nikste closed 2 years ago

nikste commented 2 years ago

My actions before raising this issue

fbrs network for semi automatic annotations does not start with the current nuclio function code.

Am i doing something wrong? (please check steps to repeat)

Expected Behaviour

fbrs network for semi automatic annotation works after running deploy_cpu.sh

Current Behaviour

fbrs network for semi automatic annotation just says unhealthy when running nuctl get functions when trying to access in cvat it returns error code 500.

Possible Solution

it says something with `pickle key´ not found in docker logs. Did something change with the download url for the checkpoint?

Steps to Reproduce (for bugs)

  1. install prerequisites for nucio
  2. run cvat with docker compose on develop branch: docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d
  3. cd serverless
  4. deploy_cpu.sh (or only start fbrs network)
  5. wait till finished
  6. wait a bit more till function startup runs into error
  7. nuctl get functions
  8. check for pth-saic-vul-fbrs, it says unhealthy

Context

cant do semi automatic annotation

Your Environment

ubuntu 22.04

Next steps

You may join our Gitter channel for community support.

simone-viozzi commented 2 years ago

Same thing, on the nuclio dashboard, there are some logs:

 [13:16:32.906]  (I)  Deploying function    
 [13:16:32.906]  (I)  Building   [versionInfo: "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"] 
 [13:16:33.322]  (I)  Cleaning up before deployment   [functionName: "pth-saic-vul-fbrs"] 
 [13:16:33.368]  (I)  Function already exists, deleting function containers   [functionName: "pth-saic-vul-fbrs"] 
 [13:16:33.410]  (I)  Staging files and preparing base images    
 [13:16:33.478]  (W)  Failed to create a function; setting the function status  Failed to prepare staging dir  [errCauses: [{"error":"Failed to prepare staging dir","errorCauses":[{"error":"Failed to copy handler object","errorCauses":[{"error":"open /dev/core: permission denied"}],"errorVerbose":"
Error - open /dev/core: permission denied
    /nuclio/pkg/processor/build/builder.go:966

Call stack:
Failed to copy handler object
    /nuclio/pkg/processor/build/builder.go:966
Failed to copy handler object"}],"errorVerbose":"
Error - open /dev/core: permission denied
    /nuclio/pkg/processor/build/builder.go:966

the logs of the container are:

22.08.04 11:20:34.933                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:34.933                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:34.933 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:20:34.933            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:20:34.933 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn0jhhegdmp159dug.sock"}
22.08.04 11:20:34.933 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn0jhhegdmp159dv0.sock"}
22.08.04 11:20:34.933 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:34.933 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:34.933 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:34.933 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:34.933 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:34.933 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:34.933 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:34.933 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:34.933 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn0jhhegdmp159dug.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:34.933 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn0jhhegdmp159dv0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
/usr/local/lib/python3.6/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /opt/nuclio/fbrs/isegm/utils/cython/_get_dist_maps.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
/usr/local/lib/python3.6/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /opt/nuclio/fbrs/isegm/utils/cython/_get_dist_maps.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
In file included from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /root/.pyxbld/temp.linux-x86_64-3.6/pyrex/isegm/utils/cython/_get_dist_maps.cpp:760:
/usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
In file included from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /root/.pyxbld/temp.linux-x86_64-3.6/pyrex/isegm/utils/cython/_get_dist_maps.cpp:760:
/usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
22.08.04 11:20:38.159 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 23}
22.08.04 11:20:38.159 sor.http.w0.python.logger (D) Waiting for start
22.08.04 11:20:38.159 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 22}
22.08.04 11:20:38.159 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:38,159", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:38,159", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:38,160", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:38,160", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:38,160", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:38,160", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:38.160 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:38.160 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:38.399 sor.http.w0.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

goroutine 40 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc0006f0000)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:39.038                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:39.038                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:39.038 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:20:39.038            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:20:39.039 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn1rhhegdmp1pn7c0.sock"}
22.08.04 11:20:39.039 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn1rhhegdmp1pn7cg.sock"}
22.08.04 11:20:39.039 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:39.039 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:39.039 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:39.039 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:39.039 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:39.039 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:39.039 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn1rhhegdmp1pn7cg.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:39.039 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:39.039 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:39.039 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn1rhhegdmp1pn7c0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:40.440 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 24}
22.08.04 11:20:40.440 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:40,440", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
22.08.04 11:20:40.441 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 23}
22.08.04 11:20:40.441 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:40,441", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:40,441", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:40,442", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:40.442 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
l{"datetime": "2022-08-04 11:20:40,442", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:40,442", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:40.443 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:40.603 sor.http.w0.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

goroutine 13 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c0c0)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:41.208                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:41.208                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:41.208 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:20:41.208            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:20:41.208 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn2bhhegdmp48qat0.sock"}
22.08.04 11:20:41.208 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn2bhhegdmp48qatg.sock"}
22.08.04 11:20:41.208 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:41.208 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:41.208 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:41.208 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:41.208 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:41.208 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:41.208 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:41.208 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn2bhhegdmp48qat0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:41.208 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:41.209 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn2bhhegdmp48qatg.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:42.587 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 24}
22.08.04 11:20:42.587 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:42,587", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:42,588", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:42,589", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:42.589 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:42.594 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 23}
22.08.04 11:20:42.594 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:42,594", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:42,595", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:42,595", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:42.595 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:42.761 sor.http.w0.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

goroutine 81 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00059a000)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:43.525                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:43.525                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:43.525 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:20:43.525            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:20:43.525 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn2rhhegdmp7ls99g.sock"}
22.08.04 11:20:43.525 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn2rhhegdmp7ls9a0.sock"}
22.08.04 11:20:43.526 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:43.526 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:43.526 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:43.526 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:43.526 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:43.526 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:43.526 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:43.526 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:43.526 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn2rhhegdmp7ls99g.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:43.526 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn2rhhegdmp7ls9a0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:44.942 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 22}
22.08.04 11:20:44.942 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:44,942", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:44,943", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
22.08.04 11:20:44.943 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 23}
22.08.04 11:20:44.943 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:44,943", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:44,944", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:44.944 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
l{"datetime": "2022-08-04 11:20:44,944", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:44,945", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:44.945 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:45.119 sor.http.w1.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 28 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc000130900)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:46.268                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:46.268                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:46.268 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:20:46.268            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:20:46.268 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn3jhhegdmp0k8le0.sock"}
22.08.04 11:20:46.268 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn3jhhegdmp0k8leg.sock"}
22.08.04 11:20:46.268 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:46.268 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:46.268 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:46.268 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:46.268 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:46.268 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:46.268 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:46.268 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn3jhhegdmp0k8le0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:46.268 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:46.268 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn3jhhegdmp0k8leg.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:47.736 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 22}
22.08.04 11:20:47.736 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:47,736", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:47,737", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:47,737", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:47.737 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:47.764 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 23}
22.08.04 11:20:47.764 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:47,765", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:47,766", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:47,766", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:47.766 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:47.907 sor.http.w1.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 56 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c9c0)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:49.871                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:49.872                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:49.872 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:20:49.872            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:20:49.872 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn4bhhegdmp4gmp70.sock"}
22.08.04 11:20:49.872 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn4bhhegdmp4gmp7g.sock"}
22.08.04 11:20:49.872 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:49.872 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:49.872 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:49.872 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:49.872 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:49.872 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:49.872 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn4bhhegdmp4gmp70.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:49.872 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:49.872 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:49.872 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn4bhhegdmp4gmp7g.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:51.324 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 20}
22.08.04 11:20:51.324 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:51,324", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:51,325", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:51,325", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:51.326 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:51.413 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 21}
22.08.04 11:20:51.413 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:51,413", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:51,414", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:51,415", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:51.415 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:51.492 sor.http.w1.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 65 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c9c0)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:55.089                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:55.089                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:55.089 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:20:55.089            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:20:55.089 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn5rhhegdmp1l8kig.sock"}
22.08.04 11:20:55.089 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn5rhhegdmp1l8kj0.sock"}
22.08.04 11:20:55.089 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:55.089 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:55.089 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:55.089 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:55.089 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:55.089 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:20:55.089 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn5rhhegdmp1l8kig.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:55.089 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:55.089 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:55.089 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn5rhhegdmp1l8kj0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:56.544 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 22}
22.08.04 11:20:56.544 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:56,544", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:56,545", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:56,546", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:56.546 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:56.548 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 21}
22.08.04 11:20:56.548 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:20:56,548", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:56,549", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:56,550", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:56.550 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:20:56.718 sor.http.w0.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

goroutine 40 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c0c0)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:21:03.489                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:21:03.489                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:21:03.489 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:21:03.489            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:21:03.489 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn7rhhegdmp29ioi0.sock"}
22.08.04 11:21:03.489 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn7rhhegdmp29ioig.sock"}
22.08.04 11:21:03.489 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:03.489 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:21:03.489 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:03.489 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:21:03.489 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:03.489 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:03.489 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:03.489 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn7rhhegdmp29ioig.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:03.489 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:03.489 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn7rhhegdmp29ioi0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:04.888 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 23}
22.08.04 11:21:04.888 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:21:04,888", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:04,889", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:04,890", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:21:04.890 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:21:04.896 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 22}
22.08.04 11:21:04.896 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:21:04,896", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:04,898", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:04,898", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:21:04.898 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:21:05.053 sor.http.w1.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 28 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c9c0)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:21:18.204                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:21:18.204                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:21:18.205 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:21:18.205            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:21:18.205 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqnbjhhegdmp08targ.sock"}
22.08.04 11:21:18.205 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqnbjhhegdmp08tas0.sock"}
22.08.04 11:21:18.205 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:18.205 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:21:18.205 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:18.205 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:18.205 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqnbjhhegdmp08tas0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:18.205 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:18.205 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:21:18.205 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:18.205 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:18.205 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqnbjhhegdmp08targ.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:19.635 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 23}
22.08.04 11:21:19.635 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:21:19,635", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:19,636", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:19,636", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:21:19.636 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:21:19.670 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 22}
22.08.04 11:21:19.670 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:21:19,670", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:19,671", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:19,672", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:21:19.672 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:21:19.789 sor.http.w1.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 69 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c9c0)
    /nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
    /nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:21:45.780                 processor (I) Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:21:45.780                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:21:45.780 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
22.08.04 11:21:45.781            processor.http (D) Creating worker pool {"num": 2}
22.08.04 11:21:45.781 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqnibhhegdmp3puiqg.sock"}
22.08.04 11:21:45.781 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-cblqnibhhegdmp3puir0.sock"}
22.08.04 11:21:45.781 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:45.781 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:45.781 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:21:45.781 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
22.08.04 11:21:45.781 sor.http.w0.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:45.781 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:45.781 sor.http.w0.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqnibhhegdmp3puir0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:45.781 sor.http.w1.python.logger (D) Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:45.781 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:45.781 sor.http.w1.python.logger (D) Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqnibhhegdmp3puiqg.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:47.252 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 24}
22.08.04 11:21:47.252 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:21:47,252", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:47,253", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:47,254", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:21:47.254 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:21:47.272 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 23}
22.08.04 11:21:47.272 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2022-08-04 11:21:47,272", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:47,273", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:47,274", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:21:47.274 sor.http.w0.python.logger (W) Failed to read from connection {"err": "EOF"}
22.08.04 11:21:47.447 sor.http.w0.python.logger (E) Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1
amteusch commented 2 years ago

I ran into this error a few months ago and forgot to make a PR for it. Google Drive changed and so the previous method to download the model weights no longer works. I tested with my fix today and FBRS worked, however I could not get the Docker image for IOG (which has the same problem) to build properly. I may try to solve the problem and make a PR next week, but if you want to try the changes on your own you can see my branch here