facebookresearch / AnimatedDrawings

Code to accompany "A Method for Animating Children's Drawings of the Human Figure"
MIT License
10.31k stars 885 forks source link

Failed to get bounding box #262

Closed gaurish1212 closed 2 months ago

gaurish1212 commented 5 months ago

Hy

When we build image after change version of mmsc-full==1.7.0 instaed mmcv-full==1.7.2. Then image build successfully and we get status healthy. Then after we hit this command "python image_to_animation.py drawings/garlic.png garlic_out" then we get error. Traceback (most recent call last): File "image_to_animation.py", line 41, in image_to_animation(img_fn, char_anno_dir, motion_cfg_fn, retarget_cfg_fn) File "image_to_animation.py", line 19, in image_to_animation image_to_annotations(img_fn, char_anno_dir) File "C:\Users\DELL4\AnimatedDrawings\examples\image_to_annotations.py", line 53, in image_to_annotations raise Exception(f"Failed to get bounding box, please check if the 'docker_torchserve' is running and healthy, resp: {resp}") Exception: Failed to get bounding box, please check if the 'docker_torchserve' is running and healthy, resp: <Response [500]> image Screenshot 2024-01-09 204931

lbj96347 commented 5 months ago

I guess Torchserve is failing because it doesn't have enough RAM. You may also try to increase RAM for docker.

How about checking your RAM after u enter python image_to_animation.py drawings/garlic.png garlic_out ?

gaurish1212 commented 5 months ago

Hy How can i increase ram for docker. In my system have 32GB ram

corporateGhost commented 5 months ago

Same Question

lbj96347 commented 5 months ago

@gaurish1212 by default, docker can use unlimited RAM in your system.

After running docker run -d --name docker_torchserve -p 8080:8080 -p 8081:8081 docker_torchserve

Try to get more detail from container : docker exec -it docker_torchserve bash

All logs while running docker_torchserve cat /home/torchserve/logs/ts_log.log

Then we may figure out which part goes wrong. Or post it here.

Also I found a simular issue: https://github.com/facebookresearch/AnimatedDrawings/issues/136

lbj96347 commented 5 months ago

@gaurish1212 try it: https://github.com/facebookresearch/AnimatedDrawings/issues/240#issuecomment-1784019869

corporateGhost commented 5 months ago

@gaurish1212 try it: #240 (comment)

I tried this. I changed it from RUN pip install torch to RUN pip install torch==2.0.0 but still it gives me the same error

corporateGhost commented 5 months ago

@gaurish1212 by default, docker can use unlimited RAM in your system.

After running docker run -d --name docker_torchserve -p 8080:8080 -p 8081:8081 docker_torchserve

Try to get more detail from container : docker exec -it docker_torchserve bash

All logs while running docker_torchserve cat /home/torchserve/logs/ts_log.log

Then we may figure out which part goes wrong. Or post it here.

Also I found a simular issue: #136

I tried this and I am getting this (base) H:\AnimatedDrawings\torchserve>docker exec -it docker_torchserve bash (base) root@6eab2021b0eb:/# cat /home/torchserve/logs/ts_log.log cat: /home/torchserve/logs/ts_log.log: No such file or directory

lbj96347 commented 5 months ago

@corporateGhost if u can't see ts_log.log , u can try to run bash inside docker container.

step 1:

docker exec -it docker_torchserve bash

step 2:

torchserve --stop

step3:

/opt/conda/bin/torchserve --start --ts-config /home/torchserve/config.properties

Now you can see logs popping up.

gaurish1212 commented 3 months ago

This is our logs 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ext_module = ext_loader.load_ext( 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ext = importlib.import_module('mmcv.' + name) 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:17:46,825 [WARN ] W-9015-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9015-drawn_humanoid_detector_1.0-stderr 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:17:46,825 [WARN ] W-9015-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9015-drawn_humanoid_detector_1.0-stdout 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9015-drawn_humanoid_detector_1.0-stdout 2024-03-20T12:17:46,825 [INFO ] W-9015-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9015 in 55 seconds. 2024-03-20T12:17:46,835 [INFO ] W-9015-drawn_humanoid_detector_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9015-drawn_humanoid_detector_1.0-stderr 2024-03-20T12:17:46,839 [WARN ] W-9023-drawn_humanoid_detector_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:17:46,839 [WARN ] W-9023-drawn_humanoid_detector_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:17:46,840 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - proceeding without onnxruntime 2024-03-20T12:17:46,840 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - Torch TensorRT not enabled 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:17:46,843 [INFO ] epollEventLoopGroup-5-23 org.pytorch.serve.wlm.WorkerThread - 9023 Worker disconnected. WORKER_STARTED 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:17:46,843 [DEBUG] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:17:46,843 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/tmp/models/8a762892a49747d4b3d43f92aafb0323/mmdet_handler.py", line 9, in 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - from mmdet.apis import inference_detector, init_detector 2024-03-20T12:17:46,843 [DEBUG] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmdet/apis/init.py", line 2, in 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - from .inference import (async_inference_detector, inference_detector, 2024-03-20T12:17:46,844 [WARN ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_detector, error: Worker died. 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmdet/apis/inference.py", line 8, in 2024-03-20T12:17:46,844 [DEBUG] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - W-9023-drawn_humanoid_detector_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - from mmcv.ops import RoIPool 2024-03-20T12:17:46,844 [WARN ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/ops/init.py", line 3, in 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - from .active_rotated_filter import active_rotated_filter 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ext_module = ext_loader.load_ext( 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ext = importlib.import_module('mmcv.' + name) 2024-03-20T12:17:46,844 [WARN ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9023-drawn_humanoid_detector_1.0-stderr 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:17:46,844 [WARN ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9023-drawn_humanoid_detector_1.0-stdout 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9023 in 55 seconds. 2024-03-20T12:17:46,844 [INFO ] W-9023-drawn_humanoid_detector_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9023-drawn_humanoid_detector_1.0-stdout 2024-03-20T12:17:46,854 [INFO ] W-9023-drawn_humanoid_detector_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9023-drawn_humanoid_detector_1.0-stderr 2024-03-20T12:18:27,222 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9001, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:27,936 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9002, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:28,050 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9001, pid=14447 2024-03-20T12:18:28,050 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9001 2024-03-20T12:18:28,056 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:28,057 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14447 2024-03-20T12:18:28,057 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:28,057 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:28,057 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:28,057 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9001 2024-03-20T12:18:28,058 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9001. 2024-03-20T12:18:28,058 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937108058 2024-03-20T12:18:28,058 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937108058 2024-03-20T12:18:28,059 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:28,438 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:28,438 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server() 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:28,439 [INFO ] epollEventLoopGroup-5-6 org.pytorch.serve.wlm.WorkerThread - 9001 Worker disconnected. WORKER_STARTED 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:18:28,439 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:18:28,439 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:18:28,439 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in 2024-03-20T12:18:28,440 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died. 2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model, 2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in 2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)), \ 2024-03-20T12:18:28,440 [DEBUG] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9001-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:28,440 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0. 2024-03-20T12:18:28,440 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9001-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:28,440 [WARN ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9001-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:28,440 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9001 in 89 seconds. 2024-03-20T12:18:28,449 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9001-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:28,449 [INFO ] W-9001-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9001-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:28,806 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9002, pid=14461 2024-03-20T12:18:28,806 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9002 2024-03-20T12:18:28,813 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:28,813 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14461 2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:28,814 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9002 2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9002. 2024-03-20T12:18:28,814 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937108814 2024-03-20T12:18:28,814 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937108814 2024-03-20T12:18:28,815 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:29,154 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:29,154 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server() 2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-03-20T12:18:29,155 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:29,156 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9002 Worker disconnected. WORKER_STARTED 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:18:29,156 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:18:29,156 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:18:29,156 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died. 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:18:29,156 [DEBUG] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9002-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:18:29,156 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in 2024-03-20T12:18:29,156 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model, 2024-03-20T12:18:29,157 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9002-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:29,157 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in 2024-03-20T12:18:29,157 [WARN ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9002-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:29,157 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)), \ 2024-03-20T12:18:29,157 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9002-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:29,157 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9002 in 89 seconds. 2024-03-20T12:18:29,166 [INFO ] W-9002-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9002-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:29,297 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:30,115 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=14479 2024-03-20T12:18:30,115 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14479 2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:30,121 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:30,121 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2024-03-20T12:18:30,122 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2024-03-20T12:18:30,122 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937110122 2024-03-20T12:18:30,122 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937110122 2024-03-20T12:18:30,122 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:30,465 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:30,465 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server() 2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-03-20T12:18:30,466 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-03-20T12:18:30,466 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:18:30,467 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:18:30,467 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:18:30,467 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died. 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:18:30,467 [DEBUG] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:18:30,467 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:30,467 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:18:30,468 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:18:30,468 [WARN ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:30,468 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 89 seconds. 2024-03-20T12:18:30,484 [INFO ] W-9000-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:31,949 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9010, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:32,747 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9007, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:32,776 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9010, pid=14495 2024-03-20T12:18:32,776 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9010 2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14495 2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:32,785 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:32,785 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9010 2024-03-20T12:18:32,786 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9010. 2024-03-20T12:18:32,786 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937112786 2024-03-20T12:18:32,786 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937112786 2024-03-20T12:18:32,786 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:33,163 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:33,163 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server() 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:18:33,165 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in 2024-03-20T12:18:33,166 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model, 2024-03-20T12:18:33,167 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in 2024-03-20T12:18:33,167 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)), \ 2024-03-20T12:18:33,167 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,167 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0. 2024-03-20T12:18:33,167 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9010 Worker disconnected. WORKER_STARTED 2024-03-20T12:18:33,167 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:18:33,167 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:18:33,167 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died. 2024-03-20T12:18:33,167 [DEBUG] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9010-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:18:33,167 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:18:33,168 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9010-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:33,168 [WARN ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9010-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:33,168 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9010 in 89 seconds. 2024-03-20T12:18:33,179 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9010-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:33,180 [INFO ] W-9010-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9010-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:33,607 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9007, pid=14509 2024-03-20T12:18:33,607 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9007 2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14509 2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:33,614 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:33,614 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9007 2024-03-20T12:18:33,615 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9007. 2024-03-20T12:18:33,615 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937113615 2024-03-20T12:18:33,615 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937113615 2024-03-20T12:18:33,615 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:33,943 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:33,944 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:33,944 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:33,944 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:33,944 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-03-20T12:18:33,944 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server() 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,945 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9007 Worker disconnected. WORKER_STARTED 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,945 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,945 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:18:33,945 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died. 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:18:33,945 [DEBUG] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9007-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:18:33,945 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:18:33,945 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model, 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)), \ 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:33,946 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9007-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0. 2024-03-20T12:18:33,946 [WARN ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9007-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9007-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:33,946 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9007 in 89 seconds. 2024-03-20T12:18:33,955 [INFO ] W-9007-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9007-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:34,688 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9005, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:34,803 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9011, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:34,852 [DEBUG] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9006, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:34,878 [DEBUG] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9008, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:34,882 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9003, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:34,889 [DEBUG] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9009, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:34,924 [DEBUG] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/opt/conda/bin/python, /opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9004, --metrics-config, /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml] 2024-03-20T12:18:35,947 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9005, pid=14527 2024-03-20T12:18:35,947 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9005 2024-03-20T12:18:35,956 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:35,956 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14527 2024-03-20T12:18:35,957 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:35,957 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:35,957 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:35,957 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9005 2024-03-20T12:18:35,958 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9005. 2024-03-20T12:18:35,958 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937115958 2024-03-20T12:18:35,958 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937115958 2024-03-20T12:18:35,958 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:36,172 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9006, pid=14533 2024-03-20T12:18:36,172 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9006 2024-03-20T12:18:36,173 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9003, pid=14539 2024-03-20T12:18:36,173 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9003 2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14533 2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:36,179 [DEBUG] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9006-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:36,179 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9006 2024-03-20T12:18:36,180 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9006. 2024-03-20T12:18:36,180 [DEBUG] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116180 2024-03-20T12:18:36,180 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116180 2024-03-20T12:18:36,181 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14539 2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:36,182 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:36,182 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9003 2024-03-20T12:18:36,183 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9003. 2024-03-20T12:18:36,183 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116183 2024-03-20T12:18:36,183 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116183 2024-03-20T12:18:36,183 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:36,187 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9011, pid=14530 2024-03-20T12:18:36,187 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9011 2024-03-20T12:18:36,194 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:36,194 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14530 2024-03-20T12:18:36,195 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:36,195 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:36,195 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:36,195 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9011 2024-03-20T12:18:36,196 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9011. 2024-03-20T12:18:36,196 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116196 2024-03-20T12:18:36,196 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116196 2024-03-20T12:18:36,196 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:36,215 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9008, pid=14536 2024-03-20T12:18:36,215 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9008 2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14536 2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:36,222 [DEBUG] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9008-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:36,222 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9008 2024-03-20T12:18:36,223 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9008. 2024-03-20T12:18:36,223 [DEBUG] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116223 2024-03-20T12:18:36,223 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116223 2024-03-20T12:18:36,224 [INFO ] W-9008-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:36,239 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9009, pid=14542 2024-03-20T12:18:36,239 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9009 2024-03-20T12:18:36,247 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:36,247 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14542 2024-03-20T12:18:36,247 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:36,247 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:36,247 [DEBUG] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9009-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:36,248 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9009 2024-03-20T12:18:36,248 [DEBUG] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116248 2024-03-20T12:18:36,248 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9009. 2024-03-20T12:18:36,249 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116249 2024-03-20T12:18:36,249 [INFO ] W-9009-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:36,319 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9004, pid=14545 2024-03-20T12:18:36,319 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9004 2024-03-20T12:18:36,328 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Successfully loaded /opt/conda/lib/python3.11/site-packages/ts/configs/metrics.yaml. 2024-03-20T12:18:36,328 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - [PID]14545 2024-03-20T12:18:36,328 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Torch worker started. 2024-03-20T12:18:36,328 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Python runtime: 3.11.7 2024-03-20T12:18:36,328 [DEBUG] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9004-drawn_humanoid_pose_estimator_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2024-03-20T12:18:36,329 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9004 2024-03-20T12:18:36,329 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9004. 2024-03-20T12:18:36,329 [DEBUG] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1710937116329 2024-03-20T12:18:36,329 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1710937116329 2024-03-20T12:18:36,330 [INFO ] W-9004-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - model_name: drawn_humanoid_pose_estimator, batchSize: 1 2024-03-20T12:18:36,420 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:36,420 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server() 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,422 [INFO ] epollEventLoopGroup-5-5 org.pytorch.serve.wlm.WorkerThread - 9005 Worker disconnected. WORKER_STARTED 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-03-20T12:18:36,422 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:18:36,422 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:18:36,423 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:18:36,423 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died. 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,423 [DEBUG] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9005-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:18:36,423 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:18:36,423 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9005-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:36,423 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:18:36,424 [WARN ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9005-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:36,424 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9005 in 89 seconds. 2024-03-20T12:18:36,424 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-03-20T12:18:36,424 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9005-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:36,439 [INFO ] W-9005-drawn_humanoid_pose_estimator_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9005-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:36,642 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:36,642 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server() 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-03-20T12:18:36,644 [INFO ] epollEventLoopGroup-5-9 org.pytorch.serve.wlm.WorkerThread - 9011 Worker disconnected. WORKER_STARTED 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:18:36,644 [WARN ] W-9003-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:36,644 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,644 [WARN ] W-9003-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:18:36,644 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,644 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died. 2024-03-20T12:18:36,644 [DEBUG] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9011-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:18:36,644 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:18:36,644 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:18:36,644 [WARN ] W-9006-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - /opt/conda/lib/python3.11/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. 2024-03-20T12:18:36,645 [WARN ] W-9006-drawn_humanoid_pose_estimator_1.0-stderr MODEL_LOG - warnings.warn( 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model, 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)), \ 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0. 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9011-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:36,645 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9011-drawn_humanoid_pose_estimator_1.0-stderr 2024-03-20T12:18:36,645 [WARN ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9011-drawn_humanoid_pose_estimator_1.0-stdout 2024-03-20T12:18:36,645 [INFO ] W-9011-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9011 in 89 seconds. 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - worker.run_server() 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 231, in run_server 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 194, in handle_connection 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 131, in load_model 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - service = model_loader.load( 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 108, in load 2024-03-20T12:18:36,646 [INFO ] epollEventLoopGroup-5-8 org.pytorch.serve.wlm.WorkerThread - 9003 Worker disconnected. WORKER_STARTED 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_loader.py", line 153, in _load_handler_file 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/importlib/init.py", line 126, in import_module 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 940, in exec_module 2024-03-20T12:18:36,646 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/tmp/models/34ba3c3d052d4021a508d4689663c503/mmpose_handler.py", line 8, in 2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - from mmpose.apis import (inference_bottom_up_pose_model, 2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/mmpose/init.py", line 25, in 2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - and mmcv_version <= digit_version(mmcv_maximum_version)), \ 2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-20T12:18:36,647 [INFO ] W-9003-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - AssertionError: MMCV==1.7.2 is used but incompatible. Please install mmcv>=1.3.8, <=1.7.0. 2024-03-20T12:18:36,646 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2024-03-20T12:18:36,647 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., responseTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:229) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-03-20T12:18:36,647 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Backend worker process died. 2024-03-20T12:18:36,647 [WARN ] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: drawn_humanoid_pose_estimator, error: Worker died. 2024-03-20T12:18:36,647 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2024-03-20T12:18:36,648 [DEBUG] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - W-9003-drawn_humanoid_pose_estimator_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2024-03-20T12:18:36,648 [WARN ] W-9003-drawn_humanoid_pose_estimator_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2024-03-20T12:18:36,648 [INFO ] W-9006-drawn_humanoid_pose_estimator_1.0-stdout MODEL_LOG - File "/opt/conda/lib/python3.11/site-packages/ts/model_service_worker.py", line 263, in

gaurish1212 commented 3 months ago

We didn't get any output image This is my docker image image

Please guide me

gaurish1212 commented 3 months ago

@corporateGhost if u can't see ts_log.log , u can try to run bash inside docker container.

step 1:

docker exec -it docker_torchserve bash

step 2:

torchserve --stop

step3:

/opt/conda/bin/torchserve --start --ts-config /home/torchserve/config.properties

Now you can see logs popping up.

Plaese Reply

gaurish1212 commented 3 months ago

Screenshot 2024-03-27 195402 Screenshot 2024-03-27 195344 Screenshot 2024-03-27 195335 Screenshot 2024-03-27 195320 Screenshot 2024-03-27 195252 Screenshot 2024-03-27 195227 Screenshot 2024-03-27 195213 This is my docker logs image what's wrong and how can i fix ?

hjessmith commented 3 months ago

I just merged a PR that deals with the Docker not correctly building. Please try pulling the latest version of the Animated Drawings code and following the installation steps again. Changing mmcv-full version as you did is triggering an assertion error in the torchserve handler code which is causing it to fail.

gaurish1212 commented 2 months ago

solved thanks