search

snowzach / doods

DOODS - Dedicated Open Object Detection Service
MIT License
303 stars 31 forks source link

Docker image tagged 'cuda' does not work. Not listening for API calls #43

Closed deepcoder closed 3 years ago

deepcoder commented 3 years ago

When I tried to use the cuda docker image, the api http does not respond. just says 'connection refused', the following command was used:

docker run --name="doods-gpu" --gpus all -i -t -d -v /home/user/doods-gpu/models:/opt/doods/models -v /home/user/doods-gpu/config.yaml:/opt/doods/config.yaml -p 8080:8080 snowzach/doods:cuda

I have successfully run the 'latest' docker image with the command:

docker run -i -t -d --device /dev/bus/usb -v /home/user/doods/models:/opt/doods/models -v /home/user/doods/config.yaml:/opt/doods/config.yaml -p 8080:8080 snowzach/doods:latest

I only have one container at time running.

Also, I was not able to build the CUDA image locally, I receive an error:

unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /home/user/doods-build/Dockerfile: no such file or directory

The docker log output from the cuda container does not seem to launch the http listener, or something like that. The log from the 'latest' container show API listening events.

Thanks for your work on this and assistance!

docker log from cuda:

2020-11-03 00:11:59.860454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2

2020-11-03 00:12:00.247512: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2020-11-03 00:12:00.257622: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1

2020-11-03 00:12:00.276125: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2020-11-03 00:12:00.276884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties: 

pciBusID: 0000:06:00.0 name: GeForce GTX 1070 computeCapability: 6.1

coreClock: 1.683GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s

2020-11-03 00:12:00.276903: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2

2020-11-03 00:12:00.286894: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10

2020-11-03 00:12:00.288231: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10

2020-11-03 00:12:00.289245: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10

2020-11-03 00:12:00.292388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10

2020-11-03 00:12:00.293156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10

2020-11-03 00:12:00.298797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7

2020-11-03 00:12:00.298929: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2020-11-03 00:12:00.299740: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2020-11-03 00:12:00.300415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0

docker log from latest:

,2020-11-02T23:24:34.695Z   INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "tensorflow", "type": "tensorflow", "model": "models/faster_rcnn_inception_v2_coco_2018_01_28.pb", "labels": 65, "width": -1, "height": -1}
,2020-11-02T23:24:34.697Z   INFO    server/server.go:284    API Listening   {"package": "server", "address": ":8080", "tls": false, "version": "v0.2.5-0-gbf6d7a1-dirty"}
,2020-11-03T00:09:08.891Z   INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "default", "type": "tflite-edgetpu", "model": "models/coco_ssd_mobilenet_v1_1.0_quant.tflite", "labels": 80, "width": 300, "height": 300}
,2020-11-03T00:09:08.908Z   INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "edgetpu", "type": "tflite-edgetpu", "model": "models/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite", "labels": 80, "width": 300, "height": 300}
,2020-11-03 00:09:09.227098: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX
,To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
,2020-11-03T00:09:09.235Z   INFO    detector/detector.go:79 Configured Detector {"package": "detector", "name": "tensorflow", "type": "tensorflow", "model": "models/faster_rcnn_inception_v2_coco_2018_01_28.pb", "labels": 65, "width": -1, "height": -1}
,2020-11-03T00:09:09.235Z   INFO    server/server.go:284    API Listening   {"package": "server", "address": ":8080", "tls": false, "version": "v0.2.5-0-gbf6d7a1-dirty"}
,

CUDA build fail…

cp Dockerfile.base.cuda Dockerfile.
user@ubuntu-macmini2012:~/doods-build$ docker build --tag localdoods .
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /home/user/doods-build/Dockerfile: no such file or directory
snowzach commented 3 years ago

Hey @deepcoder it's probably working. It has to "compile" something for the GPU you are using. I am not sure what exactly it is doing but on my GTX970, it will take a good 2-3 minutes before it starts working. (It's working and ready when it says "API Listening") I guess this is somewhat normal. It's pretty annoying though. I hope I can eventually get it sorted out and have it run faster.

deepcoder commented 3 years ago

Thanks for your reply. You are correct! Patience is a virtue and bigger is not always faster 970 vs 1070 ;-) it took almost 10 minutes for my config to initialize the model.

Log below, ignore the model config name of "edgetpu", I am sliding a different directory underneath HA's configuration.yaml so I can switch between the Coral and GTX 1070


,2020-11-03 06:53:42.753568: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
,2020-11-03 06:53:42.753599: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
,2020-11-03 06:53:42.753723: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
,2020-11-03 06:53:42.754355: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
,2020-11-03 06:53:42.755236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
,2020-11-03 06:53:42.755328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1085] Device interconnect StreamExecutor with strength 1 edge matrix:
,2020-11-03 06:53:42.755371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1091]      0 
,2020-11-03 06:53:42.755390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1104] 0:   N 
,2020-11-03 06:53:42.755523: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
,2020-11-03 06:53:42.756020: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
,2020-11-03 06:53:42.756706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1230] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7220 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:06:00.0, compute capability: 6.1)
,2020-11-03T06:53:42.881Z   INFO    tensorflow/tensorflow.go:259    Detection Complete  {"package": "detector.tensorflow", "name": "edgetpu", "id": "", "duration": 0.103930269, "detections": 0}
,2020-11-03T06:53:42.881Z   INFO    server/server.go:138    HTTP Request    {"status": 200, "took": 0.137886125, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "668518e45e16/MZ2smQsWao-000334", "remote": "192.168.2.242:42768"}
,
nvtop-doods