search

snowzach / doods

DOODS - Dedicated Open Object Detection Service
MIT License
303 stars 31 forks source link

Is the GPU being used? #45

Closed scottgrobinson closed 3 years ago

scottgrobinson commented 3 years ago

I'm slightly confused by the log line:

Adding visible gpu devices: 0

When nvidia-smi is showing as the model being loaded (Albeit I don't think I have enough VRAM on this GFX, maybe that's the issue, maybe not)

Does this mean it's adding ZERO devices, or it's adding device 0?

2020-11-22 14:04:24.562677: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.562919: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro P600 computeCapability: 6.1
coreClock: 1.5565GHz coreCount: 3 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 59.75GiB/s
2020-11-22 14:04:24.562938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-11-22 14:04:24.562948: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-11-22 14:04:24.562956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-11-22 14:04:24.562963: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-11-22 14:04:24.562970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-11-22 14:04:24.562977: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-11-22 14:04:24.562984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-22 14:04:24.563023: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.563217: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.563382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
2020-11-22 14:04:24.563401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1085] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-22 14:04:24.563408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1091]      0
2020-11-22 14:04:24.563427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1104] 0:   N
2020-11-22 14:04:24.563496: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.563687: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.563868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1230] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1545 MB memory) -> physical GPU (device: 0, name: Quadro P600, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-11-22T14:04:24.774Z        INFO    tensorflow/tensorflow.go:259    Detection Complete      {"package": "detector.tensorflow", "name": "tensorflow", "id": "", "duration": 0.200685689, "detections": 1}
2020-11-22T14:04:24.774Z        INFO    server/server.go:138    HTTP Request    {"status": 200, "took": 0.224060097, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "34f6cb495836/9zLnpxQJf5-001880", "remote": "172.200.0.1:51374"}
2020-11-22 14:04:24.797009: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.797278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro P600 computeCapability: 6.1
coreClock: 1.5565GHz coreCount: 3 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 59.75GiB/s
2020-11-22 14:04:24.797297: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-11-22 14:04:24.797306: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-11-22 14:04:24.797314: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-11-22 14:04:24.797320: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-11-22 14:04:24.797326: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-11-22 14:04:24.797334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-11-22 14:04:24.797340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-22 14:04:24.797380: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.797575: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.797736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
2020-11-22 14:04:24.797755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1085] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-22 14:04:24.797762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1091]      0
2020-11-22 14:04:24.797768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1104] 0:   N
2020-11-22 14:04:24.797824: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.798014: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:24.798182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1230] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1545 MB memory) -> physical GPU (device: 0, name: Quadro P600, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-11-22T14:04:25.022Z        INFO    tensorflow/tensorflow.go:259    Detection Complete      {"package": "detector.tensorflow", "name": "tensorflow", "id": "", "duration": 0.215041021, "detections": 2}
2020-11-22T14:04:25.023Z        INFO    server/server.go:138    HTTP Request    {"status": 200, "took": 0.236342628, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "34f6cb495836/9zLnpxQJf5-001881", "remote": "172.200.0.1:51378"}
2020-11-22 14:04:25.053201: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:25.053479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro P600 computeCapability: 6.1
coreClock: 1.5565GHz coreCount: 3 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 59.75GiB/s
2020-11-22 14:04:25.053499: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-11-22 14:04:25.053508: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-11-22 14:04:25.053517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-11-22 14:04:25.053524: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-11-22 14:04:25.053531: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-11-22 14:04:25.053537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-11-22 14:04:25.053544: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-22 14:04:25.053587: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:25.053795: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:25.053973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
2020-11-22 14:04:25.053993: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1085] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-22 14:04:25.054001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1091]      0
2020-11-22 14:04:25.054008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1104] 0:   N
2020-11-22 14:04:25.054072: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:25.054277: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-22 14:04:25.054457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1230] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1545 MB memory) -> physical GPU (device: 0, name: Quadro P600, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-11-22T14:04:25.248Z        INFO    tensorflow/tensorflow.go:259    Detection Complete      {"package": "detector.tensorflow", "name": "tensorflow", "id": "", "duration": 0.183255925, "detections": 1}
2020-11-22T14:04:25.248Z        INFO    server/server.go:138    HTTP Request    {"status": 200, "took": 0.208017169, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "34f6cb495836/9zLnpxQJf5-001882", "remote": "172.200.0.1:51382"}
snowzach commented 3 years ago

Yeah, it's working. Pretty sure that means device 0.

scottgrobinson commented 3 years ago

Thanks @snowzach, much appreciated