search

faustomorales / keras-ocr

A packaged and flexible version of the CRAFT text detector and Keras CRNN recognition model.
https://keras-ocr.readthedocs.io/
MIT License
1.39k stars 362 forks source link

Allocation of 989331456 exceeds 10% of free system memory. #81

Closed jackCplusplus closed 4 years ago

jackCplusplus commented 4 years ago

my memory is 8 GB (DDR4 2400MHz),however it's still not enough.how can i deal with this problem? 

C:\Users\jack\Desktop>python lal.python
2020-05-22 21:32:45.116443: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-05-22 21:32:45.123360: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Looking for C:\Users\jack\.keras-ocr\craft_mlt_25k.h5
2020-05-22 21:32:49.432400: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-05-22 21:32:49.655481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:08:00.0 name: GeForce GTX 1070 Ti computeCapability: 6.1
coreClock: 1.683GHz coreCount: 19 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2020-05-22 21:32:49.669996: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-05-22 21:32:49.679463: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_10.dll'; dlerror: cublas64_10.dll not found
2020-05-22 21:32:49.689185: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2020-05-22 21:32:49.698681: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2020-05-22 21:32:49.708758: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2020-05-22 21:32:49.720601: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusparse64_10.dll'; dlerror: cusparse64_10.dll not found
2020-05-22 21:32:49.864803: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-22 21:32:49.869587: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-05-22 21:32:49.891023: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-05-22 21:32:49.910393: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23463246ba0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-22 21:32:49.919019: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-22 21:32:49.925290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-22 21:32:49.933877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
WARNING:tensorflow:From C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\keras\backend.py:5871: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
Looking for C:\Users\jack\.keras-ocr\crnn_kurapan.h5
2020-05-22 21:32:52.801406: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 989331456 exceeds 10% of free system memory.
2020-05-22 21:32:53.587524: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 989331456 exceeds 10% of free system memory.
2020-05-22 21:32:55.642606: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 494403584 exceeds 10% of free system memory.
2020-05-22 21:32:56.336303: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 494403584 exceeds 10% of free system memory.
2020-05-22 21:33:05.276918: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 741605376 exceeds 10% of free system memory.
jackCplusplus commented 4 years ago

plus, thers has CUDA9.2 , Need i update it to CUDA10.1?

faustomorales commented 4 years ago

Hi there!

If you're using TensorFlow 2.x, please note that CUDA 9 is not supported. So if you wish to use the GPU, you will need to upgrade CUDA or downgrade TensorFlow (I recommend the former).

That said, the logs you're pointing to here are warnings indicating that you're using a large amount of system memory but are normal. Once you get your GPU configuration up and running, I expect they'll go away.

Hope this helps!

faustomorales commented 4 years ago

Closing as this does not appear to point to a bug or feature request for keras-ocr.

jackCplusplus commented 4 years ago

Hi, thanks! when i update CUDA9.2 to CUDA10.1 and cudnn, the version of cudnn loaded is 7.5.0,however there has cudnn v7.1.4 for CUDA9.2 and has cudnn v7.6.5 for CUDA10.1.

2020-05-23 22:23:09.758126: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Looking for C:\Users\有对象真好\.keras-ocr\craft_mlt_25k.h5
2020-05-23 22:23:14.446656: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-05-23 22:23:14.470724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: GeForce GTX 1070 Ti computeCapability: 6.1
coreClock: 1.683GHz coreCount: 19 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2020-05-23 22:23:14.475854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-05-23 22:23:14.501773: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-23 22:23:14.515525: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-05-23 22:23:14.527552: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-05-23 22:23:14.539852: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-05-23 22:23:14.555317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-05-23 22:23:14.584781: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-23 22:23:14.588281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-23 22:23:14.591930: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-05-23 22:23:14.607133: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x24171b62870 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-23 22:23:14.610707: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-23 22:23:14.619303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: GeForce GTX 1070 Ti computeCapability: 6.1
coreClock: 1.683GHz coreCount: 19 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2020-05-23 22:23:14.632733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-05-23 22:23:14.637275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-05-23 22:23:14.643415: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-05-23 22:23:14.648087: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-05-23 22:23:14.654401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-05-23 22:23:14.658880: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-05-23 22:23:14.665943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-23 22:23:14.671485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-23 22:23:15.173514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-23 22:23:15.176237: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0
2020-05-23 22:23:15.180192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N
2020-05-23 22:23:15.186064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6285 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:09:00.0, compute capability: 6.1)
2020-05-23 22:23:15.200484: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2411a2f3d10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-23 22:23:15.206422: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1070 Ti, Compute Capability 6.1
2020-05-23 22:23:17.259729: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-05-23 22:23:18.011912: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.5.0 but source was compiled with: 7.6.5.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2020-05-23 22:23:18.032912: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.5.0 but source was compiled with: 7.6.5.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
  File "lal.python", line 8, in <module>
    boxes = detector.detect(images=[image])[0]
  File "C:\ProgramData\Anaconda3\lib\site-packages\keras_ocr\detection.py", line 679, in detect
    boxes = getBoxes(self.model.predict(np.array(images), **kwargs),
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 88, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1268, in predict
    tmp_batch_outputs = predict_function(iterator)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 650, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1665, in _filtered_call
    self.captured_inputs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1746, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 598, in call
    ctx=ctx)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[node model_1/basenet.slice1.0/Conv2D (defined at C:\ProgramData\Anaconda3\lib\site-packages\keras_ocr\detection.py:679) ]] [Op:__inference_predict_function_3683]

Function call stack:
predict_function
NeighborhoodCoding commented 4 years ago

It is actually a memory error(your python may stopped) and you need to increase virtual memory about 10~50GB