Closed WANGDow closed 4 years ago
More supported info produced using the origin client.py of deepspeech:
Issue One:
root@cd69049a871f:/DeepSpeech/asr_test_server# deepspeech --model new_lm_19_train/output_graph.pb --scorer new_lm_19_train/kenlm.scorer --audio empty_audio.wav
2020-11-13 07:30:41.801501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loading model from file new_lm_19_train/output_graph.pb
TensorFlow: v1.15.0-29-g4e0e823493
DeepSpeech: v0.7.4-0-gfcd9563f
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-11-13 07:30:41.912246: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-13 07:30:41.913273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-13 07:30:41.945059: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:41.945704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.605
pciBusID: 0000:26:00.0
2020-11-13 07:30:41.945718: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-13 07:30:41.946654: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-13 07:30:41.947472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-13 07:30:41.947661: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-13 07:30:41.948748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-13 07:30:41.949540: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-13 07:30:41.952087: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-13 07:30:41.952187: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:41.952822: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:41.953342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-13 07:30:42.342786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 07:30:42.342813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-11-13 07:30:42.342818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-11-13 07:30:42.342932: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:42.343459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:42.343972: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:30:42.344458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9767 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:26:00.0, compute capability: 7.5)
Loaded model in 0.591s.
Loading scorer from files new_lm_19_train/kenlm.scorer
Loaded scorer in 0.000225s.
Running inference.
2020-11-13 07:30:43.248752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Segmentation fault (core dumped)
root@cd69049a871f:/DeepSpeech/asr_test_server#
Issue Two: Success Case:
root@cd69049a871f:/DeepSpeech/asr_test_server# deepspeech --model new_lm_19_train/output_graph.pb --scorer new_lm_19_train/kenlm.scorer --audio audio_collect_zh-HK_19be47fa770a4a12826735b395f7e8ad.wav
2020-11-13 07:38:53.148996: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loading model from file new_lm_19_train/output_graph.pb
TensorFlow: v1.15.0-29-g4e0e823493
DeepSpeech: v0.7.4-0-gfcd9563f
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-11-13 07:38:53.261789: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-13 07:38:53.262647: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-13 07:38:53.291063: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.291840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.605
pciBusID: 0000:26:00.0
2020-11-13 07:38:53.291854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-13 07:38:53.292750: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-13 07:38:53.293539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-13 07:38:53.293701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-13 07:38:53.294676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-13 07:38:53.295394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-13 07:38:53.297740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-13 07:38:53.297831: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.298394: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.298898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-13 07:38:53.703120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 07:38:53.703142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-11-13 07:38:53.703147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-11-13 07:38:53.703255: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.703766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.704260: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:38:53.704746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9790 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:26:00.0, compute capability: 7.5)
Loaded model in 0.598s.
Loading scorer from files new_lm_19_train/kenlm.scorer
Loaded scorer in 0.000179s.
Running inference.
2020-11-13 07:38:54.636989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
將軍澳居民可價車從跨灣大橋
Inference took 1.467s for 3.840s audio file.
root@cd69049a871f:/DeepSpeech/asr_test_server#
Using "Extended" mode:
root@cd69049a871f:/DeepSpeech/asr_test_server# deepspeech --model new_lm_19_train/output_graph.pb --scorer new_lm_19_train/kenlm.scorer --audio audio_collect_zh-HK_19be47fa770a4a12826735b395f7e8ad.wav --extended
2020-11-13 07:39:02.605366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loading model from file new_lm_19_train/output_graph.pb
TensorFlow: v1.15.0-29-g4e0e823493
DeepSpeech: v0.7.4-0-gfcd9563f
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-11-13 07:39:02.715122: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-13 07:39:02.716027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-13 07:39:02.744854: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:02.745402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.605
pciBusID: 0000:26:00.0
2020-11-13 07:39:02.745416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-13 07:39:02.746300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-13 07:39:02.747077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-13 07:39:02.747246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-13 07:39:02.748233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-13 07:39:02.748973: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-13 07:39:02.751328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-13 07:39:02.751420: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:02.752022: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:02.752536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-13 07:39:03.249879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 07:39:03.249904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-11-13 07:39:03.249909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-11-13 07:39:03.250025: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:03.250557: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:03.251069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:03.251559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9798 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:26:00.0, compute capability: 7.5)
Loaded model in 0.699s.
Loading scorer from files new_lm_19_train/kenlm.scorer
Loaded scorer in 0.000212s.
Running inference.
2020-11-13 07:39:04.193455: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Traceback (most recent call last):
File "/usr/local/bin/deepspeech", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/deepspeech/client.py", line 152, in main
print(metadata_to_string(ds.sttWithMetadata(audio, 1).transcripts[0]))
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-38: surrogates not allowed
root@cd69049a871f:/DeepSpeech/asr_test_server#
Using "Json" mode:
root@cd69049a871f:/DeepSpeech/asr_test_server# deepspeech --model new_lm_19_train/output_graph.pb --scorer new_lm_19_train/kenlm.scorer --audio audio_collect_zh-HK_19be47fa770a4a12826735b395f7e8ad.wav --json
2020-11-13 07:39:17.533590: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Loading model from file new_lm_19_train/output_graph.pb
TensorFlow: v1.15.0-29-g4e0e823493
DeepSpeech: v0.7.4-0-gfcd9563f
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-11-13 07:39:17.642701: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-13 07:39:17.643614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-13 07:39:17.672544: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:17.673301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.605
pciBusID: 0000:26:00.0
2020-11-13 07:39:17.673321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-13 07:39:17.674439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-13 07:39:17.675416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-13 07:39:17.675613: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-13 07:39:17.676623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-13 07:39:17.677360: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-13 07:39:17.679744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-13 07:39:17.679845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:17.680468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:17.681029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-13 07:39:18.073182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-13 07:39:18.073203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-11-13 07:39:18.073208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-11-13 07:39:18.073316: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:18.073853: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:18.074362: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-13 07:39:18.074851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9790 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:26:00.0, compute capability: 7.5)
Loaded model in 0.586s.
Loading scorer from files new_lm_19_train/kenlm.scorer
Loaded scorer in 0.000177s.
Running inference.
2020-11-13 07:39:18.927302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
{
"transcripts": [
{
"confidence": -251.61355590820312,
"words": [
{
"word": "\udce5\udcb0\udc87\udce8\udcbb\udc8d\udce6\udcbe\udcb3\udce5\udcb1\udc85\udce6\udcb0\udc91\udce5\udc8f\udcaf\udce5\udc83\udcb9\udce8\udcbb\udc8a\udce5\udcbe\udc9e\udce8\udcb7\udca8\udce7\udc81\udca3\udce5\udca4\udca7\udce6\udca9\udc8b",
"start_time ": 0.0,
"duration": 3.8
}
]
},
{
"confidence": -252.30313110351562,
"words": [
{
"word": "\udce5\udcb0\udc87\udce8\udcbb\udc8d\udce6\udcbe\udcb3\udce5\udcb1\udc85\udce6\udcb0\udc91\udce5\udc8f\udcaf\udce5\udc81\udc87\udce8\udcbb\udc8a\udce5\udcbe\udc9e\udce8\udcb7\udca8\udce7\udc81\udca3\udce5\udca4\udca7\udce6\udca9\udc8b",
"start_time ": 0.0,
"duration": 3.8
}
]
},
{
"confidence": -253.93341064453125,
"words": [
{
"word": "\udce5\udcb0\udc87\udce8\udcbb\udc8d\udce6\udcbe\udcb3\udce5\udcb1\udc85\udce6\udcb0\udc91\udce5\udc8f\udcaf\udce9\udc81\udc87\udce8\udcbb\udc8a\udce5\udcbe\udc9e\udce8\udcb7\udca8\udce7\udc81\udca3\udce5\udca4\udca7\udce6\udca9\udc8b",
"start_time ": 0.0,
"duration": 3.8
}
]
}
]
}
Inference took 1.365s for 3.840s audio file.
root@cd69049a871f:/DeepSpeech/asr_test_server#
Hope that these extra info would be helpful. Thanks again. Dow
Please read the first line when opening a new issue:
For support and discussions, please use our Discourse forums.
Please read the first line when opening a new issue:
For support and discussions, please use our Discourse forums.
Hi. Sorry if this post causes any inconvenience. I thought my issues were bugs or something. Thanks anyway. I will post again on your Discourse.
No worries, this is a legitimate problem, just post it on Discourse, there are some other Chinese users as well.
@lissyx Please close, discussion moved to Discourse.
To whom it may concern,
Hi. Thanks for your efforts first.
I am training a Cantonese Model with self-collected audio data(around 250 hours) and external decoder(around 700M uncompressed). Even though the overfitting is observed due to the data size, the curve of the training loss is quit beautiful.
However, while I am conducting some tests, two fatal issues pop up.
Here is my project setting:
Issue One: "Segmentation fault (core dumped)" is raised when an empty audio clip with nobody speaking is given to the ds.stt. It may happens as well when the model can't provide any prediction (no confirmed), since "core dumped" is also raised when I run the lm_optimizer.py with proper test audios.
In the above figure, a proper audio is used and a recognized result is provided (Shown in the red box). However, once an empty audio is fed, core dumped is given and the whole server crashes (Shown in the green box).
This is a fatal issue because the whole service is shut down. Thus, is there an approach to tackle the root cause or stop it from shutting the service?
Issue Two: Unproperly decode of the recognized result when using ds.sttWithMetaData. Shown in the following figure, the external decoder works properly with ds.stt. However, once I switch to the ds.WithMetaData, the decoding becomes a mess. Is it caused by that I didn't use the --utf8 for training?
The content in the blue box is produced by the following code.
Thanks in advanced and best regards. Dow