search

PaddlePaddle / PaddleSpeech

Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.
https://paddlespeech.readthedocs.io
Apache License 2.0
11.07k stars 1.85k forks source link

run PaddleSpeech VectorExecutor on xpu device fail #2290

Closed gyb4git closed 2 years ago

gyb4git commented 2 years ago

-Description： Environment： ARM, Kunlun R200 OS: Linux localhost.localdomain 4.19.152-801.s2500.el8_1.aarch64 #1 SMP Fri Jun 24 11:57:19 CST 2022 aarch64 aarch64 aarch64 GNU/Linux

python:3.7.4 paddle: compile and build on source with xpu

-Test code import paddle from paddlespeech.cli.vector import VectorExecutor from scipy.spatial.distance import cosine

vector_executor = VectorExecutor() audio_emb = vector_executor( model='ecapatdnn_voxceleb12', sample_rate=16000, config=None, # Set config and ckpt_path to None to use pretrained model. ckpt_path=None, audio_file='./85236145389.wav', device=paddle.get_device()) print('Audio embedding Result: \n{}'.format(audio_emb))

test_emb = vector_executor( model='ecapatdnn_voxceleb12', sample_rate=16000, config=None, # Set config and ckpt_path to None to use pretrained model. ckpt_path=None, audio_file='./123456789.wav', device=paddle.get_device()) print('Test embedding Result: \n{}'.format(test_emb))

score = vector_executor.get_embeddings_score(audio_emb, test_emb) print(f"Eembeddings Score: {score}")

score = 1 - cosine(audio_emb, test_emb) print(f"Eembeddings Score2: {score}")

-Error: [INFO][BKCL][/home/users/yangyu22/baidu/xpu/bkcl/src/globals.cpp:52] set BKCL timeout to 600 seconds [INFO][BKCL][/home/users/yangyu22/baidu/xpu/bkcl/src/globals.cpp:53] set BKCL RING BUFFER SIZE to 1048576 XPURT /usr/local/lib64/python3.7/site-packages/paddle/fluid/../libs/libxpurt.so loaded W0823 14:50:02.136422 61499 pybind.cc:2107] Cannot use get_all_custom_device_type because you have installedCPU/GPU version PaddlePaddle. If you want to use get_all_custom_device_type, please try to install CustomDevice version PaddlePaddle by: pip install paddlepaddle-core W0823 14:50:03.240737 61499 xpu_context.cc:89] Please NOTE: xpu device: 0 W0823 14:50:03.241063 61499 device_context.cc:310] Please NOTE: xpu device: 0 /usr/local/lib64/python3.7/site-packages/scipy/fftpack/init.py:103: DeprecationWarning: The module numpy.dual is deprecated. Instead of using dual, use the functions directly from numpy or scipy. from numpy.dual import register_func /usr/local/lib64/python3.7/site-packages/scipy/sparse/sputils.py:16: DeprecationWarning: np.typeDict is a deprecated alias for np.sctypeDict. supported_dtypes = [np.typeDict[x] for x in supporteddtypes] /usr/local/lib64/python3.7/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: np.int is a deprecated alias for the builtin int. To silence this warning, use int by itself. Doing this will not modify any behavior and is safe. When replacing np.int, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int, /usr/local/lib64/python3.7/site-packages/scipy/io/matlab/mio5.py:98: DeprecationWarning: np.bool is a deprecated alias for the builtin bool. To silence this warning, use bool by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool` here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from .mio5_utils import VarReader5 Traceback (most recent call last): File "test_sv_xpu.py", line 12, in device=paddle.get_device()) File "/usr/local/lib/python3.7/site-packages/paddlespeech/cli/utils.py", line 328, in _warpper return executor_func(self, *args, kwargs) File "/usr/local/lib/python3.7/site-packages/paddlespeech/cli/vector/infer.py", line 257, in call self.infer(model) File "", line 2, in infer File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 354, in _decorate_function return func(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/paddlespeech/cli/vector/infer.py", line 358, in infer embedding = self.model.backbone(feats, lengths).squeeze().numpy() File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(inputs, kwargs) File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, kwargs) File "/usr/local/lib/python3.7/site-packages/paddlespeech/vector/models/ecapa_tdnn.py", line 511, in forward x = self.mfa(x) File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, *kwargs) File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(inputs, kwargs) File "/usr/local/lib/python3.7/site-packages/paddlespeech/vector/models/ecapa_tdnn.py", line 208, in forward return self.norm(self.activation(self.conv(x))) File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, kwargs) File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "/usr/local/lib/python3.7/site-packages/paddlespeech/vector/models/ecapa_tdnn.py", line 177, in forward x_n = self.norm(x) File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(inputs, kwargs) File "/usr/local/lib64/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "/usr/local/lib64/python3.7/site-packages/paddle/nn/layer/norm.py", line 666, in forward use_global_stats=self._use_global_stats) File "/usr/local/lib64/python3.7/site-packages/paddle/nn/functional/norm.py", line 207, in batch_norm variance_out, attrs) OSError:

C++ Traceback (most recent call last):

0 paddle::imperative::Tracer::TraceOp(std::string const&, paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap, std::map<std::string, std::string, std::less, std::allocator<std::pair<std::string const, std::string > > > const&) 1 void paddle::imperative::Tracer::TraceOpImpl(std::string const&, paddle::imperative::details::NameVarMapTrait::Type const&, paddle::imperative::details::NameVarMapTrait::Type const&, paddle::framework::AttributeMap&, phi::Place const&, bool, std::map<std::string, std::string, std::less, std::allocator<std::pair<std::string const, std::string > > > const&, paddle::framework::AttributeMap, bool) 2 paddle::imperative::PreparedOp::Run(paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap const&, paddle::framework::AttributeMap const&) 3 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<phi::XPUPlace, false, 0ul, paddle::operators::BatchNormXPUKernel<paddle::platform::XPUDeviceContext, float> >::operator()(char const, char const, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) 4 paddle::operators::BatchNormXPUKernel<paddle::platform::XPUDeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const 5 phi::enforce::EnforceNotMet::EnforceNotMet(phi::ErrorSummary const&, char const, int) 6 phi::enforce::GetCurrentTraceBackStringabi:cxx11

Error Message Summary:

ExternalError: The batch_norm_infer XPU API return wrong value[1 xpu api invalid param] [Hint: Expected r == xpu::Error_t::SUCCESS, but received r:1 != xpu::Error_t::SUCCESS:0.] (at /workspace/qiyuan/Paddle/paddle/fluid/operators/batch_norm_op_xpu.cc:121) [operator < batch_norm > error]

-misc run normally with param device="cpu".

zh794390558 commented 2 years ago

xpu支持问题在paddle里反馈吧。这边看不了这种问题。