Closed MrRace closed 4 years ago
import paddle.fluid
paddle.fluid.install_check.run_check()
Running Verify Fluid Program ... W0520 16:52:06.161931 27216 device_context.cc:261] Please NOTE: device: 0, CUDA Capability: 75, Driver API Version: 10.0, Runtime API Version: 9.0 W0520 16:52:06.187935 27216 device_context.cc:269] device: 0, cuDNN Version: 7.0. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/install_check.py", line 65, in run_check fetch_list=[out.name, param_grads[1].name]) File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/executor.py", line 565, in run use_program_cache=use_program_cache) File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/executor.py", line 642, in _run exe.run(program.desc, scope, 0, True, True, fetch_var_name) paddle.fluid.core.EnforceNotMet: Invoke operator mul error. Python Callstacks: File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/framework.py", line 1654, in append_op attrs=kwargs.get("attrs", None)) File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/dygraph/layer_object_helper.py", line 52, in append_op stop_gradient=stop_gradient) File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/dygraph/nn.py", line 919, in forward "y_num_col_dims": 1 File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/dygraph/layers.py", line 158, in __call__ outputs = self.forward(*inputs) File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/install_check.py", line 37, in forward x = self._fc1(inputs) File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/dygraph/layers.py", line 158, in __call__ outputs = self.forward(*inputs) File "/home/lab/liujiepeng/MachineComprehension/DuReader/paddle_py35_even/lib/python3.5/site-packages/paddle/fluid/install_check.py", line 58, in run_check out = simple_layer(inp) File "<stdin>", line 1, in <module> C++ Callstacks: CUBLAS: execution failed, at [/paddle/paddle/fluid/operators/math/blas_impl.cu.h:34] PaddlePaddle Call Stacks: 0 0x7fb372fe4c90p void paddle::platform::EnforceNotMet::Init<char const*>(char const*, char const*, int) + 352 1 0x7fb372fe5009p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 137 2 0x7fb373250097p void paddle::operators::math::Blas<paddle::platform::CUDADeviceContext>::GEMM<float>(CBLAS_TRANSPOSE, CBLAS_TRANSPOSE, int, int, int, float, float const*, float const*, float, float*) const + 1815 3 0x7fb3736caa6cp void paddle::operators::math::Blas<paddle::platform::CUDADeviceContext>::MatMul<float>(paddle::framework::Tensor const&, bool, paddle::framework::Tensor const&, bool, float, paddle::framework::Tensor*, float) const + 844 4 0x7fb3736caea6p paddle::operators::MulKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const + 662 5 0x7fb3736cb093p std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::MulKernel<paddle::platform::CUDADeviceContext, float>, paddle::operators::MulKernel<paddle::platform::CUDADeviceContext, double>, paddle::operators::MulKernel<paddle::platform::CUDADeviceContext, paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) + 35 6 0x7fb374c74326p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 662 7 0x7fb374c74a94p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 292 8 0x7fb374c723bcp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332 9 0x7fb3731573aep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 382 10 0x7fb3731581efp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector<std::string, std::allocator<std::string> > const&, bool) + 143 11 0x7fb372fd489ep 12 0x7fb37301777ep 13 0x55c1828a34e8p PyCFunction_Call + 120 14 0x55c1828ffec4p PyEval_EvalFrameEx + 23284 15 0x55c1828ff490p PyEval_EvalFrameEx + 20672 16 0x55c1828ff490p PyEval_EvalFrameEx + 20672 17 0x55c1828fae90p PyEval_EvalFrameEx + 2752 18 0x55c18290503dp PyEval_EvalCodeEx + 525 19 0x55c182905f5cp PyEval_EvalCode + 28 20 0x55c182962454p 21 0x55c18282eb5ep PyRun_InteractiveOneObject + 709 22 0x55c18282ed01p PyRun_InteractiveLoopFlags + 239 23 0x55c18282ed62p 24 0x55c18282f080p 25 0x55c18282f871p main + 225 26 0x7fb3b31d2830p __libc_start_main + 240 27 0x55c182907808p
PS:本机TensorFlow-gpu和PyTorch-gpu都可以正常运行。
related issue https://github.com/PaddlePaddle/Paddle/issues/17217
可是在#17217中并没有提到解决方法呀 @chengduoZH
import paddle.fluid
,再输入paddle.fluid.install_check.run_check()
报错如下:PS:本机TensorFlow-gpu和PyTorch-gpu都可以正常运行。