PaddlePaddle / Paddle

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice (『飞桨』核心框架,深度学习&机器学习高性能单机、分布式训练和跨平台部署)
http://www.paddlepaddle.org/
Apache License 2.0
22.19k stars 5.57k forks source link

reshape跑一段时间之后出现错误 #21689

Closed miraclebiu closed 1 year ago

miraclebiu commented 4 years ago

为使您的问题得到快速解决,在建立Issues前,请您先通过如下方式搜索是否有相似问题:【搜索issue关键字】【使用labels筛选】【官方文档】

如果您没有查询到相似问题,为快速解决您的提问,建立issue时请提供如下细节信息:

Thank you for contributing to PaddlePaddle. Before submitting the issue, you could search issue in the github in case that there was a similar issue submitted or resolved before. If there is no solution,please make sure that this is a training issue including the following details: System information -PaddlePaddle version (eg.1.1)or CommitID -CPU: including CPUMKL/OpenBlas/MKLDNN version -GPU: including CUDA/CUDNN version -OS Platform (eg.Mac OS 10.14) -Other imformation: Distriuted training/informantion of operator/ Graphics card storage To Reproduce Steps to reproduce the behavior Describe your current behavior Code to reproduce the issue Other info / logs

UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "train_triplet.py", line 242, in main(args) File "train_triplet.py", line 202, in main losses = exe.run(compile_program, fetch_list=[v.name for v in train_fetch_vars]) File "/home/vis/jiangminyue/dependency_cuda9/dependency/minyue/lib/python2.7/site-packages/paddle/fluid/executor.py", line 775, in run six.reraise(*sys.exc_info()) File "/home/vis/jiangminyue/dependency_cuda9/dependency/minyue/lib/python2.7/site-packages/paddle/fluid/executor.py", line 770, in run use_program_cache=use_program_cache) File "/home/vis/jiangminyue/dependency_cuda9/dependency/minyue/lib/python2.7/site-packages/paddle/fluid/executor.py", line 829, in _run_impl return_numpy=return_numpy) File "/home/vis/jiangminyue/dependency_cuda9/dependency/minyue/lib/python2.7/site-packages/paddle/fluid/executor.py", line 669, in _run_parallel tensors = exe.run(fetch_var_names)._move_to_list() paddle.fluid.core_avx.EnforceNotMet:


C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const, int) 2 paddle::operators::ReshapeOp::ValidateShape(std::vector<int, std::allocator >, paddle::framework::DDim const&) 3 paddle::operators::ReshapeOp::InferShape(paddle::framework::InferShapeContext) const 4 paddle::operators::Reshape2Op::InferShape(paddle::framework::InferShapeContext) const 5 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void> const&, paddle::framework::RuntimeContext*) const 6 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void> const&) const 7 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_> const&) 8 paddle::framework::details::ComputationOpHandle::RunImpl() 9 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase) 10 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase, std::shared_ptr<paddle::framework::BlockingQueue > const&, unsigned long*) 11 std::_Function_handler<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> (), std::future_base::_Task_setter<std::unique_ptr<std::future_base::_Result, std::future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) 12 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) 13 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const


Python Call Stacks (More useful to users):

File "/home/vis/jiangminyue/dependency_cuda9/dependency/minyue/lib/python2.7/site-packages/paddle/fluid/framework.py", line 2459, in append_op attrs=kwargs.get("attrs", None)) File "/home/vis/jiangminyue/dependency_cuda9/dependency/minyue/lib/python2.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/home/vis/jiangminyue/dependency_cuda9/dependency/minyue/lib/python2.7/site-packages/paddle/fluid/layers/nn.py", line 8991, in reshape "XShape": x_shape}) File "/home/vis/jiangminyue/vreid/vehicle_reid_paddle/loss/triplet_loss.py", line 92, in hard_example_mining an_candi = fluid.layers.reshape(an_candi, [N, batch_size - num_instances]) File "/home/vis/jiangminyue/vreid/vehicle_reid_paddle/loss/triplet_loss.py", line 103, in tripletLoss dist_ap, dist_an = hard_example_mining(dist_mat, labels, batch_size, num_instances) File "train_triplet.py", line 64, in build_train_program triplet_cost = tripletLoss(fea_out, label, batch_size = cfg.batch_size, margin = cfg.margin, num_instances = cfg.num_instances) File "train_triplet.py", line 148, in main train_reader, avg_cost, cls_cost, triplet_cost, infer_node, lr_node = build_train_program(main_prog=train_prog, startup_prog=startup_prog, cfg=cfg) File "train_triplet.py", line 242, in main(args)


Error Message Summary:

PaddleCheckError: Expected capacity == in_size, but received capacity:3840 != in_size:3808. ShapeError: The 'shape' in ReshapeOp is invalid. The input tensor X'size must be equal to the capacity of 'shape'. But received X's shape = [3808, 1], X's size = 3808, 'shape' is [64, 60], the capacity of 'shape' is 3840. at [/paddle/paddle/fluid/operators/reshape_op.cc:197] [operator < reshape2 > error] W1211 11:07:32.929422 25223 init.cc:205] Aborted at 1576033652 (unix time) try "date -d @1576033652" if you are using GNU date W1211 11:07:32.929422 25236 init.cc:205] Aborted at 1576033652 (unix time) try "date -d @1576033652" if you are using GNU date W1211 11:07:32.929440 25225 init.cc:205] Aborted at 1576033652 (unix time) try "date -d @1576033652" if you are using GNU date W1211 11:07:32.929440 25227 init.cc:205] Aborted at 1576033652 (unix time) try "date -d @1576033652" if you are using GNU date W1211 11:07:32.929821 25220 init.cc:205] Aborted at 1576033652 (unix time) try "date -d @1576033652" if you are using GNU date W1211 11:07:32.931998 25227 init.cc:205] PC: @ 0x0 (unknown) W1211 11:07:32.932000 25225 init.cc:205] PC: @ 0x0 (unknown) W1211 11:07:32.932096 25223 init.cc:205] PC: @ 0x0 (unknown) W1211 11:07:32.932119 25220 init.cc:205] PC: @ 0x0 (unknown) W1211 11:07:32.932359 25227 init.cc:205] SIGTERM (@0x1f600005ecf) received by PID 25227 (TID 0x7f7871ab3700) from PID 24271; stack trace: W1211 11:07:32.932359 25225 init.cc:205] SIGTERM (@0x1f600005ecf) received by PID 25225 (TID 0x7f7871ab3700) from PID 24271; stack trace: W1211 11:07:32.932363 25220 init.cc:205] SIGTERM (@0x1f600005ecf) received by PID 25220 (TID 0x7f7871ab3700) from PID 24271; stack trace: W1211 11:07:32.932396 25223 init.cc:205] SIGTERM (@0x1f600005ecf) received by PID 25223 (TID 0x7f7871ab3700) from PID 24271; stack trace: W1211 11:07:32.932660 25227 init.cc:205] @ 0x38c040f130 (unknown) W1211 11:07:32.932672 25220 init.cc:205] @ 0x38c040f130 (unknown) W1211 11:07:32.932677 25225 init.cc:205] @ 0x38c040f130 (unknown) W1211 11:07:32.932711 25223 init.cc:205] @ 0x38c040f130 (unknown) W1211 11:07:32.932921 25227 init.cc:205] @ 0x38c040d7c0 (unknown) W1211 11:07:32.932934 25220 init.cc:205] @ 0x38c040d7c0 (unknown) W1211 11:07:32.932960 25225 init.cc:205] @ 0x38c040d7c0 (unknown) W1211 11:07:32.932978 25223 init.cc:205] @ 0x38c040d7c0 (unknown) W1211 11:07:32.934267 25220 init.cc:205] @ 0x7f78cffa6171 semlock_acquire W1211 11:07:32.934281 25225 init.cc:205] @ 0x7f78cffa6171 semlock_acquire W1211 11:07:32.934309 25227 init.cc:205] @ 0x7f78cffa6171 semlock_acquire W1211 11:07:32.934602 25223 init.cc:205] @ 0x7f78cffa6171 semlock_acquire W1211 11:07:32.936426 25220 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:32.936427 25225 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:32.936476 25227 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:32.936918 25223 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:32.938521 25225 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.938588 25227 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.938627 25220 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.939172 25223 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.940626 25225 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.940693 25227 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.940717 25220 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.941454 25223 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.942700 25225 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.942689 25228 init.cc:205] Aborted at 1576033652 (unix time) try "date -d @1576033652" if you are using GNU date W1211 11:07:32.942770 25220 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.943003 25237 init.cc:205] Aborted at 1576033652 (unix time) try "date -d @1576033652" if you are using GNU date W1211 11:07:32.943826 25223 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.944571 25227 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.944591 25226 init.cc:205] Aborted at 1576033652 (unix time) try "date -d @1576033652" if you are using GNU date W1211 11:07:32.944789 25220 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:32.945468 25237 init.cc:205] PC: @ 0x0 (unknown) W1211 11:07:32.945758 25237 init.cc:205] SIGTERM (@0x1f600005ecf) received by PID 25237 (TID 0x7f7871ab3700) from PID 24271; stack trace: W1211 11:07:32.946041 25223 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:32.946051 25237 init.cc:205] @ 0x38c040f130 (unknown) W1211 11:07:32.946147 25228 init.cc:205] PC: @ 0x0 (unknown) W1211 11:07:32.946388 25228 init.cc:205] SIGTERM (@0x1f600005ecf) received by PID 25228 (TID 0x7f7871ab3700) from PID 24271; stack trace: W1211 11:07:32.946672 25228 init.cc:205] @ 0x38c040f130 (unknown) W1211 11:07:32.946877 25220 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.946918 25228 init.cc:205] @ 0x38bfc8a039 (unknown) W1211 11:07:32.946923 25226 init.cc:205] PC: @ 0x0 (unknown) W1211 11:07:32.947144 25226 init.cc:205] SIGTERM (@0x1f600005ecf) received by PID 25226 (TID 0x7f7871ab3700) from PID 24271; stack trace: W1211 11:07:32.947150 25228 init.cc:205] @ 0x38bfc8105a (unknown) W1211 11:07:32.947443 25226 init.cc:205] @ 0x38c040f130 (unknown) W1211 11:07:32.947728 25226 init.cc:205] @ 0x38c040d7c0 (unknown) W1211 11:07:32.948405 25223 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.948606 25225 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:32.948699 25228 init.cc:205] @ 0x7f78d101f061 memory_get_block W1211 11:07:32.949046 25220 init.cc:205] @ 0x7f78f8e27797 PyEval_EvalFrameEx W1211 11:07:32.949110 25226 init.cc:205] @ 0x7f78cffa6171 semlock_acquire W1211 11:07:32.950273 25228 init.cc:205] @ 0x7f78d101f241 ImagingAllocateArray W1211 11:07:32.950994 25223 init.cc:205] @ 0x7f78f8e27797 PyEval_EvalFrameEx W1211 11:07:32.951056 25227 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:32.951176 25220 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.951313 25226 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:32.951797 25228 init.cc:205] @ 0x7f78d101f41b ImagingNewInternal W1211 11:07:32.950994 25225 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.953323 25220 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.953366 25228 init.cc:205] @ 0x7f78d0fdff8b (unknown) W1211 11:07:32.953508 25226 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.953516 25227 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.953670 25223 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.954275 25225 init.cc:205] @ 0x7f78f8e27797 PyEval_EvalFrameEx W1211 11:07:32.955425 25220 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.955713 25226 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.955742 25228 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:32.955943 25227 init.cc:205] @ 0x7f78f8e27797 PyEval_EvalFrameEx W1211 11:07:32.956269 25223 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.956851 25225 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.957454 25220 init.cc:205] @ 0x7f78f8da63a1 function_call W1211 11:07:32.957975 25226 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.958039 25228 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.958256 25227 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.958731 25223 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.959127 25225 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.959574 25220 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.960085 25226 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:32.960319 25228 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.960697 25227 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.961166 25223 init.cc:205] @ 0x7f78f8da63a1 function_call W1211 11:07:32.961613 25220 init.cc:205] @ 0x7f78f8d894bf instancemethod_call W1211 11:07:32.961722 25225 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.962273 25226 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.962612 25228 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.963112 25227 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.963634 25223 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.963734 25220 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.963914 25225 init.cc:205] @ 0x7f78f8da63a1 function_call W1211 11:07:32.964480 25226 init.cc:205] @ 0x7f78f8e27797 PyEval_EvalFrameEx W1211 11:07:32.964936 25228 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.965390 25227 init.cc:205] @ 0x7f78f8da63a1 function_call W1211 11:07:32.965798 25220 init.cc:205] @ 0x7f78f8de3320 slot_tp_init W1211 11:07:32.966028 25223 init.cc:205] @ 0x7f78f8d894bf instancemethod_call W1211 11:07:32.966161 25225 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.966702 25226 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.967275 25228 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.967689 25227 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.967859 25220 init.cc:205] @ 0x7f78f8dd9dd8 type_call W1211 11:07:32.968475 25223 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.968660 25225 init.cc:205] @ 0x7f78f8d894bf instancemethod_call W1211 11:07:32.968904 25226 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.969557 25228 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.969952 25227 init.cc:205] @ 0x7f78f8d894bf instancemethod_call W1211 11:07:32.970005 25220 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.970929 25223 init.cc:205] @ 0x7f78f8de3320 slot_tp_init W1211 11:07:32.970968 25225 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.971088 25226 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.971751 25228 init.cc:205] @ 0x7f78f8da63a1 function_call W1211 11:07:32.972146 25220 init.cc:205] @ 0x7f78f8e27f54 PyEval_EvalFrameEx W1211 11:07:32.972327 25227 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.973147 25225 init.cc:205] @ 0x7f78f8de3320 slot_tp_init W1211 11:07:32.973209 25226 init.cc:205] @ 0x7f78f8da63a1 function_call W1211 11:07:32.973340 25223 init.cc:205] @ 0x7f78f8dd9dd8 type_call W1211 11:07:32.974009 25236 init.cc:205] PC: @ 0x0 (unknown) W1211 11:07:32.974059 25228 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.974306 25236 init.cc:205] SIGTERM (@0x1f600005ecf) received by PID 25236 (TID 0x7f7871ab3700) from PID 24271; stack trace: W1211 11:07:32.974362 25220 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.974602 25227 init.cc:205] @ 0x7f78f8de3320 slot_tp_init W1211 11:07:32.974622 25236 init.cc:205] @ 0x38c040f130 (unknown) W1211 11:07:32.974902 25236 init.cc:205] @ 0x38c040d7c0 (unknown) W1211 11:07:32.975457 25226 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.975611 25225 init.cc:205] @ 0x7f78f8dd9dd8 type_call W1211 11:07:32.975793 25223 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.976382 25228 init.cc:205] @ 0x7f78f8d894bf instancemethod_call W1211 11:07:32.976539 25220 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.976891 25227 init.cc:205] @ 0x7f78f8dd9dd8 type_call W1211 11:07:32.977119 25236 init.cc:205] @ 0x7f78f8e5d008 PyThread_acquire_lock W1211 11:07:32.977643 25226 init.cc:205] @ 0x7f78f8d894bf instancemethod_call W1211 11:07:32.977807 25225 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.978260 25223 init.cc:205] @ 0x7f78f8e27f54 PyEval_EvalFrameEx W1211 11:07:32.978718 25228 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.978730 25220 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.979164 25227 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.979308 25236 init.cc:205] @ 0x7f78f8e21284 PyEval_RestoreThread W1211 11:07:32.979919 25226 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.980091 25225 init.cc:205] @ 0x7f78f8e27f54 PyEval_EvalFrameEx W1211 11:07:32.980682 25223 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.980724 25236 init.cc:205] @ 0x7f78cffa60c2 semlock_acquire W1211 11:07:32.980814 25220 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.980979 25228 init.cc:205] @ 0x7f78f8de02d5 call_method W1211 11:07:32.981514 25227 init.cc:205] @ 0x7f78f8e27f54 PyEval_EvalFrameEx W1211 11:07:32.982128 25226 init.cc:205] @ 0x7f78f8de3320 slot_tp_init W1211 11:07:32.982322 25225 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.982957 25236 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:32.983060 25220 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.983094 25223 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.983330 25228 init.cc:205] @ 0x7f78f8e22857 PyEval_EvalFrameEx W1211 11:07:32.983837 25227 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.984298 25226 init.cc:205] @ 0x7f78f8dd9dd8 type_call W1211 11:07:32.985110 25236 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.985136 25220 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.985481 25223 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.985656 25228 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.986111 25227 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.986502 25225 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.986565 25226 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.987347 25220 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.987351 25236 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.988057 25223 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.988062 25228 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.988412 25227 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.988770 25225 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.988903 25226 init.cc:205] @ 0x7f78f8e27f54 PyEval_EvalFrameEx W1211 11:07:32.989456 25220 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.989590 25236 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.990346 25228 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.990487 25223 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.990706 25227 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.990967 25225 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.991204 25226 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.991672 25220 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.991708 25236 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:32.992635 25228 init.cc:205] @ 0x7f78f8e1c55f builtin_next W1211 11:07:32.992797 25223 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.993077 25227 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.993172 25225 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.993465 25226 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.993855 25220 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.993901 25236 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:32.995048 25228 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:32.995266 25227 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.995280 25225 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.995281 25223 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.995718 25226 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:32.995957 25220 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:32.996110 25236 init.cc:205] @ 0x7f78f8e27797 PyEval_EvalFrameEx W1211 11:07:32.997436 25228 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:32.997473 25225 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.997510 25227 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:32.997546 25223 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.997869 25226 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.998260 25236 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:32.999578 25225 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.999711 25227 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:32.999725 25228 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:32.999925 25223 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.000190 25226 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.000404 25236 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:33.001760 25225 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.001946 25227 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.002101 25228 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:33.002223 25223 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.002466 25226 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:33.002571 25236 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.003877 25225 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.004158 25227 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.004501 25223 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:33.004570 25228 init.cc:205] @ 0x7f78f8e27797 PyEval_EvalFrameEx W1211 11:07:33.004817 25226 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.004999 25237 init.cc:205] @ 0x7f78d0d9eaba (unknown) W1211 11:07:33.005887 25225 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:33.006258 25227 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:33.006585 25237 init.cc:205] @ 0x7f78d0da7d79 (unknown) W1211 11:07:33.007069 25226 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:33.007133 25228 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:33.008107 25237 init.cc:205] @ 0x7f78d0da3269 (unknown) W1211 11:07:33.009475 25228 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:33.009470 25226 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.009660 25237 init.cc:205] @ 0x7f78d0d9b74a jpeg_read_scanlines W1211 11:07:33.011204 25237 init.cc:205] @ 0x7f78d100c880 ImagingJpegDecode W1211 11:07:33.011791 25228 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.011790 25226 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.012748 25237 init.cc:205] @ 0x7f78d0fe22c6 (unknown) W1211 11:07:33.013851 25226 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:33.014957 25237 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:33.016613 25236 init.cc:205] @ 0x7f78f8da63a1 function_call W1211 11:07:33.018600 25236 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:33.020452 25236 init.cc:205] @ 0x7f78f8d894bf instancemethod_call W1211 11:07:33.022389 25236 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:33.024272 25236 init.cc:205] @ 0x7f78f8de3320 slot_tp_init W1211 11:07:33.026142 25236 init.cc:205] @ 0x7f78f8dd9dd8 type_call W1211 11:07:33.028079 25236 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:33.030179 25237 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:33.032383 25237 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.034524 25237 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:33.036680 25237 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:33.038807 25237 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.040860 25237 init.cc:205] @ 0x7f78f8da63a1 function_call W1211 11:07:33.043098 25236 init.cc:205] @ 0x7f78f8e27f54 PyEval_EvalFrameEx W1211 11:07:33.045086 25236 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:33.047021 25236 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.048969 25236 init.cc:205] @ 0x7f78f8e29a55 PyEval_EvalFrameEx W1211 11:07:33.050848 25236 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:33.053654 25237 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:33.055791 25237 init.cc:205] @ 0x7f78f8d894bf instancemethod_call W1211 11:07:33.057929 25237 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:33.060022 25237 init.cc:205] @ 0x7f78f8de02d5 call_method W1211 11:07:33.062165 25237 init.cc:205] @ 0x7f78f8e22857 PyEval_EvalFrameEx W1211 11:07:33.064215 25237 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:33.065227 25236 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.067103 25236 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:33.069039 25236 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.070889 25236 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:33.073006 25236 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.075203 25237 init.cc:205] @ 0x7f78f8e2434f PyEval_EvalFrameEx W1211 11:07:33.077298 25237 init.cc:205] @ 0x7f78f8d9acec gen_send_ex W1211 11:07:33.079381 25237 init.cc:205] @ 0x7f78f8e1c55f builtin_next W1211 11:07:33.081524 25237 init.cc:205] @ 0x7f78f8e29e15 PyEval_EvalFrameEx W1211 11:07:33.083667 25237 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.085717 25237 init.cc:205] @ 0x7f78f8da64a8 function_call W1211 11:07:33.087852 25237 init.cc:205] @ 0x7f78f8d76d23 PyObject_Call W1211 11:07:33.090006 25237 init.cc:205] @ 0x7f78f8e27797 PyEval_EvalFrameEx W1211 11:07:33.092156 25237 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:33.094316 25237 init.cc:205] @ 0x7f78f8e29b95 PyEval_EvalFrameEx W1211 11:07:33.096457 25237 init.cc:205] @ 0x7f78f8e2aa2e PyEval_EvalCodeEx W1211 11:07:33.098522 25237 init.cc:205] @ 0x7f78f8da63a1 function_call

miraclebiu commented 4 years ago

paddle 1.6.1 k40,3卡,py-reader方式

miraclebiu commented 4 years ago

image

liym27 commented 4 years ago

您好,报错信息中显示:

PaddleCheckError: Expected capacity == in_size, but received capacity:3840 != in_size:3808. ShapeError: The 'shape' in ReshapeOp is invalid. The input tensor X'size must be equal to the capacity of 'shape'. But received X's shape = [3808, 1], X's size = 3808, 'shape' is [64, 60], the capacity of 'shape' is 3840. at [/paddle/paddle/fluid/operators/reshape_op.cc:197] [operator < reshape2 > error]

即使用reshape接口的时候,输入X的形状与参数’shape‘指定的形状不匹配。请确认是否是该问题。

miraclebiu commented 4 years ago

不是该问题导致的,print出来的label是正确的,但是reshape之后会多出来几个数据。请问这个bug什么时候会修复? @liym27