导出模型出错了

[thomascatlee]

下载了预训练的模型，导出时候报错。

[2022-11-08 09:54:46.507814 INFO   ] trainer:export:518 - 成功恢复模型参数和优化方法参数：models/conformer_online_fbank/best_model/model.pdparams
/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/jit.py:839: UserWarning: What you save is a function, and `jit.save` will generate the name of the model file according to `path` you specify. When loading these files with `jit.load`, you get a `TranslatedLayer` whose inference result is the same as the inference result of the function you saved.
  'What you save is a function, and `jit.save` will generate the name of the model file according to `path` you specify. When loading these files with `jit.load`, you get a `TranslatedLayer` whose inference result is the same as the inference result of the function you saved.'
Traceback (most recent call last):
  File "export_model.py", line 28, in <module>
    resume_model=args.resume_model.format(configs['use_model'], configs['preprocess_conf']['feature_method']))
  File "/data/PPASR/PPASR-develop/ppasr/trainer.py", line 527, in export
    paddle.jit.save(infer_model, infer_model_path)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 26, in __impl__
    return wrapped_func(*args, **kwargs)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/jit.py", line 649, in wrapper
    func(layer, path, input_spec, **configs)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 26, in __impl__
    return wrapped_func(*args, **kwargs)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 52, in __impl__
    return func(*args, **kwargs)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/jit.py", line 960, in save
    inner_input_spec)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 577, in concrete_program_specify_input_spec
    is_train=self._is_train_mode())
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 482, in get_concrete_program
    concrete_program, partial_program_layer = self._program_cache[cache_key]
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 952, in __getitem__
    self._caches[item_id] = self._build_once(item)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 941, in _build_once
    **cache_key.kwargs)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 26, in __impl__
    return wrapped_func(*args, **kwargs)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 52, in __impl__
    return func(*args, **kwargs)
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 892, in from_func_spec
    error_data.raise_new_exception()
  File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/error.py", line 350, in raise_new_exception
    six.exec_("raise new_exception from None")
  File "<string>", line 1, in <module>
ValueError: In transformed code:

    File "/data/PPASR/PPASR-develop/ppasr/model_utils/conformer/model.py", line 173, in get_encoder_out_chunk
        xs, att_cache, cnn_cache = self.encoder.forward_chunk(xs=speech,
    File "/data/PPASR/PPASR-develop/ppasr/model_utils/conformer/encoder.py", line 388, in forward_chunk
        if self.global_cmvn is not None:
    File "/data/PPASR/PPASR-develop/ppasr/model_utils/conformer/encoder.py", line 389, in forward_chunk
        xs = self.global_cmvn(xs)
    File "/data/PPASR/PPASR-develop/ppasr/model_utils/utils/cmvn.py", line 29, in forward
            (paddle.Tensor): normalized feature
        """
        x = x - self.mean
        ~~~~~~~~~~~~~~~~~ <--- HERE
        if self.norm_var:
            x = x * self.istd

    File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/layers/math_op_patch.py", line 407, in __impl__
        attrs={'axis': axis})
    File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3828, in append_op
        attrs=kwargs.get("attrs", None))
    File "/root/.conda/envs/paddle-gpu/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2874, in __init__
        self.desc.infer_shape(self.block.desc)

    ValueError: (InvalidArgument) Broadcast dimension mismatch. Operands could not be broadcast together with the shape of X = [1, -1, 80] and the shape of Y = [161]. Received [80] in X is not equal to [161] in Y at i:2.
  [Hint: Expected x_dims_array[i] == y_dims_array[i] || x_dims_array[i] <= 1 || y_dims_array[i] <= 1 == true, but received x_dims_array[i] == y_dims_array[i] || x_dims_array[i] <= 1 || y_dims_array[i] <= 1:0 != true:1.] (at /paddle/paddle/phi/kernels/funcs/common_shape.h:84)
  [operator < elementwise_sub > error]

[yeyupiaoling]

paddlepaddle版本安装错了，你看我使用的版本。

[thomascatlee]

用了conda的版本，python也换了3.8，还是一样的错：

[2022-11-09 14:03:23.099602 INFO ] trainer:export:518 - 成功恢复模型参数和优化方法参数：models/conformer_online_fbank/best_model/model.pdparams /root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/jit.py:838: UserWarning: What you save is a function, and jit.save will generate the name of the model file according to path you specify. When loading these files with jit.load, you get a TranslatedLayer whose inference result is the same as the inference result of the function you saved. warnings.warn( Traceback (most recent call last): File "export_model.py", line 27, in trainer.export(save_model_path=args.save_model, File "/data/PPASR/PPASR-develop/ppasr/trainer.py", line 527, in export paddle.jit.save(infer_model, infer_model_path) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), kw) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/wrapped_decorator.py", line 26, in impl return wrapped_func(args, kwargs) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/jit.py", line 649, in wrapper func(layer, path, input_spec, configs) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/decorator.py", line 232, in fun return caller(func, (extras + args), kw) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/wrapped_decorator.py", line 26, in impl return wrapped_func(*args, kwargs) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/base.py", line 52, in impl return func(*args, *kwargs) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/jit.py", line 959, in save concrete_program = attr_func.concrete_program_specify_input_spec( File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 574, in concrete_program_specify_input_spec concreteprogram, = self.get_concrete_program( File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 482, in get_concrete_program concrete_program, partial_program_layer = self._program_cache[cache_key] File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 952, in getitem self._caches[item_id] = self._build_once(item) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 936, in _build_once concrete_program = ConcreteProgram.from_func_spec( File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/decorator.py", line 232, in fun return caller(func, (extras + args), kw) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/wrapped_decorator.py", line 26, in impl return wrapped_func(*args, *kwargs) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/base.py", line 52, in impl return func(args, **kwargs) File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 892, in from_func_spec error_data.raise_new_exception() File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/error.py", line 350, in raise_newexception six.exec("raise new_exception from None") File "", line 1, in ValueError: In transformed code:

File "/data/PPASR/PPASR-develop/ppasr/model_utils/conformer/model.py", line 173, in get_encoder_out_chunk
    xs, att_cache, cnn_cache = self.encoder.forward_chunk(xs=speech,
File "/data/PPASR/PPASR-develop/ppasr/model_utils/conformer/encoder.py", line 388, in forward_chunk
    if self.global_cmvn is not None:
File "/data/PPASR/PPASR-develop/ppasr/model_utils/conformer/encoder.py", line 389, in forward_chunk
    xs = self.global_cmvn(xs)
File "/data/PPASR/PPASR-develop/ppasr/model_utils/utils/cmvn.py", line 29, in forward
        (paddle.Tensor): normalized feature
    """
    x = x - self.mean
    ~~~~~~~~~~~~~~~~~ <--- HERE
    if self.norm_var:
        x = x * self.istd

File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/layers/math_op_patch.py", line 401, in __impl__
    current_block(self).append_op(type=op_type,
File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/framework.py", line 3823, in append_op
    op = Operator(block=self,
File "/root/.conda/envs/ppasr/lib/python3.8/site-packages/paddle/fluid/framework.py", line 2874, in __init__
    self.desc.infer_shape(self.block.desc)

ValueError: (InvalidArgument) Broadcast dimension mismatch. Operands could not be broadcast together with the shape of X = [1, -1, 80] and the shape of Y = [161]. Received [80] in X is not equal to [161] in Y at i:2.

[Hint: Expected x_dims_array[i] == y_dims_array[i] || x_dims_array[i] <= 1 || y_dims_array[i] <= 1 == true, but received x_dims_array[i] == y_dims_array[i] || x_dims_array[i] <= 1 || y_dims_array[i] <= 1:0 != true:1.] (at /paddle/paddle/phi/kernels/funcs/common_shape.h:84) [operator < elementwise_sub > error]

[thomascatlee]

packages in environment at /root/.conda/envs/ppasr:

#

Name Version Build Channel

_libgcc_mutex 0.1 main defaults _openmp_mutex 5.1 1_gnu defaults astor 0.8.1 pyh9f0ad1d_0 conda-forge babel 2.11.0 pypi_0 pypi bce-python-sdk 0.8.74 pypi_0 pypi brotlipy 0.7.0 py38h0a891b7_1004 conda-forge ca-certificates 2022.9.24 ha878542_0 conda-forge certifi 2022.9.24 pyhd8ed1ab_0 conda-forge cffi 1.14.6 py38ha65f79e_0 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge click 8.1.3 pypi_0 pypi cn2an 0.5.19 pypi_0 pypi coloredlogs 15.0.1 pypi_0 pypi colorlog 6.7.0 pypi_0 pypi contourpy 1.0.6 pypi_0 pypi cryptography 37.0.2 py38h2b5fc30_0 conda-forge cudatoolkit 11.2.2 hbe64b41_10 conda-forge cudnn 8.1.0.77 h90431f1_0 conda-forge cycler 0.11.0 pypi_0 pypi decorator 5.1.1 pyhd8ed1ab_0 conda-forge dill 0.3.6 pypi_0 pypi ffmpeg-python 0.2.0 pypi_0 pypi flask 2.1.3 pypi_0 pypi flask-babel 2.0.0 pypi_0 pypi flask-cors 3.0.10 pypi_0 pypi flatbuffers 22.10.26 pypi_0 pypi fonttools 4.38.0 pypi_0 pypi freetype 2.10.4 h0708190_1 conda-forge future 0.18.2 pypi_0 pypi humanfriendly 10.0 pypi_0 pypi idna 3.4 pyhd8ed1ab_0 conda-forge ijson 3.1.4 pypi_0 pypi importlib-metadata 5.0.0 pypi_0 pypi itsdangerous 2.1.2 pypi_0 pypi jbig 2.1 h7f98852_2003 conda-forge jinja2 3.1.2 pypi_0 pypi joblib 1.2.0 pypi_0 pypi jpeg 9e h166bdaf_1 conda-forge kiwisolver 1.4.4 pypi_0 pypi lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.38 h1181459_1 defaults lerc 2.2.1 h9c3ff4c_0 conda-forge libblas 3.9.0 15_linux64_openblas conda-forge libcblas 3.9.0 15_linux64_openblas conda-forge libdeflate 1.7 h7f98852_5 conda-forge libffi 3.3 he6710b0_2 defaults libgcc-ng 11.2.0 h1234567_1 defaults libgfortran-ng 12.2.0 h69a702a_19 conda-forge libgfortran5 12.2.0 h337968e_19 conda-forge libgomp 11.2.0 h1234567_1 defaults liblapack 3.9.0 15_linux64_openblas conda-forge libopenblas 0.3.20 pthreads_h78a6416_0 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libprotobuf 3.18.0 h780b84a_1 conda-forge libstdcxx-ng 11.2.0 h1234567_1 defaults libtiff 4.3.0 hf544144_1 conda-forge libwebp-base 1.2.2 h7f98852_1 conda-forge llvmlite 0.39.1 pypi_0 pypi lz4-c 1.9.3 h9c3ff4c_1 conda-forge markupsafe 2.1.1 pypi_0 pypi matplotlib 3.6.2 pypi_0 pypi mpmath 1.2.1 pypi_0 pypi multiprocess 0.70.14 pypi_0 pypi ncurses 6.3 h5eee18b_3 defaults numba 0.56.4 pypi_0 pypi numpy 1.22.3 py38h99721a1_2 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge onnxruntime 1.13.1 pypi_0 pypi openjpeg 2.4.0 hb52868f_1 conda-forge openssl 1.1.1s h7f8727e_0 defaults opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge packaging 21.3 pypi_0 pypi paddleaudio 1.0.2 pypi_0 pypi paddlepaddle-gpu 2.4.0rc0.post112 pypi_0 pypi pandas 1.5.1 pypi_0 pypi pathos 0.2.8 pypi_0 pypi pillow 8.3.2 py38h8e6f84c_0 conda-forge pip 22.2.2 py38h06a4308_0 defaults pox 0.3.2 pypi_0 pypi ppasr 2.0.1 pypi_0 pypi ppft 1.7.6.6 pypi_0 pypi proces 0.1.3 pypi_0 pypi protobuf 3.18.0 py38h709712a_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pycryptodome 3.15.0 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pyopenssl 22.0.0 pyhd8ed1ab_1 conda-forge pyparsing 3.0.9 pypi_0 pypi pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.8.13 haa1d7c7_1 defaults python-dateutil 2.8.2 pypi_0 pypi python-levenshtein 0.12.2 pypi_0 pypi python_abi 3.8 2_cp38 conda-forge pytz 2022.6 pypi_0 pypi pyyaml 5.4.1 pypi_0 pypi readline 8.2 h5eee18b_0 defaults requests 2.28.1 pyhd8ed1ab_1 conda-forge resampy 0.2.2 pypi_0 pypi scikit-learn 1.1.3 pypi_0 pypi scipy 1.9.3 pypi_0 pypi setuptools 65.5.0 py38h06a4308_0 defaults six 1.16.0 pyh6c4a22f_0 conda-forge sklearn 0.0.post1 pypi_0 pypi soundfile 0.11.0 pypi_0 pypi sqlite 3.39.3 h5082296_0 defaults sympy 1.11.1 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi threadpoolctl 3.1.0 pypi_0 pypi tk 8.6.12 h1ccaba5_0 defaults tqdm 4.64.1 pypi_0 pypi typeguard 2.13.3 pypi_0 pypi urllib3 1.26.11 pyhd8ed1ab_0 conda-forge visualdl 2.4.1 pypi_0 pypi webrtcvad 2.0.10 pypi_0 pypi websockets 10.4 pypi_0 pypi werkzeug 2.2.2 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0 defaults xz 5.2.6 h5eee18b_0 defaults zhconv 1.4.3 pypi_0 pypi zipp 3.10.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0 defaults zstd 1.5.0 ha95c52a_0 conda-forge

[thomascatlee]

和我cudnn版本低有关系吗，装的8.1

[yeyupiaoling]

PaddlePaddle要2.4.0rc0以上

[thomascatlee]

paddlepaddle-gpu 2.4.0rc0.post112 pypi_0 pypi，已经是了啊。

PPASR是git上直接下载的，有关系嘛？

[yeyupiaoling]

如果代码是最近下载的，应该是没有问题的

[yeyupiaoling]

下载的模型文件里面，有没有全部复制到项目根目录，替换原来的文件？

[thomascatlee]

是复制的，全替换

[thomascatlee]

我已经找到问题了，我的代码是2号的，从git上拉一下，重新装一遍就好了

[thomascatlee]

感谢大大耐心解答