search

awslabs / sockeye

Sequence-to-sequence framework with a focus on Neural Machine Translation based on PyTorch
https://awslabs.github.io/sockeye/
Apache License 2.0
1.21k stars 324 forks source link

RuntimeError: "LayerNormKernelImpl" not implemented for 'Half' (sockeye==3.1.27) #1083

Closed SamuelLarkin closed 1 year ago

SamuelLarkin commented 1 year ago

Hi, I'm trying to quantize at inference time a float32 model into float16. It looks like pytorch doesn't support this or am I missing some environment variable that I need to set to enable this? I'm using sockeye==3.1.27. I also tried to use a float16 model (aka sockeye-quantize --model model/params.best --config model/args.yaml --dtype float16) and then sockeye-translate ... --use-cpu --dtype int8 and got the same error message. If I tried to translate using a float32 model and --dtype int8, I get some translations. My goal here is to save the model into a smaller file and use it to translate on CPUs.

Command

python \
  -m sockeye.translate \
  --output-type json \
  --batch-size 32 \
  --models ../model \
  --input source.en \
  --use-cpu \
  --dtype float16

Conda Environment

conda env export

name: sockeye-3.1.27.cpuonly
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - blas=1.0=mkl
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2023.01.10=h06a4308_0
  - certifi=2022.12.7=py310h06a4308_0
  - cpuonly=2.0=0
  - flit-core=3.6.0=pyhd3eb1b0_0
  - intel-openmp=2022.1.0=h9e868ea_3769
  - ld_impl_linux-64=2.38=h1181459_1
  - libffi=3.4.2=h6a678d5_6
  - ld_impl_linux-64=2.38=h1181459_1                                                                                                                           [0/401]
  - libffi=3.4.2=h6a678d5_6
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - libuuid=1.41.5=h5eee18b_0
  - mkl=2022.1.0=hc2b9512_224
  - ncurses=6.4=h6a678d5_0
  - openssl=1.1.1s=h7f8727e_0
  - pip=22.3.1=py310h06a4308_0
  - python=3.10.9=h7a1cb2a_0
  - pytorch=1.13.1=py3.10_cpu_0
  - pytorch-mutex=1.0=cpu
  - readline=8.2=h5eee18b_0
  - setuptools=65.6.3=py310h06a4308_0
  - sqlite=3.40.1=h5082296_0
  - tk=8.6.12=h1ccaba5_0
  - typing_extensions=4.4.0=py310h06a4308_0
  - tzdata=2022g=h04d1e81_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - xz=5.2.10=h5eee18b_1
  - zlib=1.2.13=h5eee18b_0
  - pip:
      - click==8.1.3
      - colorama==0.4.6
      - jedi==0.18.2
      - lxml==4.9.2
      - numpy==1.24.2
      - packaging==23.0
      - parso==0.8.3
      - portalocker==2.7.0
      - pudb==2022.1.3
      - pygments==2.14.0
      - pyyaml==6.0
      - regex==2022.10.31
      - sacrebleu==2.3.1
      - sockeye==3.1.27
      - tabulate==0.9.0
      - urwid==2.1.2
      - urwid-readline==0.13
prefix: /gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly

Error Message

[INFO:sockeye.utils] Sockeye: 3.1.27, commit 288baa7bb3989418f33f91dc73a328601dca61e6, path /gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib
/python3.10/site-packages/sockeye/__init__.py
[INFO:sockeye.utils] PyTorch: 1.13.1 (/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/__init__.py)
[INFO:sockeye.utils] Command: /gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/translate.py --output-type j
son --batch-size 32 --models ../model --input source.en --use-cpu --dtype float16
[INFO:sockeye.utils] Arguments: Namespace(config=None, input='source.en', input_factors=None, json_input=False, output=None, models=['../model'], checkpoints=None, nb
est_size=1, beam_size=5, greedy=False, beam_search_stop='all', batch_size=32, chunk_size=None, sample=None, seed=None, ensemble_mode='linear', bucket_width=10, max_in
put_length=None, max_output_length_num_stds=2, max_output_length=None, restrict_lexicon=None, restrict_lexicon_topk=None, skip_nvs=False, nvs_thresh=0.5, strip_unknow
n_words=False, prevent_unk=False, output_type='json', length_penalty_alpha=1.0, length_penalty_beta=0.0, brevity_penalty_type='none', brevity_penalty_weight=1.0, brev
ity_penalty_constant_length_ratio=0.0, dtype='float16', clamp_to_dtype=False, device_id=0, use_cpu=True, env=None, tf32=True, quiet=False, quiet_secondary_workers=Fal
se, no_logfile=False, loglevel='INFO', loglevel_secondary_workers='INFO')
[INFO:__main__] CUDA not available, using cpu
[INFO:__main__] Translate Device: cpu
[INFO:sockeye.model] Loading 1 model(s) from ['../model'] ...
[INFO:sockeye.vocab] Vocabulary (32170 words) loaded from "../model/vocab.src.0.json"
[INFO:sockeye.vocab] Vocabulary (32170 words) loaded from "../model/vocab.trg.0.json"
[INFO:sockeye.model] Model version: 3.1.27
[INFO:sockeye.model] Loaded model config from "../model/config"
[INFO:sockeye.model] Disabling dropout layers for performance reasons
[INFO:sockeye.model] ModelConfig(config_data=DataConfig(data_statistics=DataStatistics(num_sents=18792562, num_discarded=4514, num_tokens_source=396805440, num_tokens
_target=452828161, num_unks_source=151, num_unks_target=150, max_observed_len_source=201, max_observed_len_target=201, size_vocab_source=32170, size_vocab_target=3217
0, length_ratio_mean=1.149165240213179, length_ratio_std=0.3331394866848643, buckets=[(8, 8), (16, 16), (24, 24), (32, 32), (40, 40), (48, 48), (56, 56), (64, 64), (7
2, 72), (80, 80), (88, 88), (96, 96), (104, 104), (112, 112), (120, 120), (128, 128), (136, 136), (144, 144), (152, 152), (160, 160), (168, 168), (176, 176), (184, 18
4), (192, 192), (200, 200), (201, 201)], num_sents_per_bucket=[2488902, 5093385, 3839506, 2546445, 1811528, 1189585, 737992, 443043, 261406, 152182, 89417, 52498, 310
55, 18857, 11863, 7560, 4930, 3414, 2367, 1796, 1381, 1081, 892, 762, 633, 82], average_len_target_per_bucket=[4.876037945069864, 12.773357231261594, 19.5802367533320
52, 27.41336983211651, 35.28823864603507, 43.238106587956544, 51.26582055583915, 59.238620628568825, 67.207513428303, 75.18162933867328, 83.15075619854574, 91.1199173
0707263, 99.07102421459187, 106.99094438055295, 114.94170231821508, 122.88471587159029, 130.96489273583816, 138.7659185760993, 146.50905591051549, 154.6460870395282,
162.2368031213424, 170.14799935605743, 178.83465301964753, 186.0052493281894, 193.85046425851448, 199.38909318975016], length_ratio_stats_per_bucket=[(1.0693756944877
173, 0.2734448342497526), (1.0857894201553209, 0.28019690452817625), (1.1544188404375997, 0.3868604549199259), (1.1861185841999833, 0.3074059313735606), (1.2060657841
545896, 0.2981587515456939), (1.226324650666722, 0.30493095494070144), (1.2444125341378565, 0.3242223370686047), (1.2611266481183327, 0.3709455795374948), (1.27464433
7588064, 0.4302137928506163), (1.2860484970016222, 0.48695434193951453), (1.302799569788393, 0.5653045419192184), (1.3120006329314209, 0.6142970487431451), (1.3295351
237968134, 0.7814292394252162), (1.3384637458091257, 0.8116763474141028), (1.351862242960138, 0.9642116646873813), (1.3368067683991776, 0.7653903732034699), (1.367077588064, 0.4302137928506163), (1.2860484970016222, 0.48695434193951453), (1.302799569788393, 0.5653045419192184), (1.3120006329314209, 0.6142970487431451), (1[40/345]
237968134, 0.7814292394252162), (1.3384637458091257, 0.8116763474141028), (1.351862242960138, 0.9642116646873813), (1.3368067683991776, 0.7653903732034699), (1.367075
2245352829, 0.9719727938185959), (1.3805636470652694, 1.0975590088160094), (1.3476927572822692, 0.696317634165507), (1.3496332871268524, 0.7573960914043955), (1.30467
33705213736, 0.6783789455528596), (1.3753328246704346, 1.6470598091351123), (1.3040674746204497, 1.059827519965373), (1.253641535651391, 0.5013375061317442), (1.24804
87830675664, 0.3590853095382778), (1.2543975958596236, 0.31963245113954747)]), max_seq_len_source=201, max_seq_len_target=201, num_source_factors=1, num_target_factor
s=1), vocab_source_size=32170, vocab_target_size=32170, config_embed_source=EmbeddingConfig(vocab_size=32170, num_embed=1024, dropout=0.0, num_factors=1, factor_confi
gs=None, allow_sparse_grad=False), config_embed_target=EmbeddingConfig(vocab_size=32170, num_embed=1024, dropout=0.0, num_factors=1, factor_configs=None, allow_sparse
_grad=False), config_encoder=TransformerConfig(model_size=1024, attention_heads=16, feed_forward_num_hidden=4096, act_type='relu', num_layers=6, dropout_attention=0.0
, dropout_act=0.0, dropout_prepost=0.0, positional_embedding_type='fixed', preprocess_sequence='n', postprocess_sequence='dr', max_seq_len_source=201, max_seq_len_tar
get=201, decoder_type='transformer', use_lhuc=False, depth_key_value=1024, use_glu=False), config_decoder=TransformerConfig(model_size=1024, attention_heads=16, feed_
forward_num_hidden=4096, act_type='relu', num_layers=6, dropout_attention=0.0, dropout_act=0.0, dropout_prepost=0.0, positional_embedding_type='fixed', preprocess_seq
uence='n', postprocess_sequence='dr', max_seq_len_source=201, max_seq_len_target=201, decoder_type='transformer', use_lhuc=False, depth_key_value=1024, use_glu=False)
, config_length_task=None, weight_tying_type='src_trg_softmax', lhuc=False, dtype='float32', neural_vocab_selection=None, neural_vocab_selection_block_loss=False)
[INFO:sockeye.model] Loaded params from "../model/params.best" to "cpu"
[INFO:sockeye.model] Casting SockeyeModel to dtype torch.float16
[INFO:sockeye.model] Model dtype: overridden to float16
[INFO:sockeye.model] 1 model(s) loaded in 16.9071s
[INFO:sockeye.inference] Translator (1 model(s) beam_size=5 algorithm=BeamSearch, beam_search_stop=all max_input_length=200 nbest_size=1 ensemble_mode=None max_batch_
size=32 dtype=torch.float16 skip_nvs=False nvs_thresh=0.5)
[INFO:__main__] Translating...
[ERROR:root] Uncaught exception
Traceback (most recent call last):
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/translate.py", line 263, in <module>
    main()
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/translate.py", line 44, in main
    run_translate(args)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/translate.py", line 145, in run_translate
    read_and_translate(translator=translator,
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/translate.py", line 231, in read_and_translate
    chunk_time = translate(output_handler, chunk, translator)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/translate.py", line 254, in translate
    trans_outputs = translator.translate(trans_inputs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/inference.py", line 942, in translate
    batch_translations = self._translate_np(*self._get_inference_input(translator_inputs))
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/inference.py", line 1183, in _translate_np
    return self._get_best_translations(self._search(source,
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/beam_search.py", line 918, in forward
    model_states, estimated_reference_lengths, nvs_prediction = self._inference.encode_and_initialize(source, source_length)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/beam_search.py", line 77, in encode_and_initialize
    states, predicted_output_length, nvs_prediction = self._model.encode_and_initialize(inputs, valid_length, self._const_lr)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/model.py", line 239, in encode_and_initialize
    source_encoded, source_encoded_lengths, att_mask = self.encode(inputs, valid_length=valid_length)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/model.py", line 220, in encode
    self.traced_encoder = pt.jit.trace(self.encoder, (source_embed, valid_length))
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/jit/_trace.py", line 759, in trace
    return trace_module(
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/jit/_trace.py", line 976, in trace_module
    module._c._create_method_from_trace(
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/encoder.py", line 218, in forward
    data = layer(data, att_mask=att_mask)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/transformer.py", line 105, in forward
    data_self_att, _ = self.self_attention(inputs=self.pre_self_attention(data),
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/sockeye/transformer.py", line 300, in forward
    data = self.layer_norm(data)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/modules/normalization.py", line 190, in forward
    return F.layer_norm(
  File "/gpfs/projects/DT/mtp/WMT20/opt/miniconda3/envs/sockeye-3.1.27.cpuonly/lib/python3.10/site-packages/torch/nn/functional.py", line 2515, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'
mjdenkowski commented 1 year ago

Hi Samuel,

Sockeye doesn't currently support FP16 inference on CPUs since PyTorch doesn't have CPU FP16 implementations of all the operators we use. For 16-bit CPU inference, you could try BF16: sockeye-translate --dtype bfloat16 ...

Best, Michael

SamuelLarkin commented 1 year ago

Thanks for the pointer. Turns out I was using sockeye-3.1.27 which didn't have that option. It initially failed using pytorch-1.11.0 but I was successful doing sockeye-translate --use-cpu --dtype bfloat16 when I used pytorch-1.13.1.

I opened a separate issue #1084.