search

CUNY-CL / yoyodyne

Small-vocabulary sequence-to-sequence generation with optional feature conditioning
Apache License 2.0
25 stars 15 forks source link

Fix prediction bug with pointer_generator_transformer #172

Closed michaelpginn closed 2 months ago

michaelpginn commented 2 months ago

Right now, if you

  1. Train a model with pointer_generator_transformer and features
  2. Run prediction with the trained model

There is an "index out of range error":

Traceback (most recent call last):
  File "/Users/USER/.pyenv/versions/3.10.13/bin/yoyodyne-predict", line 8, in <module>
    sys.exit(main())
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/yoyodyne/predict.py", line 158, in main
    predict(trainer, model, datamodule, args.output)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/yoyodyne/predict.py", line 101, in predict
    for batch in trainer.predict(model, loader):
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 892, in predict
    return call._call_and_handle_interrupt(
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 938, in _predict_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1190, in _run_stage
    return self._run_predict()
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1244, in _run_predict
    return self.predict_loop.run()
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/prediction_loop.py", line 100, in advance
    dl_predictions, dl_batch_indices = self.epoch_loop.run(
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/prediction_epoch_loop.py", line 100, in advance
    self._predict_step(batch, batch_idx, dataloader_idx)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/prediction_epoch_loop.py", line 129, in _predict_step
    predictions = self.trainer._call_strategy_hook("predict_step", *step_kwargs.values())
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1494, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 408, in predict_step
    return self.model.predict_step(*args, **kwargs)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/yoyodyne/models/base.py", line 337, in predict_step
    predictions = self(batch)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/yoyodyne/models/pointer_generator.py", line 676, in forward
    source_encoded = self.source_encoder(batch.source).output
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/yoyodyne/models/modules/transformer.py", line 197, in forward
    embedding = self.embed(source.padded)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/yoyodyne/models/modules/transformer.py", line 182, in embed
    word_embedding = self.esq * self.embeddings(symbols)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 160, in forward
    return F.embedding(
  File "/Users/USER/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

See an example at https://github.com/michaelpginn/yoyodyne-bug

The issue is being caused by the pointer_generator_transformer using a separate features encoder, but not expecting separate features at inference time. I added pointer_generator_transformer to the models that expect separate features (and checked with @Adamits).