search

CUNY-CL / yoyodyne

Small-vocabulary sequence-to-sequence generation with optional feature conditioning
Apache License 2.0
25 stars 15 forks source link

Index error with hard monotonic attention and features #194

Closed kylebgorman closed 2 weeks ago

kylebgorman commented 2 weeks ago

(It is always possible that I am just doing something wrong here.)

Invocation:

yoyodyne-train \
    --experiment "${LANGUAGE}" \
    --train "${TRAIN}" \
    --val "${VAL}" \
    --model_dir "${MODEL_DIR}" \
    --features_col 3 \
    --arch hard_attention_lstm \
    --enforce_monotonic \
    --hard_attention_context 1 \
    --hidden_size 400 \
    --batch_size 128 \
    --embedding_size 200 \
    --accelerator gpu \
    --max_epochs 60 \
    --log_every_n_step 20 \
    --learning_rate .001 \
    --dropout .4 \
    --seed 49 \
    --gradient_clip_val 3

The trace below is the CPU one because it's more informative---you get a less informative error from CUDA if you're on GPU.

  File "/home/kbg/.miniconda3/envs/py310/bin/yoyodyne-train", line 8, in <module>
    sys.exit(main())
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/yoyodyne/train.py", line 406, in main
    best_checkpoint = train(trainer, model, datamodule, args.train_from)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/yoyodyne/train.py", line 267, in train
    trainer.fit(model, datamodule, ckpt_path=train_from)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
    self._run_train()
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1204, in _run_train
    self._run_sanity_check()
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1276, in _run_sanity_check
    val_loop.run()
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
    output = self._evaluation_step(**kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
    output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1494, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 390, in validation_step
    return self.model.validation_step(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/yoyodyne/models/hard_attention.py", line 372, in validation_step
    predictions, likelihood = self(batch)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/yoyodyne/models/hard_attention.py", line 101, in forward
    encoder_out = self.source_encoder(batch.source).output
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/yoyodyne/models/modules/lstm.py", line 73, in forward
    embedded = self.embed(source.padded)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/yoyodyne/models/modules/base.py", line 155, in embed
    embedded = self.embeddings(symbols)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward
    return F.embedding(
  File "/home/kbg/.miniconda3/envs/py310/lib/python3.10/site-packages/torch/nn/functional.py", line 2237, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
bonham79 commented 2 weeks ago

What's the dataset you're using? (For comparison)

On Sun, Jun 9, 2024, 1:18 PM Kyle Gorman @.***> wrote:

Assigned #194 https://github.com/CUNY-CL/yoyodyne/issues/194 to @bonham79 https://github.com/bonham79.

— Reply to this email directly, view it on GitHub https://github.com/CUNY-CL/yoyodyne/issues/194#event-13092483191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIUDR73VXOTKQKSV3NSYURTZGS2CBAVCNFSM6AAAAABJBFBSNGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGA4TENBYGMYTSMI . You are receiving this because you were assigned.Message ID: @.***>

kylebgorman commented 2 weeks ago

What's the dataset you're using? (For comparison)

Polish abstractness data. Concatenating segments 0-7 together for train, using segment 8 for validation. LMK offthread if you need links etc.

bonham79 commented 2 weeks ago

Ah got it, forgot to add hard_attention_lstm to the separate_features portion of our datamodule creator.