Open prithvijaunjale opened 4 years ago
I don't even get truncation. I am thrown an error:
Traceback (most recent call last):
File "/usr/local/bin/fairseq-interactive", line 11, in <module>
load_entry_point('fairseq==0.9.0', 'console_scripts', 'fairseq-interactive')()
File "/usr/local/lib/python3.6/dist-packages/fairseq_cli/interactive.py", line 190, in cli_main
main(args)
File "/usr/local/lib/python3.6/dist-packages/fairseq_cli/interactive.py", line 136, in main
for batch in make_batches(inputs, args, task, max_positions, encode_fn):
File "/usr/local/lib/python3.6/dist-packages/fairseq_cli/interactive.py", line 48, in make_batches
max_positions=max_positions,
File "/usr/local/lib/python3.6/dist-packages/fairseq/tasks/fairseq_task.py", line 150, in get_batch_iterator
indices, dataset, max_positions, raise_exception=(not ignore_invalid_inputs),
File "/usr/local/lib/python3.6/dist-packages/fairseq/data/data_utils.py", line 188, in filter_by_size
).format(ignored[0], dataset.size(ignored[0]), max_positions))
Exception: Size of sample #0 is invalid (=(611, 0)) since max_positions=(512, 512), skip this example with --skip-invalid-size-inputs-valid-test
Input:
fairseq-interactive ../ProphetNet_resources/cnndm/processed \
--path ../ProphetNet_resources/prophetnet_large_160G_cnndm_model.pt \
--user-dir ./src/prophetnet \
--max-source-positions 6000 --max-target-positions 512 \
--task translation_prophetnet
I finetuned the pretrained ProphetNet model for 1 epoch on my own dataset on Google Colab for a summarization task. For inference I used:
Output:
Before passing as input, source text length = 2735 After = 593 Every source text longer than appx. 600 tokens gets truncated even though I have mentioned the source and target lengths (6000 & 512 respectively).
Would appreciate help on this! Thank you.