search

microsoft / ProphetNet

A research project for natural language generation, containing the official implementations by MSRA NLC team.
MIT License
654 stars 105 forks source link

Increasing --max-source-positions --max-target-positions #21

Closed ShoubhikBanerjee closed 3 years ago

ShoubhikBanerjee commented 3 years ago

Hi again,

I was finetuning some data with --max-source-positions 1024 --max-target-positions 1024.

But it paused at epoch 001: 8%.
and showed: WARNING: overflow detected, setting loss scale to: 64.0 Is there, any upper limit with --max-source-positions & --max-target-positions .

I am training with 4 Tesla T4 GPUs.

Please help.

smita181298 commented 3 years ago

Hi @ShoubhikBanerjee .I am working on abstractive summarization using the prophetnet right now. I am very new to pytorch and abstractive summarization. Can you help me with how to change the number of sentences in the summary predicted by prophetnet?

yuyan2do commented 3 years ago

@ShoubhikBanerjee the overflow warning is not relatede to max position. Pause maybe caused by OOM, could you try with smaller batch size?

yuyan2do commented 3 years ago

@smita181298 You want control how many sentence it generated or the length?

smita181298 commented 3 years ago

I want to control both for different usecases @yuyan2do .It will be really helpful if you can tell how to change the length and number of sentences both.

Thanks

ShoubhikBanerjee commented 3 years ago

@ShoubhikBanerjee the overflow warning is not relatede to max position. Pause maybe caused by OOM, could you try with smaller batch size?

Thank @yuyan2do , That means there is not any upper limit form --max-source-positions 1024 --max-target-positions 1024. right?

@smita181298 I saw that you have already opened a new issue so I am closing it. @yuyan2do , I am closing this issue, thank you again! Just reply back whether I am right or not, regarding the upper limit.

Thanks All

yuyan2do commented 3 years ago

@ShoubhikBanerjee Your understand is right.