How to accelerate the generation?

[ztcintokyo]

Hi I would like to generate 1M sentences using your ACCESS. However, using generate.py seems slow, I'll appreciate it if you can give me some ideas to accelerate it. Thank you!

[louismartin]

Hi, thanks for your interest!

I am not sure how to speed this up, however here are some ideas that might help. The generation uses the fairseq-generate command in the _fairseq_generate function.

• Are you using a GPU? If not, you should definitely use one
• You could lower the beam size here, at the extreme, a beam size of 1 would be much faster at the cost of quality of the simplifications. Maybe you could try lowering it progressively until it's fast enough (8->5->3->1).
• Do you have access to multiple GPUs? AWS? For a relatively low price you could parallelize the simplification by splitting your sentences among multiple GPUs. If you have only one GPU maybe having multiple processes on the same GPU could help.
• The preprocessing step (computing the control ratios) might be distributable as well, so trying to generate using multiple processes can help with this respect.

Good luck, Best, Louis

[ztcintokyo]

Hi, thank you for your reply. I'm wondering how to use GPU for the generation? I tried "CUDA_VISIBLE_DEVICES=3 python scripts/generate.py < train-1-en.txt > train-1-en-simp.txt" but nvidia-smi shows that this GPU didn't work. Maybe it's a stupid question, hoping for your reply, thank you!

[louismartin]

Can you try CUDA_VISIBLE_DEVICES=0 python -c 'import torch; print(torch.cuda.device_count())' as per this issue? Maybe it's a pytorch problem.

[ztcintokyo]

I tried CUDA_VISIBLE_DEVICES=3 python -c 'import torch; print(torch.cuda.device_count())' and the output is 1

[louismartin]

Hi @ztcintokyo I am not sure what is the problem then, maybe you should drop into pdb and check that use_cuda is set to True here and check that during the generation, the tensors are indeed on the GPU.

[ztcintokyo]

I have solved the problem. Thank you for your patience!