Closed tommccoy1 closed 4 years ago
Hey! I've also observed this with the CMU Transformer-XL codebase. The main difference with other Transformers is the adaptive softmax, so that's the first I'd look at; does an XL model with a normal projection layer also have problems with batching ?
I was actually planning to investigate a suspected bug in HF's Transformer-XL training performance tomorrow morning, so if it's not urgent I can also take a look at that at the same time.
It's encouraging to hear that someone else has observed this! Thanks for the suggestion - I just tried turning off the adaptive softmax (by changing the line model = model_class.from_pretrained(args.model_name_or_path)
to model = model_class.from_pretrained(args.model_name_or_path, adaptive=False)
), but that did not change the runtimes.
It's not urgent, so it would be much appreciated if you can take a look!
So here are my observations for now, running on my laptop's RTX 2070 (transformers 2.11.0, torch 1.5.0, python 3.6.9, CUDA 10.2, no mixed precision) at training time for that other bug hunt:
adaptive=False
does not actually do anything as far as I can tell, the adaptive
attribute of config
isn't used anywherebatch size | Adaptive XL | Linear XL | GPT-2 |
---|---|---|---|
1 | 33.27 it/s | 29.16 it/s | 35.06 it/s |
2 | 31.06 it/s | 19.93 it/s | 24.86 it/s |
4 | 29.30 it/s | 13.63 it/s | 14.87 it/s |
8 | 23.03 it/s | 7.85 it/s | 8.49 it/s |
So that's pretty strange. What is your version of transformers ? I'll be looking at inference time now, as it may be different from training to inference. EDIT: also the case for me at inference time | batch size | Adaptive XL | Linear XL | GPT-2 |
---|---|---|---|---|
1 | 286.92 it/s | 197.25 it/s | 216.45 it/s | |
2 | 264.54 it/s | 102.02 it/s | 109.74 it/s | |
4 | 214.71 it/s | 56.27 it/s | 59.91 it/s | |
8 | 148.69 it/s | 30.35 it/s | 31.97 it/s |
Another lead is the einsum function; it's used in transformer-XL but doesn't look like it is used in GPT-2, and I know that it can behave poorly sometimes especially in mixed-precision settings. Are you using apex?
Interesting!
I'm using transformers 2.10.0, and am not using apex.
If you're able to share the code you were using for inference time, that would be helpful, so I can try it & see if it's my code or my environment that's giving us different results.
I cleaned up the code a bit and uploaded it on Google Drive. It uses lightning and operates on real wt103 data (included in the zip) so it's not quite minimal though.
Another (more remote) possibility is an issue in batching, after looking again at my dataloader code it was a bit more complex than usual to support transfoXL memories.
Thanks for the code! The main difference I see between your code and mine is that I am using the generate
function, whereas you don't. After looking into the generate
function for Transformer-XL, I believe I have found a bug.
Here is code that uses greedy generation without the generate
function:
from transformers import TransfoXLLMHeadModel, TransfoXLTokenizer
import torch
tokenizer = TransfoXLTokenizer.from_pretrained("transfo-xl-wt103")
model = TransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103')
generated = tokenizer.encode("The Manhattan Bridge")
context = torch.tensor([generated])
mems = None
for i in range(100):
print(i)
output, mems = model(context, mems=mems)[:2]
token = torch.argmax(output[..., -1, :])
generated += [token.tolist()]
context = token.unsqueeze(0).unsqueeze(0)
sequence = tokenizer.decode(generated)
print(sequence)
This generates the following text:
The Manhattan Bridge,
= = = = The Bridge = = = = The bridge over the Delaware River was built in the late 19th century by the Delaware and Hudson Canal Company. The bridge was built in the style of a drawbridge, with a single span of 1 @,@ 200 feet ( 370 m ). The bridge was designed by John Roebling, who also designed the Delaware River Bridge. The bridge was built in the style of a drawbridge, with a single span of 1 @,@ 200 feet ( 370 m
The code below should also generate the same text, just using the generate
function:
from transformers import TransfoXLLMHeadModel, TransfoXLTokenizer
import torch
tokenizer = TransfoXLTokenizer.from_pretrained("transfo-xl-wt103")
model = TransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103')
model.to("cuda")
generated = tokenizer.encode("The Manhattan Bridge")
context = torch.tensor([generated]).to("cuda")
mems = None
print(context)
output_sequences = model.generate(
input_ids=context,
max_length=100 + len(generated),
min_length=100 + len(generated),
eos_token_id=267734,
#temperature=1.0,
#top_k=1,
#top_p=1.0,
#do_sample=True,
#num_return_sequences=1,
)
sequence = tokenizer.decode(output_sequences[0])
print(sequence)
However, it does not give the same output; instead, it generates:
The Manhattan Bridge,
the Brooklyn Bridge, the Brooklyn Bridge, the Brooklyn Bridge, the Brooklyn Bridge, the Brooklyn Bridge, the Brooklyn Bridge, the Brooklyn Bridge, the Brooklyn Bridge, = = = The Manhattan Bridge, =, " the the the the.. the , , The, The The, The New York Bridge, is a double @-@ A @-@ A @-@ The Manhattan Bridge, the Brooklyn Bridge,
I was able to fix the discrepancy by changing the prepare_inputs_for_generation
function of Transformer-XL to the code below (similar to the code used for that function in GPT-2):
def prepare_inputs_for_generation(self, input_ids, past, **model_kwargs):
inputs = {}
# if past is defined in model kwargs then use it for faster decoding
if past:
inputs["mems"] = past
inputs["input_ids"] = input_ids[:, -1].unsqueeze(-1)
else:
inputs["input_ids"] = input_ids
return inputs
With this code, the generate
function gives the same output as a for-loop. In addition, this also speeds up generation substantially: My use case is generating 500-token text from 512-token prompts, and that now takes about 30 seconds per prompt, while previously it was 3 minutes per prompt. Batching also is now more helpful than before - still not as helpful as I would expect, but that doesn't matter because it's now fast enough to be perfectly useful for me.
I've made a draft pull request here: https://github.com/huggingface/transformers/pull/4826. But I'm not sure if it's ready to be submitted (I've never submitted a pull request before): some of the tests in make test
fail, and I'm not sure what is required for step 5 of the pull request checklist ("Add high-coverage tests.").
Fixed by #4826
I have modified the example
[run_generation.py](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py)
so that it can use batches. My code (pared down for the example) is below, calledbatch_gen.py
:To test the speedup provided by batching, I use a text file called
prompts.txt
with the following prompts:The following command is used to run the code with GPT-2:
With GPT-2, batching speeds up the runtime as expected: Each batch takes approximately 1 second, regardless of whether the batch size is 1, 5, or 10. However, with Transformer-XL, this is not the case. Here is the command to run with Transformer-XL:
With a batch size of 1, each batch takes 3 seconds. With a batch size of 5, each batch takes 12 seconds. With a batch size of 10, each batch takes 21 seconds. Thus, batching is not providing much of a speedup compared to generating examples serially. (You can see the amount of time each batch takes by looking at the time stamps on the log messages that are printed out).
Therefore, I am wondering if there is a bug in the batching for Transformer-XL? Or is there some reason why the architecture cannot support efficient batching?
I am running this code on a p100 GPU through Ubuntu version 18.04 with PyTorch version 1.5.0 and Python version 3.7.7.
Thank you!