Exceeding the FF buffer size when running mt5_ff.py with BERT

flexflow / FlexFlow

FlexFlow Serve: Low-Latency, High-Performance LLM Serving

https://flexflow.readthedocs.io

Apache License 2.0

1.73k stars 232 forks source link

Exceeding the FF buffer size when running mt5_ff.py with BERT #490

Closed tnoyola closed 2 years ago

tnoyola commented 2 years ago

The following error is occurring when I run mt5_ff.py with BERT instead of MT5.

lockshaw commented 2 years ago

@tnoyola small note: in the future can you post formatted code, i.e.,

example

rather than an image? Search doesn't index the text in images :slightly_smiling_face:

lockshaw commented 2 years ago

Potentially linked to #417?

lockshaw commented 2 years ago

Tentatively assigning to @jiazhihao as he likely has more experience with this code. @jiazhihao feel free to correct me if this is wrong or transfer to someone else :slightly_smiling_face:

tnoyola commented 2 years ago

Resolved this issue by enlarging the buffer size here (multiplied LEGION_MAX_RETURN_SIZE by 4)