Closed DeepTecher closed 2 months ago
it looks like server oom and didn't return result. The default implementation is the original implementation, without introducing additional memory optimization methods.
tp、kvcache、seperate schedule already realized in latest commit.
We run on A100-40G to get all test with the below configuration:
however, when it run on bs=8, input_len=2048, it raise error :