Closed albusdemens closed 3 years ago
Hi!
Longformer and Sparse Transformer require specialized cuda kernels to get the speed up/memory gain. This does not play well with hardware such as TPUs.
In our experiments, we benchmarked the quality (accuracy) of Longformer/Sparse Transformer by simulating the sparsity (which does not result in memory or speed gains). Hence, it does not make sense to include Longformer and Sparse Transformer in the plot.
Thanks.
Hello, is there a reason why you didn't include the Longformer in Fig. 3 of your paper? Cheers