search

pytorch / nestedtensor

[Prototype] Tools for the concurrent manipulation of variably sized Tensors.
BSD 3-Clause "New" or "Revised" License
252 stars 28 forks source link

Faster padding - fuse fill, reduce overhead #419

Closed cpuhrsch closed 3 years ago

cpuhrsch commented 3 years ago

Master

model_name:   resnext101_32x4d, bsz:  16, mean±std shapes[2]: 256.75±135.80, mean±std shapes[3]: 370.56±148.10, padded_size: (16, 3, 561, 563), loop:  364.19ms, nt:   98.25ms, padded:  113.14ms, speedup: 3.71x
model_name:   resnext101_32x4d, bsz:  32, mean±std shapes[2]: 304.19±155.05, mean±std shapes[3]: 330.72±145.72, padded_size: (32, 3, 566, 564), loop:  740.61ms, nt:  157.64ms, padded:  199.59ms, speedup: 4.70x
model_name:   resnext101_32x4d, bsz:  64, mean±std shapes[2]: 323.17±145.32, mean±std shapes[3]: 363.86±141.50, padded_size: (64, 3, 567, 591), loop: 1481.89ms, nt:  324.53ms, padded:  410.88ms, speedup: 4.57x
model_name:   resnext101_32x4d, bsz: 128, mean±std shapes[2]: 319.36±143.50, mean±std shapes[3]: 361.93±141.42, padded_size: (128, 3, 593, 599), loop: 2969.81ms, nt:  663.22ms, padded:  899.91ms, speedup: 4.48x

Branch

model_name:   resnext101_32x4d, bsz:  16, mean±std shapes[2]: 256.75±135.80, mean±std shapes[3]: 370.56±148.10, padded_size: (16, 3, 561, 563), loop:  370.88ms, nt:   94.31ms, padded:  113.61ms, speedup: 3.93x
model_name:   resnext101_32x4d, bsz:  32, mean±std shapes[2]: 304.19±155.05, mean±std shapes[3]: 330.72±145.72, padded_size: (32, 3, 566, 564), loop:  756.39ms, nt:  152.60ms, padded:  199.53ms, speedup: 4.96x
model_name:   resnext101_32x4d, bsz:  64, mean±std shapes[2]: 323.17±145.32, mean±std shapes[3]: 363.86±141.50, padded_size: (64, 3, 567, 591), loop: 1518.46ms, nt:  310.15ms, padded:  410.08ms, speedup: 4.90x
model_name:   resnext101_32x4d, bsz: 128, mean±std shapes[2]: 319.36±143.50, mean±std shapes[3]: 361.93±141.42, padded_size: (128, 3, 593, 599), loop: 3042.62ms, nt:  629.73ms, padded:  899.47ms, speedup: 4.83x