My main question is can I just use accelerated decoding functions and ops from ffcv and not worry about integrating a custom data format like beton or use a custom data loader?
More context
Preprocessing performance matters for my problem: In some early experiments folks like @min-jean-cho have found large 3x end-to-end speeds in optimizing our image pre-processing by using accimage but we paused that work because it wasn't clear how maintained accimage was https://github.com/pytorch/serve/pull/1545
Expensive data transforms don't make sense for low latency inference: Lots of my customers use batch size = 1 because they really care about latency, if the batch size is really large I can imagine that beton may make sense for offline inference but I need to run more benchmarks to be sure
A custom data loader limits integrations with serving frameworks: We have our loader with its own optimizations and while we could in principle support another one with more work, if I can instead work with a function like fast_image_preprocessing(mytensor : torch.Tensor) -> torch.Tensor then I be done with a prototype in 5 min
TL;DR
Hi! I work on https://github.com/pytorch/serve and I was interested in exploring optimizing preprocessing for inference.
My main question is can I just use accelerated decoding functions and ops from
ffcvand not worry about integrating a custom data format likebetonor use a custom data loader?More context
accimagebut we paused that work because it wasn't clear how maintainedaccimagewas https://github.com/pytorch/serve/pull/1545batch size = 1because they really care about latency, if the batch size is really large I can imagine thatbetonmay make sense for offline inference but I need to run more benchmarks to be surefast_image_preprocessing(mytensor : torch.Tensor) -> torch.Tensorthen I be done with a prototype in 5 min