animegan on video

[spikejones1040]

Area Select

react-native-pytorch-core (core package)

Description

I want to apply Animegan model using Playtorch on video, since I am developing React Native Application. However -

• There is an example in demos of playtorch ( https://playtorch.dev/docs/tutorials/snacks/anime-gan/ ) that does inference on an image. It takes about 3-4 seconds on a single image
• AnimeGAN model architecture works real-time with CoreML on iOS as well as TFlite on Android
• With CoreML, it uses NPU and with TFLite it uses GPU of the device to get faster inference speeds
• Is it possible to use AnimeGAN in Playtorch and use HW acceleration (either GPU or NPU)?
• Does anyone have any pointers where someone has tried Playtorch for style-transfer or UNet style models on video for real-time performance?

Any help/pointers will be greatly appreciated.

[raedle]

Hi @spikejones1040, the short answer is PlayTorch doesn't support ML accelerators, and is limited by the options of the PyTorch Mobile lite interpreter runtime. The longer answer is explained in https://github.com/facebookresearch/playtorch/issues/81#issuecomment-1197660296

[spikejones1040]

Thanks for pointer and explanation. Is the support for accelerators planned in future releases ? as there are tons of use cases for videos (style transfer, super resolution, background removal etc. ) It will make playtorch very powerful. Currently with image only support it is a bit limited.

[raedle]

There are no plans to add support. But we are accepting PRs if anyone wants to take a stab