Porting First-Order-Model to iOS

[Ease78]

I need some Architecture help to make First Order Motion for iPhone.

1. Make an asynchronous function call that sends a video + picture to the cloud, process the video, then return trained video

2. OR convert the pth.tar model** to ONNX then into .mlmodel and then follow this tutorial https://www.raywenderlich.com/7960296-core-ml-and-vision-tutorial-on-device-training

**There's no direct conversion tool, and I think most people would be apprehensive about sharing their data on the cloud?

Any high-level help is welcomed here.

[col-in-coding]

I need some Architecture help to make First Order Motion for iPhone.

1. Make an asynchronous function call that sends a video + picture to the cloud, process the video, then return trained video
2. OR convert the pth.tar model** to ONNX then into .mlmodel and then follow this tutorial https://www.raywenderlich.com/7960296-core-ml-and-vision-tutorial-on-device-training

**There's no direct conversion tool, and I think most people would be apprehensive about sharing their data on the cloud?

Any high-level help is welcomed here.

I've been trying to convert it to ONNX but failed, the grid_sampler is not supported. Do you have more ideas

[HashedViking]

@AliaksandrSiarohin @Ease78 hello, is it possible to trace model execution? where to put: traced_model = torch.jit.trace(torch_model, example_input) as suggested here https://coremltools.readme.io/docs/pytorch-conversion getting traced_model would heavily help porting to iOS CoreML

[AliaksandrSiarohin]

@HashedViking Never did this, but since there is no runtime branching I guess it should be possible.

[HashedViking]

Well, generating .pt scripts is pretty straightforward, just add this inside make_animation function at demo.py

traced_detector = torch.jit.trace(kp_detector.module, source, strict=False)
traced_generator = torch.jit.trace(generator.module, (source, kp_norm, kp_source), strict=False)
traced_detector.save('kp_detector.pt')
traced_generator.save('generator.pt')

Next comes the hard part: generate .mlmodel from traced models.

import coremltools as ct
                model = ct.convert(
                    traced_detector,
                    inputs=[ct.TensorType(name="source", shape=source.shape)]
                )
                model.save("kp_detector.mlmodel") <<-- error here
                print(model.get_spec())

"PyTorch convert function for op '{}' not implemented.".format(node.kind) RuntimeError: PyTorch convert function for op 'unsqueeze_' not implemented.

That got me blocked, if someone have any idea how to fix it, please, share.

Also, to get a single model out of the OcclusionAwareGenerator and KPDetector we have to use coremltools's pipeline

[ptmdmusique]

RuntimeError: PyTorch convert function for op 'unsqueeze_' not implemented.

@HashedViking 👋 have you solved the issue or could you share any work-around you found ❓

[gasparitiago]

@HashedViking @Ease78 any news regarding that?