Converting a pipeline

[MatthewWaller]

What would it take to convert an entire pipeline to a coreml model?

For instance, I have saved the stable-diffusion checkpoint, and several of the models have their own configs, but of course they're not the ready-made configs.

Would this be just a long, hard, custom slog via exporters and not worth it? Or is there something here worth pursuing?

[MatthewWaller]

For folks interested, I added a comment to this gist that has a bunch of the models converted. 3 out of 4!

One bit I'm trying to convert is CLIP. Since the model uses the CLIPTextModel, and the CLIPTextEncoder.

Thoughts about how I might go about this? I get a Torch var mask.3 not found in context and a whole bunch of warnings that a tensor is getting converted to a boolean.

And then I'm also not sure how I could convert the scheduler. Or if I can ignore that.

https://gist.github.com/madebyollin/86b9596ffa4ab0fa7674a16ca2aeab3d

[MatthewWaller]

Ha! The original author of the post was gracious enough to fix up that CLIPTextEncoder. Nice.

One thing I'll need to tackle implementing to start experiment running on iPhone and iPad and Mac app is the tokenizer.

I've opened an issue here if folks know the answer: https://github.com/huggingface/swift-coreml-transformers/issues/28

[rovo79]

@MatthewWaller Did you ever get anywhere with this pipeline conversion?

[MatthewWaller]

@MatthewWaller Did you ever get anywhere with this pipeline conversion?

@rovo79 i eventually converted a stable diffusion pipeline, but it was just a long hard slog, and there is now an entire Apple repo on converting stable diffusion and other libraries. So much happened in a year! So I don’t think this issue needs addressing anymore