AnimeGANv2 to CoreML

[ManluZhang]

Hi, thanks for the great work!

I'm trying to convert the AnimeGANv2 Shinkai.pb model to CoreML, and I've got some questions.

1. Did you use the check points to generate a .pb file first, maybe using this script then convert it to coreml using coremltools?
2. In coremltools, should I use ct.ImageType(shape=(1, 256, 256, 3)) as input format of the converter?

Here's my converter code:

image_inputs = ct.ImageType(shape=(1, 256, 256, 3))
mlmodel = ct.convert('./Shinkai_53.pb', inputs=[image_inputs],  source='tensorflow')
mlmodel.save('./output.mlmodel')

And then I use:

import coremltools.proto.FeatureTypes_pb2 as ft
spec = ct.utils.load_spec("output.mlmodel")
output = spec.description.output[0]
output.type.imageType.colorSpace = ft.ImageFeatureType.RGB
output.type.imageType.height = 256
output.type.imageType.width = 256
ct.utils.save_spec(spec, "new.mlmodel")

to convert the output format to Image. After I drag the new mlmodel to Xcode, and preview the model with an image, the stylized image can't be generated. It seems to be loading forever.

Do u have a cue on what's going wrong? Or could u please tell me how u convert the AnimeGANv2 models? Again, thanks for your great work. Really appreciate it.

[john-rocky]

Hi. You need to permute output shape.

2022年2月15日(火) 20:34 Manlu Zhang @.***>:

Hi, thanks for the great work!

I'm trying to convert the AnimeGANv2 Shinkai.pb model https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/Shinkai_53.pb to CoreML, and I've got some questions.

1. Did you use the check points to generate a .pb file first, maybe using this script https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/animegan2pb.py then convert it to coreml using coremltools?
2. In coremltools, should I use ct.ImageType(shape=(1, 256, 256, 3)) as input format of the converter?

Here's my converter code: image_inputs = ct.ImageType(shape=(1, 256, 256, 3)) mlmodel = ct.convert('./Shinkai_53.pb', inputs=[image_inputs], source='tensorflow') mlmodel.save('./output.mlmodel') And then I use: import coremltools.proto.FeatureTypes_pb2 as ft spec = ct.utils.load_spec("output.mlmodel") output = spec.description.output[0] output.type.imageType.colorSpace = ft.ImageFeatureType.RGB output.type.imageType.height = 256 output.type.imageType.width = 256 ct.utils.save_spec(spec, "new.mlmodel") to convert the output format to Image. After I drag the new mlmodel to Xcode, and preview the model with an image, the stylized image can't be generated. It seems to be loading forever. Do u have a cue on what's going wrong? Or could u please tell me how u convert the AnimeGANv2 models?

Again, thanks for your great work. Really appreciate it.

— Reply to this email directly, view it on GitHub https://github.com/john-rocky/CoreML-Models/issues/11, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRTLEDX3BNN5GRTVRERO3DU3I25NANCNFSM5OOH3SSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ManluZhang]

Hi. You need to permute output shape. 2022年2月15日(火) 20:34 Manlu Zhang @.>: … Hi, thanks for the great work! I'm trying to convert the AnimeGANv2 Shinkai.pb model https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/Shinkai_53.pb to CoreML, and I've got some questions. 1. Did you use the check points to generate a .pb file first, maybe using this script https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/animegan2pb.py then convert it to coreml using coremltools? 2. In coremltools, should I use ct.ImageType(shape=(1, 256, 256, 3)) as input format of the converter? Here's my converter code: image_inputs = ct.ImageType(shape=(1, 256, 256, 3)) mlmodel = ct.convert('./Shinkai_53.pb', inputs=[image_inputs], source='tensorflow') mlmodel.save('./output.mlmodel') And then I use: import coremltools.proto.FeatureTypes_pb2 as ft spec = ct.utils.load_spec("output.mlmodel") output = spec.description.output[0] output.type.imageType.colorSpace = ft.ImageFeatureType.RGB output.type.imageType.height = 256 output.type.imageType.width = 256 ct.utils.save_spec(spec, "new.mlmodel") to convert the output format to Image. After I drag the new mlmodel to Xcode, and preview the model with an image, the stylized image can't be generated. It seems to be loading forever. Do u have a cue on what's going wrong? Or could u please tell me how u convert the AnimeGANv2 models? Again, thanks for your great work. Really appreciate it. — Reply to this email directly, view it on GitHub <#11>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRTLEDX3BNN5GRTVRERO3DU3I25NANCNFSM5OOH3SSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.>

Hi, Thanks for the reply. I'm actually new to machine learning. How do I permute the output shapes? Do you mean using the add_ permute function in coremltools?

[john-rocky]

Yes 2022年2月16日水曜日 Manlu Zhang @.***>:

Hi. You need to permute output shape. 2022年2月15日(火) 20:34 Manlu Zhang @.

>: … <#m6073560659558634855> Hi, thanks for the great work! I'm trying to convert the AnimeGANv2 Shinkai.pb model https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/Shinkai_53.pb https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/Shinkai_53.pb to CoreML, and I've got some questions. 1. Did you use the check points to generate a .pb file first, maybe using this script https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/animegan2pb.py https://github.com/TachibanaYoshino/AnimeGANv2/blob/master/pb_and_onnx_model/animegan2pb.py then convert it to coreml using coremltools? 2. In coremltools, should I use ct.ImageType(shape=(1, 256, 256, 3)) as input format of the converter? Here's my converter code: image_inputs = ct.ImageType(shape=(1, 256, 256, 3)) mlmodel = ct.convert('./Shinkai_53.pb', inputs=[image_inputs], source='tensorflow') mlmodel.save('./output.mlmodel') And then I use: import coremltools.proto.FeatureTypes_pb2 as ft spec = ct.utils.load_spec("output.mlmodel") output = spec.description.output[0] output.type.imageType.colorSpace = ft.ImageFeatureType.RGB output.type.imageType.height = 256 output.type.imageType.width = 256 ct.utils.save_spec(spec, "new.mlmodel") to convert the output format to Image. After I drag the new mlmodel to Xcode, and preview the model with an image, the stylized image can't be generated. It seems to be loading forever. Do u have a cue on what's going wrong? Or could u please tell me how u convert the AnimeGANv2 models? Again, thanks for your great work. Really appreciate it. — Reply to this email directly, view it on GitHub <#11 https://github.com/john-rocky/CoreML-Models/issues/11>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRTLEDX3BNN5GRTVRERO3DU3I25NANCNFSM5OOH3SSQ https://github.com/notifications/unsubscribe-auth/AFRTLEDX3BNN5GRTVRERO3DU3I25NANCNFSM5OOH3SSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.>

Hi, Thanks for the reply. I'm actually new to machine learning. How do I permute the output shapes? Do you mean using the add_ permute function in coremltools?

— Reply to this email directly, view it on GitHub https://github.com/john-rocky/CoreML-Models/issues/11#issuecomment-1041054689, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFRTLECHCQLWJIMMAN4TS33U3MJVFANCNFSM5OOH3SSQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

[john-rocky]

Try this

spec = mlmodel.get_spec()
builder = ct.models.neural_network.NeuralNetworkBuilder(spec=spec)
builder.spec.description

builder.add_permute(name="permute",dim=[0,3,1,2], input_name={output name in printed description}, output_name="permute_out")
builder.add_squeeze(name="squeeze", input_name="permute_out", output_name="squeeze_out", axes = None, squeeze_all = True)
builder.add_activation(name="activation",non_linearity="LINEAR",input_name="squeeze_out",output_name="image",params=[127.5,127.5])
from coremltools.proto import FeatureTypes_pb2 as ft
builder.spec.description.output.pop()
builder.spec.description.output.add()
output = builder.spec.description.output[0]
output.name = "image"

output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('RGB')
output.type.imageType.width = 256 
output.type.imageType.height = 256

[ManluZhang]

It works perfectly! Thanks a lot!!!

[john-rocky]

Happy machine learning🐥

[ioskevinshah]

@ManluZhang , can you please provide the CoreML model.

[ManluZhang]

@ManluZhang , can you please provide the CoreML model.

Yes, here.

[ManluZhang]

Hi, @john-rocky I've converted the animeGANv2 Paprika checkpoints to coreML, and compared with the model that you provided in this repo. The output of the same image looks quite different. Left is from your model, and the right one is from mine. I also found that, the tensorflow inference of the animeGANv2 is very different after I converted it to coreML. Any ideas why ?

My converting code:

    mlmodel = ct.convert(model=path, inputs=[ct.ImageType(shape=(1, 256, 256, 3))],
                         source='tensorflow')
    mlmodel.save(name)
    spec = ct.utils.load_spec(name)
    builder = ct.models.neural_network.NeuralNetworkBuilder(spec=spec)
    print(builder.spec.description)

    builder.add_permute(name="permute", dim=[0, 3, 1, 2], input_name="generator_G_MODEL_out_layer_Tanh", output_name = "permute_out")
    builder.add_squeeze(name="squeeze", input_name="permute_out", output_name="squeeze_out", axes=None,
                        squeeze_all=True)
    builder.add_activation(name="activation", non_linearity="LINEAR", input_name="squeeze_out", output_name="image",
                           params=[127.5, 127.5])

    builder.spec.description.output.pop()
    builder.spec.description.output.add()

    output = builder.spec.description.output[0]
    output.name = "image"
    output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('RGB')
    output.type.imageType.width = 256
    output.type.imageType.height = 256

    ct.utils.save_spec(builder.spec, 'output.mlmodel')

[john-rocky]

I think Activation is too high.

Try to change activation param. Like [255,0]

[john-rocky]

Activation params are changed on the range of the output of the model. If model has -1~1 output, We need 125 +127.5. If 0 ~ 1, we need 255. I forgot AnimeGAN output range. So, please try...

[ManluZhang]

I've already tried [255,0], looks not same either.

[john-rocky]

[127.5,0]?

[ManluZhang]

let me try

[ManluZhang]

too dark maybe I should figure out the AnimeGAN output range first do you know how to see the output range?

[john-rocky]

Wait, I missed your conversion script. I think you need add preprocessing input.

like

mlmodel = ct.convert('./frozen_model.pb',
                     inputs=[ct.ImageType(shape=[1,256,256,3],bias=[-1,-1,-1], scale=1/127)])

Then activate with [127.5,127.5]

About preprocessing, please see my article.

https://rockyshikoku.medium.com/generate-images-using-gan-on-ios-devices-1188e01283d2

[ManluZhang]

OK, thanks a lot !!!