model spec for image2image models

[Sxela]

Area Select

react-native-pytorch-core (core package)

Description

What should I do to make a correct spec for image2image model? (unet, for example) The "pack" spec part seems to be okay, as for "unpack" - I can't find anything that would fit my use case in your model spec API here - https://pytorch.org/live/docs/api/model-spec/ I need to convert a predicted tensor back to an image with reverse normalization.

[Sxela]

I found it in the code as plain and simple "tensor_to_image", I guess it should be added to the spec API docs.

[chrisklaiber]

@Sxela yes, you have found the correct unpacker! However, note that it is currently undocumented and the behavior may change in a future release. Exposing this image unpacker is on our roadmap.

One gotcha to be aware of: the current reverse normalization dynamically determines the range of values in the data. If your range is <1 it will cause a divide by zero, which results in a float value of -Infinity or Infinity on Android for all pixels, resulting in an all-white or all-black image. A hacky workaround to control the range is to ensure that your model sets one pixel to the minimum desired value and one to the maximum value. A reasonable solution might be for the unpacker to allow specification of the range in JSON, and falling back to the current dynamic range detection. If that is something you would be interested in tackling, let us know, your PR would be welcome!

[Sxela]

I suppose you can just use normalization stats from the packer settings, and fall back to autodetection if none were provided.

[mitsunami]

I also would like to use image2image models but I'm not sure how I can "tensor_to_image" unpacker because the following command seems to install the stable version of pytorch-live, which does not include "tensor_to_image" unpacker: npx react-native init MyProject --template react-native-template-pytorch-live

How can I specify to use the pytorch-live version with the unpacker? I'm using it on Windows for Android.

[raedle]

@Sxela and @mitsunami, the react-native-pytorch-core release candidate v0.2.0-rc.2 has a PyTorch-inspired API to support a more flexible data processing. For example, have a look at the useAnimeGANv2.ts for an example how image to image works with this API.

More details on available ops are is in the API docs:

• https://pytorch.org/live/docs/next/api/core/interfaces/torchlive_torch.torch/
• https://pytorch.org/live/docs/next/api/core/interfaces/torchlive_torch.tensor/

Feel free to reach out on Discord or here if any ops are missing for your use cases!

[chrisklaiber]

Closing this issue out since the packer / unpacker approach was removed with 0.2.2. Check out the PyTorch-inspired API in the README as an alternative that avoids these issues: https://github.com/facebookresearch/playtorch/tree/main/react-native-pytorch-core#example-usage