Help with reading and writing images

[ghost]

Dear team, After integrating Siv3D with Libtorch (https://github.com/QuantScientist/Siv3DTorch) I am now trying to read and write images from and to Siv3D. The way it works is:

1. An image is read from disk (usually using OpenCV which is easy but I am trying to avoid)
2. The image is converted to torch::tensor
3. A DL model is run on the tensor
4. A tensor is returned from the model
5. The tensor is converted to an image for display purposes.

This is one example where they used stb_image to this, avoiding the use of OpenCV. https://github.com/prabhuomkar/pytorch-cpp/blob/master/utils/image_io/src/image_io.cpp

I did something similar and was able to read the image, but I am not sure how to proceed and display it on Siv3D. I don't mind using libpng / libjpg that you are linking to if you think I should do so.

Code for reading an image and displaying its dimensions on Siv3D (https://github.com/QuantScientist/Siv3DTorch/blob/master/src/loadmodel003.cpp):

# include <Siv3D.hpp>
#include <torch/script.h>
#include <torch/torch.h>
#include <vector>
#include <typeinfo> 
#include <thread>
#include <future>
#define STB_IMAGE_IMPLEMENTATION
#include "../include/stb_image/stb_image.h"

#define STB_IMAGE_WRITE_IMPLEMENTATION
#include "../include/stb_image_write/stb_image_write.h"

#define STB_IMAGE_RESIZE_IMPLEMENTATION
#include "../include/stb_image_resize/stb_image_resize.h"

torch::Device device(torch::kCUDA);
torch::Tensor tensor = torch::eye(3).to(device);
torch::data::transforms::Normalize<> normalize_transform({ 0.485, 0.456, 0.406 }, { 0.229, 0.224, 0.225 });

// Loads a tensor from an image file
torch::Tensor load_image(const std::string& file_path,
    torch::IntArrayRef shape, std::function<torch::Tensor(torch::Tensor)> transform) {
    if (!shape.empty() && shape.size() != 1 && shape.size() != 2) {
        throw std::invalid_argument("Shape must be empty or contain exactly one or two elements.");
    }

    int width = 0;
    int height = 0;
    int depth = 0;

    std::unique_ptr<unsigned char, decltype(&stbi_image_free)> image_raw(stbi_load(file_path.c_str(),
        &width, &height, &depth, 0), &stbi_image_free);

    if (!image_raw) {
        throw std::runtime_error("Unable to load image file " + file_path + ".");
    }

    if (shape.empty()) {
        return transform(torch::from_blob(image_raw.get(),
            { height, width, depth }, torch::kUInt8).clone().to(torch::kFloat32).permute({ 2, 0, 1 }).div_(255));
    }

    int new_width = 0;
    int new_height = 0;

    if (shape.size() == 1) {
        double scale = static_cast<double>(shape[0]) / std::max(width, height);
        new_width = width * scale;
        new_height = height * scale;
    }
    else {
        new_width = shape[1];
        new_height = shape[0];
    }

    if (new_width < 0 || new_height < 0) {
        throw std::invalid_argument("Invalid shape.");
    }

    size_t buffer_size = new_width * new_height * depth;

    std::vector<unsigned char> image_resized_buffer(buffer_size);

    stbir_resize_uint8(image_raw.get(), width, height, 0,
        image_resized_buffer.data(), new_width, new_height, 0, depth);

    return transform(torch::from_blob(image_resized_buffer.data(),
        { new_height, new_width, depth }, torch::kUInt8).clone().to(torch::kFloat32).permute({ 2, 0, 1 }).div_(255));
}

void tensorDIMS(const torch::Tensor& tensor) {
    auto t0 = tensor.size(0);
    auto s = tensor.sizes();
    Print (tensor.size(0), U",", tensor.size(1), U",", tensor.size(2));
    //Print(tensor.size(0));
}

void Main()
{
    Window::SetTitle(U"TorchSiv3D C++");
    const Texture icn0(Emoji(U"✡"));
    icn0.draw(0, 0);                
    Scene::SetBackground(Color(90, 81, 95));    

    const std::string modelName = "erfnet_fs.pt";
    const std::string content_image_path = "windmill.png";

    auto module = torch::jit::load(modelName, device);
    //module->to(at::kCUDA);
    if (!std::ifstream(modelName)) {

        Print  (U"ERROR: Could not open the required module file from path:");
    }
    else {
        Print(U"Loaded required module file from path");
    }
    assert(module != nullptr);
    const int64_t max_image_size = 256;
    auto content = load_image(content_image_path, max_image_size, normalize_transform).unsqueeze_(0);

    tensorDIMS(content);
    //auto x = (content).data()[0]; // Move it to the CPU
    //Print(x); //Use it from Siv3D
    while (System::Update())
    {   

    }
}

For reference this is the OpenCV to Libtorch conversion utils:

at::Tensor matToTensor(cv::Mat frame, int h, int w, int c) {
    cv::cvtColor(frame, frame, CV_BGR2RGB);
    frame.convertTo(frame, CV_32FC3, 1.0f / 255.0f);
    auto input_tensor = torch::from_blob(frame.data, {1, h, w, c});
    input_tensor = input_tensor.permute({0, 3, 1, 2});

    torch::DeviceType device_type = torch::kCPU;
//    if (torch::cuda::is_available()) {
    device_type = torch::kCUDA;
//    }
    input_tensor = input_tensor.to(device_type);
    return input_tensor;
}

cv::Mat tensorToOpenCv(at::Tensor out_tensor, int h, int w, int c) {
    out_tensor = out_tensor.squeeze().detach().permute({1, 2, 0});
    out_tensor = out_tensor.mul(255).clamp(0, 255).to(torch::kU8);
    out_tensor = out_tensor.to(torch::kCPU);
    cv::Mat resultImg(h, w, CV_8UC3);
    // cv::Mat resultImg(h, w, CV_8UC1);
    std::memcpy((void *) resultImg.data, out_tensor.data_ptr(), sizeof(torch::kU8) * out_tensor.numel());
    return resultImg;
}

Many thanks,

[ghost]

Done! Working with VC 19 and not CMake is really hard :)

Code: https://github.com/QuantScientist/Siv3DTorch/blob/master/src/readpng004.cpp

[Reputeless]

You can use s3d::Image class for image I/O and image processing.

[ghost]

For now, I am converting the tensor to a PNG and writing the PNG to disk and then reading it with siv3d. It is not very fast bu that Is what I have now. I looked at s3d::Image but spent a day trying to figure out how to do s3d::Image to torch::tensor and vice versa. This is my torch to PNG and vice versa code: https://github.com/QuantScientist/PngTorch/blob/master/include/utils/vision_utils.hpp

Thanks,

[Reputeless]

Conversion functions will be like this:

torch::Tensor ImageToTensor(const Image& image)
{
    Array<uint8> buffer(image.num_pixels() * 3);
    uint8* pDst = buffer.data();

    for (const auto& pixel : image)
    {
        *pDst++ = pixel.r;
        *pDst++ = pixel.g;
        *pDst++ = pixel.b;
    }

    const int32 width  = image.width();
    const int32 height = image.height();

    // ...
}

Image TensorToImage(const torch::Tensor& tensor)
{
    size_t width  = ??;
    size_t height = ??;
    const uint8* pSrc = tensor.data_ptr<uint8>();

    Image image(width, height);

    for (auto& pixel : image)
    {
        pixel.r = *pSrc++;
        pixel.g = *pSrc++;
        pixel.b = *pSrc++;
        pixel.a = 255;
    }

    return image;
}