[Feature Request] Ability to read an image from memory

[spiffmon]

Following up on a conversation with @lgritz , I'd like to request the other direction of what #1895 is requesting.

The Need

In USD and Hydra, we’ve been relying on OIIO for marquee image services, and it’s far and away the most compelling choice. However, recently we’ve grown a need to be able to decode formats (USD, images, volumes) from in-memory buffers, which may have been mmapped from files or created or transmitted. There are two different scenarios: 1) We’re providing an archive format that, in a single (uncompressed zip) file, contains USD files, textures, and anything else a scene requires. To be able to consume the scene without unpacking it, we want to access data out of it by mmapping with offsets into the zip file. 2) People can create “asset resolvers” that deliver data to USD directly over the network, from dbs, etc, and would like to void the step of writing that data out to disk.

So...

We'd like to be able to hand OIIO a buffer-pointer, plus length, plus "identifier" that corresponds to a filename and has an extension that indicate format; that buffer represents an actual serialized image. When we're using OIIO as primarily an image reader, the problem is reasonably straightforward - as Larry suggested, we should be able to initialize an ImageInput from these inputs. This is sufficient for our more immediate needs.

The more involved case is when a client wants to use OIIO as a texture manager, using ImageCache/TextureSystem . In that case, we need to acknowledge that our buffer is a resource that needs to be managed, so we will need a way to allow OIIO to do that. On the USD-side, we'll have a resource object whose lifetime manages the buffer's lifetime, but we'd need a way to adapt that for OIIO's consumption.

Thanks!

[lgritz]

It's actually a great coincidence that you posted this, because this is the very problem I was fooling around with over the weekend.

I actually am re-debating whether the right approach is the simple pointer-and-length (which works fine specifically for memory), or if the right idea is to make an abstract "IOProxy" with virtual methods for seek, read, etc., and it can be subclassed for wrapping file access, memory access, whatever.

I'm kinda leaning toward this more general approach at the moment, but I'm on the fence, I honestly don't know if it's better to do the fully general thing, or if it feels over-engineered. I will try to finish mocking it up today and post a design review PR to see what you think. Maybe if we see it in action for one plugin, it will be clear if it's the right approach.

[spiffmon]

One potential advantage of the IOProxt approach is that it could easily provide a solution for the resource management problem as well, so two birds, one stone.

On Sun, Apr 29, 2018 at 5:05 PM Larry Gritz notifications@github.com wrote:

It's actually a great coincidence that you posted this, because this is the very problem I was fooling around with over the weekend.

I actually am re-debating whether the right approach is the simple pointer-and-length (which works fine specifically for memory), or if the right idea is to make an abstract "IOProxy" with virtual methods for seek, read, etc., and it can be subclassed for wrapping file access, memory access, whatever.

I'm kinda leaning toward this more general approach at the moment, but I'm on the fence, I honestly don't know if it's better to do the fully general thing, or if it feels over-engineered. I will try to finish mocking it up today and post a design review PR to see what you think. Maybe if we see it in action for one plugin, it will be clear if it's the right approach.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenImageIO/oiio/issues/1930#issuecomment-385292445, or mute the thread https://github.com/notifications/unsubscribe-auth/AF7qaHAgGuKISc3Ft8eg8QiD8s11-Z1iks5ttlVigaJpZM4TsADP .

-- --spiffiPhone

[lgritz]

This is looking good. Hope to send PR tomorrow.

[lgritz]

Proposed fix #1931

[lgritz]

Closing. This was implemented in #1931, I'm not sure why this issue didn't close when that was merged.

[spiffmon]

Thanks, Larry! Working on the timing for us to upgrade to 1.8.x so we can start leveraging this in our distro. --spiff

On Tue, Nov 20, 2018 at 5:48 PM Larry Gritz notifications@github.com wrote:

Closed #1930 https://github.com/OpenImageIO/oiio/issues/1930.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenImageIO/oiio/issues/1930#event-1978911802, or mute the thread https://github.com/notifications/unsubscribe-auth/AF7qaCUAKcsPSeplVQYspuSl_U1ppV5Eks5uxLEAgaJpZM4TsADP .

[lgritz]

Skip 1.8 (which I think doesn't have this feature backported).

2.0 is in beta with release in a couple weeks.

[egorodet]

Hello @lgritz,

I'm trying to use OpenImageIO 2.0.5 to read JPEG image from memory buffer, instead of file directly, but something goes wrong and result ImageBuf has wrong ImageSpec (w=1, h=1, d=1). I've written my code based on image_buf_test.cpp, test_write_exr_to_memory(), since I did not find any documentation about this feature other than this issue. Please take a look at my code:

std::ifstream fs(file_path, std::ios::binary);
if (!fs.good())
    throw std::invalid_argument("Resource file path does not exist: " + file_path);
std::vector<unsigned char> image_data(std::istreambuf_iterator<char>(fs), {});

OIIO::Filesystem::IOMemReader image_reader(image_data);
OIIO::ImageSpec init_spec;
init_spec.attribute("oiio:ioproxy", OIIO::TypeDesc::PTR, &image_reader);
OIIO::ImageBuf image_buf(init_spec);
const OIIO::ImageSpec& image_spec = image_buf.spec();
if (image_spec.undefined() || !image_buf.read())
    throw std::runtime_error("Failed to read image from memory, error: " + image_buf.geterror());

OIIO::ROI image_roi = OIIO::get_roi(image_spec);
const size_t texture_channels_count = 4;
std::vector<uint8_t> texture_data(texture_channels_count * image_roi.npixels(), 255);
const OIIO::TypeDesc texture_format(OIIO::TypeDesc::BASETYPE::UCHAR);
if (!image_buf.get_pixels(image_roi, texture_format, texture_data.data(), texture_channels_count * sizeof(texture_data[0])))
    throw std::runtime_error("Failed to decode image from memory, error: " + image_buf.geterror());

When I try to load JPEG image from memory (as shown above) image_spec is wrong (w=1, h=1, d=1) and get_pixels fails because it tries to decode image to 4 channels. But when I load the same JPEG directly from file i.e. OIIO::ImageBuf image_buf(file_path), everything works fine. Could you please tell me what I'm doing wrong?

[lgritz]

@egorodet Sorry, so far support for this has only been added to OpenEXR and PNG. I just looked over the JPEG code, I think it should be straightforward for the memory case (perhaps less so for the general IOProxy, but memory is the most important one). Let me give it a stab and see if it's something I can do quickly.

In the mean time, it's worth noting that you ought to be able to test a specific format using ImageInput::supports("ioproxy") to see if it supports this or not.

[lgritz]

Well, sheesh, after I wrote that I looked and I did not in fact ensure that supports() reports the right thing. So I need to fix that, too.

[lgritz]

PR #2180 proposes a patch that adds memory read support to jpeg.

[lukas-sl]

Hello @lgritz, I'm currently stuck trying to implement a JPEG-2000 encoder. I have a 16 bit grayscale image (uncompressed) that is already in memory as input and need to write the JPEG-2000 encoded grayscale image to memory as well. Could you give me a hint how to solve this issue?

[spiffmon]

Just wanted to post a note of thanks - we updated USD's imaging system to OIIO 2.x just prior to siggraph this year, and enhanced it to use IOProxy, which means that you can now have exr textures in usdz archives! It'll be a while (if ever) before iOS/AR-Quicklook supports it, but for studios, it's pretty huge. Thanks, Larry!

[lgritz]

Thanks for letting me know, @spiffmon !

I have some further changes coming soon to make IO proxies a bit more of a first class idea. That won't really hit until 2.2 (2.1 is just released, so it will be a while, but it will also be compatible with the way you've been doing it).

[antoche]

Hi, are there python bindings to achieve the same thing in python?

In my case, I have a zip file containing images that I am opening in python. I can get the file contents for each image as a byte buffer using the zipfile module, but I can't see any way to pass that buffer for oiio to interpret. I could even use io.BytesIO to present the buffer as a file, but oiio seems to only take filenames as input, not buffers or file objects.

I'm essentially looking for something like cv2.imdecode, which would return an ImageBuf rather than a numpy array.

[aweinmann]

@antoche did you find a solution to your problem? I'm trying to do the same thing

[antoche]

Nope. I'm still using opencv in that codepath, which is annoying because it's a different colour pipeline from the rest.

On Thu, 27 Aug 2020, 19:50 aweinmann, notifications@github.com wrote:

@antoche https://github.com/antoche did you find a solution to your problem? I'm trying to do the same thing

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenImageIO/oiio/issues/1930#issuecomment-681720213, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMNJ6BA4JYZECRISTHAAGTSCYF27ANCNFSM4E5QADHQ .

[lgritz]

Do you need a fully general IOProxy-like thing in Python? Or is it sufficient to just have a direct way to pass a memory buffer, that doesn't necessarily have an exact analog in C++ (though it would be implemented with an IOMemReader underneath)?

[aweinmann]

For me it would be sufficient to be able to create an oiio.ImageBuf from a memory buffer

[antoche]

I don't think having an IOProxy wrapper would be very pythonic. Python already has a concept of file-like objects as well as memory buffers, and already provides conveniences for passing one as the other (either via io.BytesIO or mmap.mmap). So I think it would be sufficient to provide an interface which takes a memory buffer as input, and adding support for file-like objects can then be trivially done with the mmap module (or conversely, provide an interface which take files and add support for buffers via the io module - whatever is more convenient/performant).

[blackencino]

Hey @lgritz , I've read through the merged PRs here, but can't find a quick example of how you'd ultimately solve the problem proposed above by egorodet, to read a JPEG directly from a memory buffer. Should that code sample work now, that you've merged the various fixes?

[lgritz]

@blackencino From C++ or Python?

[blackencino]

@lgritz, I was looking at this C++ code posted by @egorodet in Feb, 2019:

std::ifstream fs(file_path, std::ios::binary);
if (!fs.good())
    throw std::invalid_argument("Resource file path does not exist: " + file_path);
std::vector<unsigned char> image_data(std::istreambuf_iterator<char>(fs), {});

OIIO::Filesystem::IOMemReader image_reader(image_data);
OIIO::ImageSpec init_spec;
init_spec.attribute("oiio:ioproxy", OIIO::TypeDesc::PTR, &image_reader);
OIIO::ImageBuf image_buf(init_spec);
const OIIO::ImageSpec& image_spec = image_buf.spec();
if (image_spec.undefined() || !image_buf.read())
    throw std::runtime_error("Failed to read image from memory, error: " + image_buf.geterror());

OIIO::ROI image_roi = OIIO::get_roi(image_spec);
const size_t texture_channels_count = 4;
std::vector<uint8_t> texture_data(texture_channels_count * image_roi.npixels(), 255);
const OIIO::TypeDesc texture_format(OIIO::TypeDesc::BASETYPE::UCHAR);
if (!image_buf.get_pixels(image_roi, texture_format, texture_data.data(), texture_channels_count * sizeof(texture_data[0])))
    throw std::runtime_error("Failed to decode image from memory, error: " + image_buf.geterror());

[blackencino]

Though I guess I don't see in that example where it explicitly says that the image is a JPEG. I've got a case where I'm reading memory into a buffer via libcurl, and I know that memory is a JPEG, and I'd like to use OIIO to decode it rather than libjpeg.

[lgritz]

That code above is a little confused. OIIO::ImageBuf(ImageSpec&) makes an in-memory writeable ImageBuf, not a file-reading IB.

If you're using OIIO 2.2, the ImageInput, ImageOutput, and ImageBuf classes now have direct support for passing IOProxy*'s, you don't need the tricky hinting by stashing of pointers in a config spec. I think this is what you want:

OIIO::Filesystem::IOMemReader image_reader(image_data);
OIIO::ImageBuf::ImageBuf("name", /*subimage=*/ 0, /*miplevel=*/ 0,
    /*imagecache=*/ nullptr, /*config=*/ nullptr, /*ioproxy=*/ &image_reader);

Just a couple notes: the name parameter is just a placeholder since it's reading from the IOMemReader, also I just used default values for most of the other parameters, but that's where you'd pass custom config attributes or a non-default imagecache.

[DougRogers]

Is TIF supported for reading from memory? I have raw files with TIF previews that I would like to have OpenImage IO read.

[paulbrittain]

Are there plans to support passing IOProxy* compliant formats as input in the Python bindings?

[lgritz]

@paulbrittain On the C++ side, IOProxy support is about directly passing in a class that does the I/O, for use in the C++ code that reads and writes the files. So, for example, instead of calling fopen() or fread(), it might call the right methods in the IOProxy. Once you have this ability, you can do some interesting things such as making one of these proxies that doesn't actually do disk I/O at all, but works by emulating those call semantics by reading from a memory buffer.

I had never considered exposing this on the Python side. I'm not sure what that would entail, exactly, since it's not Python code that's doing the format reading or writing. Are you just seeking a way to read the file from a memory buffer? Or do you really want something akin to the whole IOProxy generality? If the latter, how would it work? I mean, can you sketch out what you'd like the calls to look like and what they would do?