update metadata without rewriting the entire file

[dekekincaid]

It would be nice to have the ability to update metadata on an EXR without rewriting all the channels in a file.

• This is especially arduous around very long shots (35k frame oner/one-take shots) or location-based entertainment where the shots can easily be 25-50k frames.
• Recently, with the need for machine learning where you can have 10-100k+ frames of training media. File ingestion can take days or weeks because so much media was shot, and all you want to do is copy it into place and then use it without needing to pre-process it first.

More specific use cases for this:

• Inserting metadata into a render as a post-process from the pipeline. A pipeline has to write the data twice for every frame, which is very expensive on long shots or large frames. We actually disable all metadata modifications on Deep files, for example, because OIIO needs too much memory to rewrite them every frame, and the beefy machines with lots of RAM are reserved for FX sims. This cannot be wasted on simply inserting metadata into deep exr files which can take 30-60 minutes per frame to do.
• creating a "meta" shot for the "oner" shots where you merge together 100-200 different shots into one master shot because the client wants the keys in the EXR a specific way. Delivery for these kinds of shots takes way too long to do.
• The client embedded CDLs in the header, and the CDL changed. Artists will wrongly trust the metadata and try to pull the values into their script, but this is wrong. Rewriting thousands or even millions of files from the client is expensive when things like this are found after the fact. I can also see this becoming an issue with FDL in the future when they change, too.
• republishing a plate or render from one show to another or one shot to another, and all I want to do is copy the files and then update a "SHOT" or "SEQ" key.

[peterhillman]

Have you tried exrstdattr instead of OIIO? That's much faster and less memory intensive because it doesn't decode the pixel data, just writes a new header then copies the compressed image data chunks into the new file. You would need to create a temp file then rename over the original, but that can be used to guarantee you don't end up with a corrupt file if the metadata modification gets interrupted.

It could be possible to modify attributes in place in the header, but only if they stay exactly the same size, and you can't insert, remove or rename attributes that way. If the header size changes, any tool would have to move the rest of the file by reading it in and writing it back out, which is what exrstdattr does anyway.

[dekekincaid]

Yes, the idea is similar to ffmpeg, where you can simply "copy" the channels from one file to another without writing every pixel again.

This would also help when using files compressed with DWAA and DWAB, so you do not recompress a lossy image.

[lgritz]

Somebody just added that to oiio trunk (copy without decode for exr files), though it doesn't yet work for the exrcore code path.

But back to the original question -- I'm not sure you could do this without an incompatible change to the file format, but you could imagine a change to the exr file layout spec itself that would make it more amenable to adding metadata "in place" in an existing file.

[kdt3rd]

there is a stubbed function in the core to support in-place edit of metadata (provided you do not change the length of strings or add values, but just updating a float or whatever). However, it is not yet finished, but was a desired tool to finish at some point! You can more easily do the copy without any decode with core, in that there are direct "chunk" read / write routines, completely skipping any decode / encode path, which you'll see me using in the rawPixel-ish functions in the cpp rewrite branch I'm working on currently. Would be lovely to have some contributors