encode and write depth data to heic file

[yizhaoyanbo]

Hello, I wanna to write depth data to heic file. but I cannot find the proper API. I find that this labrary only have two APIs to encode and write primary image ( heif_context_encode_image() ) and thumbnail image ( heif_context_encode_thumbnail() ) .So, if I use the former API to encode and write depth data to heic file, the depth map will be a part of primary images. How can I deal with it ? thanks very much.

[jerett]

same problem, plz.

[supertobi]

I would like depth map support too. And if you have implemented it here then please add it to the Gimp plugin too.

[supertobi]

Is this fixed with issue https://github.com/strukturag/libheif/issues/145 ?

[hyacin75]

Is this fixed with issue #145 ?

I don't believe so as that is referring to decoding.

Four years later and I'm trying to do the same thing - encode an HEIF file with image data, and a depth map, and it still seems to be impossible :-(

[farindk]

I'm doing all work in my private free time. Everyone is invited to send pull-requests or sponsor the development, so that I can allocate more time on it.

This is no easy change since a clean implementation will probably require a major architectural change in how image files are constructed.

[hyacin75]

@farindk Yeah sorry man, not meant as a dig or anything. I'd send a PR if I could but I looked at the code and made a face like a meme with my confusion lol. I did then see and check the sponsorship options but didn't see a "help bump your issue up" or anything tier, and with someone as big as Shopify on there I figured throwing $5 at you or something would just be a pointless drop in an ocean anyway. Incredible job man, I wish I could write something this cool. It's a shame there aren't more people willing/able to help with stuff like this.

[farindk]

No problem, I guess I was in a bad mood because I found out that a huge company is using my code in many of their products and violating the license.

But back to the depth image: what would be the use-case? Is there any software available that would read the depth image?

[hyacin75]

I can't speak for the others who were interested obviously, but the only software I've seen that does is Lightroom, and it's probably an extremely limited use image mask. It works out of the box with iPhone portrait HEIFs, so I just wanted to see if I could convert a Samsung depth-map embedded JPEG (which LR will support naively the exact second I accomplish this) into a properly embedded and linked HEIF and then see if LR would support it!

Honestly probably not worth the effort for my use case, especially if it's a huge arch change ... I was just hoping to find something I could hack one together with just to see if I could get that mask option to light up in LR 😃

This library here - https://github.com/nokiatech/heif has some 'examples' (though I'm honestly not even sure what language they are at this point) and seems to have all the functions for adding the second image, labelling it correctly and then linking it, so depending just how bored I am if/when my coffee kicks in, I may try to do something with that!

I may be misunderstanding the spec/docs I read though if you're saying it would take a huge arch overall of your library, because yours already supports putting two images in there - I did do that, but they show up as ID1 and ID2 - from what I gathered from what I read last night, all it takes from there is putting the right label on the ID2 image and then linking it to the first ... both sounding like metadata type operations more than anything ... it was 2 AM though and I have no idea what I'm doing, lol.

[hyacin75]

I found out that a huge company is using my code in many of their products and violating the license.

Yes... that would make me borderline homicidal 😬

[hyacin75]

This is how bad I am at this btw 🤣 ... not kidding when I said I have no idea what I'm doing, but that's served me just fine in the past, I muddle through 🤣

me@manager:~/depth-map-play/heif/srcs/examples$ g++ example.cpp -L../api/reader -L../api/writer -L../api/common -L../api-cpp -o asdf
example.cpp:26:10: fatal error: heifreader.h: No such file or directory
   26 | #include "heifreader.h"
      |          ^~~~~~~~~~~~~~
compilation terminated.
me@manager:~/depth-map-play/heif/srcs/examples$

[farindk]

How about this, if you just want to do some experiments: Put the depth data into the alpha channel and then change the auxiliary-image ID from "urn:mpeg:hevc:2015:auxid:1" (alpha) to "urn:mpeg:hevc:2015:auxid:2" (depth). You can hack this into the libheif code or just use a hex-editor on the final file to change the one byte.

[hyacin75]

That would probably do the trick yeah! I couldn't figure out how to do it with heif-enc though, I was looking at sticking it in as alpha and then just seeing what happened to start at least, but the only way I could find to get it in at all was as an entirely additional and separate image -

me@manager:~/depth-map-play$ heif-info testfile.heic
MIME type: image/heic
main brand: heic
compatible brands: mif1, heic, miaf, mif1, heic, miaf

image: 4032x3024 (id=1), primary
  color profile: no
  alpha channel: no
  depth channel: no
image: 756x1008 (id=2)
  color profile: no
  alpha channel: no
  depth channel: no
me@manager:~/depth-map-play$

Is there an undocumented flag or a flag I'm misunderstanding the definition of? Or would it require a modification to the heif-enc code? I'm certainly up for trying the latter if you tell me it's doable, as I seem to be quite obsessed with making this work for some reason 🤣

Update: Ah, it looks like I'm going to have to add it to the before image eh? I found some instructions for doing it with GIMP, I'll see if I can figure out the same with PS and then export it to a format that retains that! TY for the pointer!! 😄

Update2: Hrm, nope, it's lost on export as PNG or JPEG ☹️

Update3: GIMP to the rescue, woot -

me@manager:~/depth-map-play$ exiftool image-with-alpha-maybe.png  | grep -i alpha
File Name                       : image-with-alpha-maybe.png
Color Type                      : RGB with Alpha
me@manager:~/depth-map-play$ heif-enc image-with-alpha-maybe.png -o image-with-alpha-maybe.heic
me@manager:~/depth-map-play$ heif-info image-with-alpha-maybe.heic
MIME type: image/heic
main brand: heic
compatible brands: mif1, heic, miaf

image: 3024x4032 (id=1), primary
  color profile: prof
  alpha channel: yes
  depth channel: no
me@manager:~/depth-map-play$

Now on to the hex editing!

Update4:

Et, voila -

me@manager:~/depth-map-play$ heif-info image-with-alpha-maybe.heic
MIME type: image/heic
main brand: heic
compatible brands: mif1, heic, miaf

image: 3024x4032 (id=1), primary
  color profile: prof
  alpha channel: no
  depth channel: yes
 (3024x4032)
me@manager:~/depth-map-play$

Update5: Sadly no luck, perhaps something to do with the additional parameters that are missing compared to an image from an iPhone -

me@manager:~/depth-map-play$ heif-info 20190521_0112.heic
MIME type: image/heic
main brand: heic
compatible brands: mif1, heic, jpeg

image: 3024x4032 (id=49), primary
  color profile: prof
  alpha channel: no
  depth channel: yes
 (576x768)
    z-near: undefined
    z-far:  undefined
    d-min:  0.496826
    d-max:  2.302734
    representation: uniform disparity
    disparity_reference_view: 0
me@manager:~/depth-map-play$

In LR I'm actually able to see the depth map of my image, like, blended in or something ... my image is hazy and washed out, so I suspect that is what is going on ... and still no depth mask option 😞

I'll probably play more later, but my optimism is quickly shrinking ☹️

[silverbacknet]

One thing I see with your depth map is that it's 1/4 the size in both dimensions and rotated. When you merged it in, did you correct for that? (Just a shot in the dark, I don't have LR so I can't test.)

[hyacin75]

One thing I see with your depth map is that it's 1/4 the size in both dimensions and rotated. When you merged it in, did you correct for that? (Just a shot in the dark, I don't have LR so I can't test.)

Yeah I noticed that after the fact and did do a subsequent one with it rotated, and another with it rotated and enlarged to the same size. LR won't even display those images!

The one I linked above from someone's iPhone that they put online as a test image (which renders fine in LR and supports the depth masking) appears to have the same though, funny enough, smaller AND non-rotated depth-map. I think there is metadata that says the image is rotated, so it maps it correctly or something? Perhaps I should try my above clobbered and hacked HEIC with the incorrect depth map again!! 😁

[Michael-Z-Freeman]

Hi. I’m looking for this as well and have a usage example. I’m on a BA Hons Creative Music Technology course and as part of an assessment I'm doing some VJ’ing. This has involved solving a long standing problem with exporting 3D fractals from Mandelbulber. It’s an insane amount of work to prepare exported meshes or voxels to make a sane 3D model that is not too complex for real time VJ software and game engines. So I had a brainwave about using Apple Object Capture that uses AI to stitch together photo’s taken by the iPhone camera into a 3D model, with depth buffer of course ! So I “photographed” my 3D fractal in Mandelbulber, however Object Capture get’s confused and renders some parts of the model incorrectly. "Where’s my depth buffer ?” I thought. Which of course led me here. So I rendered the frames again, each with a seperate depth buffer image (which Mandelbulber calls the zbuffer). I just need to get it into the final HEIC image frames for Object Capture.

I will have a look at your code. There’s a chance I might be able to make coding it part of my assessment, but I’m not a trained software engineer.