search

panrafal / depthy

Extract depth map and original from photos made with Google Camera's Lens Blur.
http://depthy.stamina.pl
MIT License
1.31k stars 162 forks source link

Add support for HTC One M8 depth images #22

Open Verteiron opened 9 years ago

Verteiron commented 9 years ago

Supporting the HTC Dual Lens format

The HTC One M8 has an extra camera that adds depth information to images. Unfortunately they don't seem to have used the same storage format as Google Lens Blur, so Depthy doesn't load them correctly. While this is clearly HTC's fault, not Depthy's, it would nevertheless be awesome if Depthy could support for depth images created with the HTC One M8.

Obstacles

  1. HTC embeds their depth data in a non-standard way. Their API and SDK is written with the assumption that the images are being accessed via Android with the HTC Sense extensions. There is no API function to pull or dump the depth data from a dual lens image.
  2. The actual depth map is not included in the image, just the offset image from the second camera.
  3. HTC (may) be hiding the depth data from non-Sense apps, including MTP file copies. I cannot verify this right now, except to say that images I took with the HTC camera app do not appear to have embedded depth data. Since I have installed a custom ROM, however, I can't be sure the depth data is even being generated in the first place.

    Resolutions

  4. The offset image appears to be simply appended to the first image. It is easily isolated by splitting the file at the file's second JPEG (ÿØÿá) header. The second half of the split contains the offset image and its thumbnail. In the example file duallenssample.jpg, the second image starts at 0x1d4982.
  5. While I know little of the actual processes involved, generating depth data from an offset image seems to be a well-explored bit of math, and there are several libraries around that appear to do this automatically.
  6. The described method of extraction does work on the duallenssample.jpg sample image included in the HTC Dual View SDK, so the lack of depth data in my Dual Lens files may be a problem on my end.

    Sample data

Here is the sample file included with the HTC Dual Lens SDK: http://i.imgur.com/8SWMXPb.jpg And here is the offset image JPG I extracted from that file: http://i.imgur.com/Z8TvrUQ.jpg

This is all I have for this right now. I do not believe this is a duplicate of #9, as that was regarding Google Lens Blur images taken on the M7, not HTC Dual Lens images.

panrafal commented 9 years ago

Hello @Verteiron, thanks for the great rundown. I think that adding support for HTC's dual lens is non trivial as calculating depth maps from two images in the browser is a bit over-the-top. The only sensible way to support them, would be to have the grayscale depth image extracted from the image itself. If it's not possible, than it's not suitable in my opinion...

ghost commented 9 years ago

Hi @Verteiron, I'm working on HTC M8's dual lens now and I have some problem running DualLensDemo provided by HTC:https://www.htcdev.com/devcenter/opensense-sdk/htc-dual-lens-api/sdk-overview-sample-code/ When I run the demo, I got an exception that depth map info can't be found in jpeg. I'm wondering if you can get the color map of the picture. Thank you very much.

malenie commented 8 years ago

Hi, @kanamevs, This sample requires an HTC One (M8) with Sense 6. I upgraded to Sense 7 and I cannot get the depth map from the color image now.