Support Gcam's "portrait mode" (and editing depth of field?)

[e-t-l]

Is your feature request related to a problem? Please describe. Aves is a wonderful FOSS replacement for Google Photos, with its a ility to view and edit complex image types like Motion Photos. However, Gcam also produces other types like Portrait Mode, where multiple images are taken at different depths of field, then combined to create a bokeh effect. In Google Photos, these Portrait Mode images can be edited, so users can manually select which field depth to be in sharp focus (this is helpful in case Gcam's algorithm puts the wrong subject in focus). Currently, I don't believe Aves is capable of performing this sort of depth selection/edit.

Describe the solution you'd like Aves should be able to edit Portrait Mode-format images from Gcam in order to manually select different focus depths and save the resulting image.

Describe alternatives you've considered Installing Google Photos :/ I'm not aware of any other FOSS gallery app or editor that can do this sort of focus depth edit, but my research is ongoing.

[deckerst]

I'm not familiar with the format. Is it a single file with embedded metadata/images, or multiple files on storage? Please share a sample here.

Also, if the metadata are in a private format (which is likely), Aves won't be able to support it.

[e-t-l]

Hey, thanks for your responsiveness! That question pushes the limit of my (very limited) technical knowledge, but from some preliminary research it looks like it's a single image with a "depth map" in the EXIF/XMP data. Extracting it with Exiftool looks promising. I'm using the answers in] this Google support thread](https://support.google.com/pixelphone/thread/23336967/how-do-i-extract-the-depth-map-image-from-a-portrait-mode-photo?hl=en) (which includes an example image that can be examined) as a jumping-off point.

There's an (unaffiliated) exiftool android app which let's me view the depth layer data but not edit is with a GUI. From the Google support link, it looks like Photopea does have the ability to edit depth maps in the GUI, and since it's open-source maybe that code can be useful to look at?

[deckerst]

Thanks, I'll take a closer look at the sample.

In the short term, I'm interested in the extraction/view of the depth map. Not sure about depth editing yet.

[deckerst]

It doesn't look like photopea is actually open-source, but i checked the sample metadata and it seems promising:

The interesting part is the Container section, where there are 4 items: 1) primary_image 2) android/original_image 3) android/depthmap 4) android/confidencemap

Each of these items are embedded in the file, at different offsets. primary_image is the default image displayed by simple viewers. I guess the other items allow playing with the depth.

I'm confident about extracting the different parts, but whether I'll be knowledgeable/clever enough for depth editing is an open question!

edit:

• Dynamic Depth spec from Google: https://developer.android.com/static/training/camera2/Dynamic-depth-v1.0.pdf
• info on the xiaomi variant of portrait mode: https://github.com/VincenzoLaSpesa/XiaomiDepthMapExtractor/wiki

[e-t-l]

Awesome! That's already far more than I'm capable of. And you're right about Photopea's code, sorry I should've looked more closely at the repo I linked you. It's still possible the dev would be willing to help you with this, since they seem like a pretty libre-oriented person.

I also found this on Stackoverflow, where it looks like some people were trying to do the same or similar thing as us: https://stackoverflow.com/a/65632120

[deckerst]

@e-t-l in Aves v1.7.9, you can extract individual items contained in a Google portrait mode image. It's in the Info page > XMP > Container section. There should be 4 items. I only had one sample to work with, so I'm not sure the extraction works well in general. If you could give it a try, that would be nice.

[e-t-l]

Just snapped a photo in portrait mode. I'm attaching it, along with the exported metadata, so you can check if I missed something... While I do see stuff about 4 containers in Info>XMP, I don't see any option in the UI to extract any items.

PXL_20230116_000639752.PORTRAIT-metadata.txt

[deckerst]

Damn, apparently i committed non functional code for the release...

Thanks for the sample!

[e-t-l]

~@deckerst I have to apologize! I was going back through some photos, and I think the sample I shared with you does NOT actually contain depth data. Although Gcam was in portrait mode when I snapped that photo, if the camera's algorithm can't identify distinct depth planes, it just takes a normal photo. I think that's what happened here.~

~So I don't think you did anything wrong; it was just a bad sample. I opened some other Portrait Mode photos in Aves and I WAS able to extract the base image and depth map from the containers in Info, just like you described.~

[deckerst]

Actually, your sample did contain a depth map! There was some random issue that prevented Aves from recognizing XMP content in some cases. But it should be fixed now in v1.7.10. Thanks again for the sample and the follow-up.

[e-t-l]

Really? Oh that's interesting. Glad you were able to figure it out! You're clearly much better at this than me