[REQ] HDR settings

[MarcoRavich]

Nice to see so mutch "bugfix" releases, anyway it would be great to add more functions too.

For example it would be useful to have more controls for HDR shootings (on/off, exposure bracketing, etc).

Unfortunally we can't provide you any "inspiring app" released under a MIT-compatible license, sorry.

Hope that inspires.

[thestinger]

Our app provides modes for HDR, Night, Portrait, etc. on devices with support for them as CameraX vendor extensions. Most recent mid to high end Samsung devices support these and other vendors will be shipping them. It provides a far more sophisticated implementation than traditional HDR. We aren't interested in making a naive traditional HDR implementation to have a portable fallback. Implementing our own computational photography is outside the scope of the project. We'll have the modes provided by devices via CameraX vendor extensions which will become more configurable over time.

HDR+ is available on the Pixel 2 and later as part of the standard Camera mode. Night mode will be available on the Pixel 6 and Pixel 6 Pro in the near future. It's partially shipped but isn't usable via CameraX yet. It could potentially be enabled by a CameraX update without an OS update.

We also need to wait for the CameraX API to support features before we can provide them ourselves.

It would be possible to get computational photography HDR features on more devices by detecting and enabling the HDR scene mode but that feature is very strange and non-portable. It often requires that you do a lot of the work yourself. CameraX HDR extension is what we're really meant to use and time is what's going to provide those features, by waiting for vendors to ship CameraX extensions. There isn't a shortcut we can take to achieve what we want with this.

[WhyNotHugo]

How can I determine if my phone supports the CameraX vendor extensions for HDR?

Is there some list of devices that support this?

[thestinger]

GrapheneOS Camera supports all 5 CameraX vendor extension modes and displays them at the bottom to the left of the Camera mode when they're supported. Devices may only provide 1-2 of the extensions rather than all 5 extensions. CameraX has a partial list of devices which provide extensions at https://developer.android.com/training/camerax/devices. One of the columns is whether devices provide extensions.

[WhyNotHugo]

There are no tabs at the bottom. Does this mean that no AndroidX extensions are supported?

[thestinger]

You have the baseline 3 modes:

QR SCAN, CAMERA, VIDEO

On a device with all five CameraX vendor extensions, it will instead look like this:

QR SCAN, AUTO, FACE RETOUCH, PORTRAIT, NIGHT, HDR, CAMERA, VIDEO

[thestinger]

i.e. your device doesn't support any CameraX vendor extensions

[WhyNotHugo]

The availability of CameraX extensions unfortunately depends on whether the OEM/manufacturer of your device decides to implement them or not.

Do you mean the hardware vendor here?

Night mode will be available on the Pixel 6 and Pixel 6 Pro in the near future. It's partially shipped but isn't usable via CameraX yet. It could potentially be enabled by a CameraX update without an OS update.

So the only thing missing is the API? So the hardware is out, but software support does not exist yet?

[thestinger]

Pixels have HDR+ for the normal Camera mode. They could add an HDR mode if they wanted and could gradually move away from providing HDR+ in the normal mode but it doesn't matter much. They probably don't want to lose HDR+ in the vast majority of apps not using CameraX extensions.

Pixels could fairly easily add Portrait, Night and Face Retouch since Google Camera has them. It's not about implementing the features but rather just providing the extensions.

I'm just explaining the Pixel 6 and Pixel 6 Pro will get Night mode in the near future since it's partially shipped already as an extension but it isn't a CameraX extension yet.

[WhyNotHugo]

So I take it that CameraX extensions are not very widespread either.

Pixels could fairly easily add Portrait, Night and Face Retouch since Google Camera has them.

So you could say that "a better version" of camerax exists, and its what google uses for its proprietary camera app?

[thestinger]

So I take it that CameraX extensions are not very widespread either.

They are widespread. Samsung phones are extremely popular and other vendors are providing them too.

So you could say that "a better version" of camerax exists, and its what google uses for its proprietary camera app?

No.

[WhyNotHugo]

How does Google's camera app manage to surface the Portrait and Night modes?

[thestinger]

They implement the processing for HDR+, Night, Portrait, Face Retouch and other features in Google Camera. The same APIs it uses are available to other apps. The purpose of CameraX extensions is to allow vendors to provide the implementations of those modes in a way other apps can use their implementation instead of making their own.

[WhyNotHugo]

So HDR+, Night, Portrait, Face Retouch and other features are software-implemented, not hardware-implemented?

The purpose of CameraX extensions is to allow vendors to provide the implementations of those modes in a way other apps can use their implementation instead of making their own.

I don't fully follow. Vendors implement this feature in hardware, right? So how would, for example, Google rol out an update that implements it for the HDR+?

The purpose of CameraX extensions is [...] to provide the implementations of those modes in a way other apps can use their implementation instead of making their own.

This part makes sense, but how does the vendor implement this? I don't understand if you're talking about the hardware manufacturer (e.g.: OnePlus, Google), or the software vendor (e.g.: LineageOS, GrapheneOS).

[thestinger]

So HDR+, Night, Portrait, Face Retouch and other features are software-implemented, not hardware-implemented?

They're a mix of hardware features with processing done with varying amounts of hardware acceleration.

I don't fully follow. Vendors implement this feature in hardware, right? So how would, for example, Google rol out an update that implements it for the HDR+?

Pixels already provide HDR+ for GrapheneOS Camera on the Pixel 2 and later. The implementation varies based on the device. Pixel Visual Core, Pixel Neural Core, Qualcomm DSP or Tensor hardware acceleration features are used to make the HDR+ implementation high performance.

CameraX vendor extensions are a way for vendors to expose an implementation of special image capture or in the future video features. These can be based around different hardware features in the image sensor and hardware acceleration for the processing or can be implemented purely in software.

This part makes sense, but how does the vendor implement this? I don't understand if you're talking about the hardware manufacturer (e.g.: OnePlus, Google), or the software vendor (e.g.: LineageOS, GrapheneOS).

That depends on how the device chooses to implement it. We're obviously not going to make our own implementation for each device.

[thestinger]

It is not for GrapheneOS to implement CameraX vendor extensions. It's part of the device support code the vendor has to provide for AOSP. GrapheneOS will have CameraX vendor extensions when they ship for the stock OS and AOSP. It is not something we are going to implement on our own... that would be incredibly unrealistic. If Google has been unable to get this done so far with their resources, how could we be expected to do it?

[WhyNotHugo]

The implementation varies based on the device. Pixel Visual Core, Pixel Neural Core, Qualcomm DSP or Tensor hardware acceleration features are used to make the HDR+ implementation high performance.

Right, so the implementation is device specific... so this is done via firmware/drivers?

CameraX vendor extensions are a way for vendors to expose an implementation of special image capture or in the future video features. These can be based around different hardware features in the image sensor and hardware acceleration for the processing or can be implemented purely in software.

But there's no software-based fallback, is there? Meaning, something analogous to how we have CPU-based video rendering on platforms where there's no support for hardware acceleration.

It is not for GrapheneOS to implement CameraX vendor extensions.

Yes, this is very clear... I'm mostly trying to understand who have to do it and how.

It's part of the device support code the vendor has to provide for AOSP.

So basically vendors have to upstream it into AOSP and then that trickles down.

It is not something we are going to implement on our own... that would be incredibly unrealistic.

Agreed, I did not mean to imply that either, mostly trying to understand what needs to be done, and by whom.

It would seem like support for these extensions is a great criteria for picking a hardware vendor. OTOH, some features (like Night Mode) are not supported by the extensions, so using CameraX implies I cannot have things like Night Mode due to missing APIs.

It doesn't sound like the missing Night Mode API is an issue for Google though, since they don't use CameraX anyway.

[thestinger]

Right, so the implementation is device specific... so this is done via firmware/drivers?

Answered above.

But there's no software-based fallback, is there? Meaning, something analogous to how we have CPU-based video rendering on platforms where there's no support for hardware acceleration.

A vendor can provide an implementation in software. There isn't a standard AOSP fallback implementation.

Yes, this is very clear... I'm mostly trying to understand who have to do it and how.

See the link above.

So basically vendors have to upstream it into AOSP and then that trickles down.

No that's absolutely not how it works.

Agreed, I did not mean to imply that either, mostly trying to understand what needs to be done, and by whom.

OEMs need to implement CameraX vendor extensions. GrapheneOS Camera fully supports them on any device and there is nothing more to do within the app right now until there are more image capture extensions to support or video capture extensions.

It would seem like support for these extensions is a great criteria for picking a hardware vendor. OTOH, some features (like Night Mode) are not supported by the extensions, so using CameraX implies I cannot have things like Night Mode due to missing APIs.

That's wrong. The whole point of Camera2/CameraX extensions is to provide a way for OEMs to include an implementation of these 5 modes (one of which is Night mode) fully taking advantage of their hardware. I don't know why you're trying to use Night mode as a counterexample when it's one of the 5 CameraX vendor extensions.

You've got a very incorrect understanding about how these things work and you aren't properly taking in the information that I've provided.

It doesn't sound like the missing Night Mode API is an issue for Google though, since they don't use CameraX anyway.

They do use CameraX in some of their apps. CameraX is a library using Camera2 or when not available Camera1. Camera2 and the legacy Camera1 are the OS APIs. Camera2/CameraX extensions (both of which exist and provide the same functionality) are vendor provided modules taking full advantage of non-portable APIs under the hood and which are tuned for the hardware.

Nowhere in this conversation have I ever suggested that it wasn't possible to implement these features without CameraX extensions but our app is obviously not going to be making complex post-processing features taking advantage of non-portable hardware capabilities on a per-device basis which have to be customized, tuned and rewritten based on what the hardware has available...

We chose to implement the app in a portable way and to gradually expose more and more of the capabilities provided by the vendor. GrapheneOS Camera is not limited to GrapheneOS or Pixels, and it is explicitly not a goal of the project to implement our own version of features like Night mode rather than using the vendor provided implementations.

You need to be patient and wait for vendors to ship the implementations. Asking all these questions isn't going to make anything go faster and there is nothing for GrapheneOS to do at this time. We support all 5 existing modes. We can easily add more modes or video modes if those become available. We can extend the app with extension configuration if that becomes part of using the extensions. There is NOTHING for us to do right now.

[WhyNotHugo]

OEMs need to implement CameraX vendor extensions.

I don't understand how this part works entirely...

Suppose tomorrow a hardware vendor implements this for one of their devices that has already shipped. How does this implementation reach the devices already in our hands? If they don't upstream it, how does it get to your phone or mine? Do we (end users) need to visit the vendor's website and check for firmware updates or is there some app that does that?

Nowhere in this conversation have I ever suggested that it wasn't possible to implement these features without CameraX extensions but our app is obviously not going to be making complex post-processing features taking advantage of non-portable hardware capabilities on a per-device basis which have to be customized, tuned and rewritten based on what the hardware has available...

Sure, this sounds very reasonable; it's clearly outside the scope of the project. I was mostly curious if it would be possible for a fallback to exist [as a separate project]. Again, kinda like software implementations of graphics stacks exist for devices where hardware acceleration does not exist. That obviously is an entirely separate project.

You need to be patient and wait for vendors to ship the implementations. Asking all these questions isn't going to make anything go faster

On the desktop side, if I have hardware, and the vendor implements support for new features, this functionality is upstreamed, and then trickles down to each user during OS/package updates. Or shipped via firmware updates.

Given that you mentioned it's not upstreamed in this case, I was curious how it would reach users. Given that the support is not upstreamed in this case, I'm just unsure how it would reach end users. Is the implementation just bundled with firmware?

[thestinger]

I don't understand how this part works entirely...

Suppose tomorrow a hardware vendor implements this for one of their devices that has already shipped. How does this implementation reach the devices already in our hands? If they don't upstream it, how does it get to your phone or mine? Do we (end users) need to visit the vendor's website and check for firmware updates or is there some app that does that?

In that case you don't appear to understand that Android is an OS family, not a specific OS, and that the OS provided on a device is provided by the device vendor. The stock OS on a Samsung phone is a Samsung OS. The stock OS on an HTC phone is an HTC OS. The firmware and software is shipped together in OS updates. The vendor is responsible for providing the CameraX vendor extension implementation. They're responsible for publishing the code that's needed to build AOSP for their device if they want to support alternate OSes. AOSP has the code to support Pixels, not other OEMs.

Sure, this sounds very reasonable; it's clearly outside the scope of the project. I was mostly curious if it would be possible for a fallback to exist [as a separate project]. Again, kinda like software implementations of graphics stacks exist for devices where hardware acceleration does not exist. That obviously is an entirely separate project.

It's possible for a camera app to implement their own Night mode, Portrait mode, etc. by implementing processing which takes advantage of many non-portable APIs and has to be written for specific device families and tuned for specific device models. So, sure, you can do that, and it's not within the scope of this app at all because it's a portable app not an app only written for specific devices like Pixels. It would not make much sense to invest a massive amount of effort in making a poor alternative to the features on Pixels that are going to be available via CameraX vendor extensions.

On the desktop side, if I have hardware, and the vendor implements support for new features, this functionality is upstreamed, and then trickles down to each user during OS/package updates. Or shipped via firmware updates.

It's not any different. If it's implemented in an update, the OS ships the update to the firmware/software. Most vendors other than Samsung who already shipped it and Google are probably not going to implement this in an update for existing phones.

Given that you mentioned it's not upstreamed in this case, I was curious how it would reach users. Given that the support is not upstreamed in this case, I'm just unsure how it would reach end users. Is the implementation just bundled with firmware?

It's provided by the OS. The OS ships all the firmware.

[thestinger]

I explained above that these features are a mix of hardware, firmware and software. The whole point is that's developed together and then shipped as part of an overall product where these features take advantage of the image sensor capabilities, hardware acceleration capabilities for processing and whatever else is available. Most of these features is software written to take advantage of the hardware features, and which runs as much as possible on special hardware like the Qualcomm DSP. I explained all this above.

[WhyNotHugo]

They're responsible for publishing the code that's needed to build AOSP for their device if they want to support alternate OSes. AOSP has the code to support Pixels, not other OEMs.

So I should not expect support for CameraX on any device except for Pixels, given that OS vendors don't have an upstream from where they can pull the firmware, right? An exception might be when using the stock OS.

It's provided by the OS. The OS ships all the firmware.

Oh, this is the part I was missing! :)

So this aspect does kinda work the same as on desktops (although firmware is commonly updated without requiring an OS update to happen at the same time).

So I guess that GrapheneOS ship firmware for Pixels, LineageOS ship firmware for OnePlus, etc.

Thanks for your patience!

[thestinger]

So I should not expect support for CameraX on any device except for Pixels, given that OS vendors don't have an upstream from where they can pull the firmware, right? An exception might be when using the stock OS.

CameraX works on every Android device. CameraX vendor extensions are an optional feature for the vendor to provide.

Samsung is currently shipping the best implementation of CameraX vendor extensions. They provide a high quality implementation of all 5 of the current extension modes (HDR, Bokeh/Portrait, Night, Face Retouch, Auto) matching their own vendor camera app implementation.

Several other vendors are also shipping them on at least some of their devices.

Pixels are behind on shipping CameraX vendor extensions, because they wanted the extension API to be enhanced to be able to fully support the features they provide. They've shipped a Camera2 vendor extension providing Night mode for the Pixel 6 and Pixel 6 Pro and that will have CameraX vendor extension support soon. It just isn't quite available yet because of the ongoing process of CameraX supporting the new more advanced extension API.

[thestinger]

So this aspect does kinda work the same as on desktops (although firmware is commonly updated without requiring an OS update to happen at the same time).

SoC firmware images can be shipped without updating the OS images, but it's still an OS update and you need to reboot to load the new SoC firmware. It wouldn't make much sense to ship both a firmware image OS and an OS image update every month instead of a single unified update. It's possible, just not sensible, since it would be far more inconvenient for users. It's most efficient to ship all firmware and OS components together since delta updates mean only the actual differences are shipped, and there's a single reboot for all the AOSP, downstream and firmware updates every month on typical Android devices.

Most firmware other than the baseline SoC firmware is usually included in the OS vendor image and loaded by the OS. If the firmware is shipped in an APEX module, it can be updated out-of-band. For example, Pixel 6 camera firmware and other components are shipped in a camera APEX module in vendor.

One reason to use APEX modules is providing faster, more convenient updates to components that are not updated as part of the monthly AOSP release cycle. GrapheneOS does far more than monthly OS releases so we have little use for APEX modules as a whole right now. It's too much trouble to do things that way especially since only about 10% of the changes we end up making are currently in AOSP APEX modules. Most APEX modules are for AOSP components and exist so that they can be updated via the Play Store for devices using the Google Play releases of the mainline APEX modules which are open source as part of AOSP and tagged after the monthly release.

[thestinger]

So I guess that GrapheneOS ship firmware for Pixels, LineageOS ship firmware for OnePlus, etc.

We ship all the firmware updates as part of our OS updates including all the SoC firmware images and the firmware included in the vendor image. It often doesn't work that way on LineageOS. On most of their devices, they aren't shipping all the firmware updates and users are expected to flash the firmware updates every month on their own via fastboot, which is highly unrealistic. They do tend to ship the firmware updates for firmware included in the vendor image but they miss the SoC firmware image updates for most devices. It's also very hit or miss if they ship the vendor's updates for the device support code and firmware included in vendor, etc.

[WhyNotHugo]

Thanks again for explaining all this and you patience. This has been very informative!

It often doesn't work that way on LineageOS. On most of their devices, they aren't shipping all the firmware updates and users are expected to flash the firmware updates every month on their own via fastboot, which is highly unrealistic. They do tend to ship the firmware updates for firmware included in the vendor image but they miss the SoC firmware image updates for most devices. It's also very hit or miss if they ship the vendor's updates for the device support code and firmware included in vendor, etc.

Yeah, I just hit this issue a couple of days ago; in order to upgrade to Android 12, I need to manually upgrade the device's firmware first by downloading if from the vendor's website (oneplus, in my case)

I get an impression that oneplus does not ship the firmwares by itself in an easily re-distributable package; so I'm not sure if LOS could do anything better.

[thestinger]

I get an impression that oneplus does not ship the firmwares by itself in an easily re-distributable package; so I'm not sure if LOS could do anything better.

They release it the same way it's released for Pixels. It's easy to handle. LineageOS has chosen not to ship full updates for most devices. It requires investing the time to properly handling it. Recently, they started dealing with it, but it has been years where they are leaving users without many important security fixes that are available but not shipped to users with the excuse being that users need to flash them every month on their own. That also rules out ever supporting locking the bootloader and using verified boot, but they explicitly don't support that and largely disable verified boot so it doesn't really matter. One of many robustness / security issues with LineageOS.

[WhyNotHugo]

LineageOS has chosen not to ship full updates for most devices. It requires investing the time to properly handling it. Recently, they started dealing with it, but it has been years where they are leaving users without many important security fixes that are available but not shipped to users with the excuse being that users need to flash them every month on their own.

Yeah, I've had to deal with this nuisance these last couple of days. Updating from Android 11 to 12 is a pain and requires lots of manual work.

That also rules out ever supporting locking the bootloader and using verified boot, but they explicitly don't support that and largely disable verified boot so it doesn't really matter.

I've also come across this. It just sounds wrong to deliberately leave a device unlocked if you're going to handle sensitive data on it!