Merging with 32bit precision, reduced memory usage and improved performance

[chris-rank]

Closes #36.

• Improved precision of merging: merging is now performed with full 32 bit float precision
• Improved precision of alignment textures: black level is subtracted before writing into the 16 bit float textures improving bit precision in the shadows => this ever so slightly improves alignment quality in the deep shadows
• Strong reduction of peak system memory usage and cumulative memory allocation
• Several performance improvements leading to a reduction of processing times
• Bug fix: previous app crashed when Adobe DNG converter was not installed
• Bug fix: settings icon was not shown on older versions of macOS
• Bug fix: settings panel did not open on macOS 14 Sonoma when clicking on the settings button

Main changes that led to reduction of system memory usage and cumulative memory allocation:

• In current V1.5 the input burst stored in the cache are int16 textures being identical to the pixel values from the input DNGs. It is then mapped to float16 textures with some slight corrections for the pipeline execution. This approach stores information in a redundant way. In the current PR, the complete burst is only stored as int16. The conversion is now to float32, but it is applied at a later stage of the pipeline frame by frame. This can reduce the system memory usage considerably, e.g. for example 1 with 48 x 24.3 MPix this change reduces system memory usage by approx. 2.6 GB when textures are stored as float16. Savings would double for float32 precision.
• In the current PR, the storage mode of textures is changed from 'shared' to 'private'. As a consequence, only the GPU can access the textures. It seems that Metal manages memory in this storage mode in a much more dynamic and extremely efficient way. The textures do not contribute to the system memory usage nor the cumulative memory allocation anymore. I found that initialization as close as possible to the location where the textures are needed is beneficial for calculation times. There seems to be no disadvantage in terms of memory usage or cumulative allocation. In contrast to very early allocation of textures that block memory during the whole pipeline execution, the more dynamic memory management by Metal leads to lower peaks when profiling memory usage. All these changes are based on my observations and I fear what really happens can only be explained by Metal experts at Apple.

Some tests performed on a MacBook Air M1 with 7 GPU cores and 16 GB memory using the frequency merging:

Example burst	Parameter	Burst Photo V1.5	Current PR
1) Uniform exposure: 48 x 24.3 MPix	peak system memory usage	7.3 GB	4.0 GB
	cumulative memory allocation	55.8 GB	7.1 GB
	processing time: align+merge	24.3 s	21.4 s
2) Bracketed exposure: 7 x 20.3 MPix	peak system memory usage	3.0 GB	0.6 GB
	cumulative memory allocation	11.7 GB	1.0 GB
	processing time: align+merge	4.24 s	3.23 s

[martin-marek]

Wow, this is great Chris! Do you want to make this a v1.6 release?

[chris-rank]

Yes, this could be a version 1.6 release. But we should align with Alex first as he has also an open PR. While I already tested the code a lot, some review or testing of someone else would be great.

[Alex-Vasile]

Yes, this could be a version 1.6 release. But we should align with Alex first as he has also an open PR. While I already tested the code a lot, some review or testing of someone else would be great.

I don't really care about merging order or release number.

I would like to review this. I particularly want to see how this relates to #32. I imagine the cumulative memory allocation difference is from reusing textures.

[Alex-Vasile]

Re: dealing with the black level. I was thinking about this, but how come we don't subtract the black level once as part of reading the DNG from file into the first metal texture. Beyond that we shouldn't need it anymore.

[chris-rank]

I had a look into the description in Apple's documentation and did not find a way to explicitly trigger lossless texture compression. It might not even be supported properly for the macOS version 11 we currently build the app for. As the peak memory usage is much lower now, I thought that it may not be needed anymore. My interpretation of your issue was that some evaluation of that feature shall be done.

[Alex-Vasile]

I had a look into the description in Apple's documentation and did not find a way to explicitly trigger lossless texture compression. It might not even be supported properly for the macOS version 11 we currently build the app for. As the peak memory usage is much lower now, I thought that it may not be needed anymore. My interpretation of your issue was that some evaluation of that feature shall be done.

The peak is much better now, but the peak will still slowly increase until the limit set for the cache. The compression was for trying to increase the amount of photos that could be cached without increase the memory usage.

[chris-rank]

If you prefer, I can remove the #32 from this PR. You are right, a compression would enable more frames on the cache.

[Alex-Vasile]

If you prefer, I can remove the #32 from this PR. You are right, a compression would enable more frames on the cache.

I think you should. We can close that issue separately if we don't find a solution, but I don't believe that this PR (good as it is) closes it.

[Alex-Vasile]

Overall, could you please provide more information about how this PR reduces peak and cumulative memory usage?