💭 Image Compression and Frame Processing Question

[rmroot]

Question

I have two questions. One relating to image capture and one related to frame processing.

What we are trying to do: We are trying to apply a technique called "Background-Oriented Schlieren" to identify airflow using images. To do this you can subtract the pixels of 2 stationary images from each other to see if there has been a pixel shift that indicates airflow (dependent on air temperature).

What we have implemented: So far I have been able to use this library to implement the image capture portion. We then send the images to a server for processing to identify the pixels shifting and return manipulated images to the phone for display. It's a pretty straightforward approach but not the most user friendly. Ideally we would like to do this processing on the device but it's fairly computationally intensive to iterate through all the pixels and a cloud base solution has been the most effective so far.

Question 1: How does this library handle the compression of image files? When I call takePhoto() or takeSnapshot() it creates a temporary image file that I am able to grab from the device and send off for processing. Would you be able to tell me what sort of compression is being done to those .png files that are temporarily stored within the application data? Is this something that the library is doing or is it device/hardware specific?

Question 2: Is it possible to use 2 frames within a frame processor? My first attempt at an implementation was using a frame processor but I pivoted from their because of inexperience and trouble using 2 frames within the frame processor. Is it possible to use 2 frames within a frame processor? All of the examples I have seen are looking at single frames and I would appreciate knowing if I'm on a fools errand trying to compare two frames within the frame processor architecture.

Thank you for any info that can be provided!

What I tried

No response

VisionCamera Version

4.5.0

Additional information

• [ ] I am using Expo
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar questions in the issues page as well as in the discussions page and found none.

[maintenance-hans[bot]]

Guten Tag, Hans here 🍻

Thank you for your interesting questions!

1. Regarding ze image compression, ze library does not specify ze compression method for PNG files. It may depend on ze device and hardware. It would be best to check if ze device itself has default compression settings or specifics for image saving.

2. About using two frames in a frame processor, currently, ze architecture is designed for single frame processing. If you want to compare two frames, you might need to implement zese comparisons manually outside of ze frame processor or create a custom solution.

Feel free to keep exploring, and remember, if you want more attention from mrousavy, consider supporting ze project on GitHub Sponsors!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

[mrousavy]

Hi!

1.: How does this library handle the compression of image files?

Compression is handled using normal device JPEG encoders. That's it. For very obvious reasons, I don't choose any opinionated compression algorithms or settings. "to those .png files" What png files? VisionCamera does not use png.

2. Is it possible to use 2 frames within a frame processor?

A Frame Processor is a stream of Frames. You can queue up frames if you want. So yes if that answers your question.

[rmroot]

Hi!

1.: How does this library handle the compression of image files?

Compression is handled using normal device JPEG encoders. That's it. For very obvious reasons, I don't choose any opinionated compression algorithms or settings. "to those .png files" What png files? VisionCamera does not use png.

2. Is it possible to use 2 frames within a frame processor?

A Frame Processor is a stream of Frames. You can queue up frames if you want. So yes if that answers your question.

My bad, definitely meant "jpg" not "png". Thanks for the feedback. Good to know I can use multiple frames and I am not trying to do something impossible. Appreciate your work!