Missing: Short explanation of the image capture pipeline

[CaptainSifff]

Hi all, as a followup to todays DataCarpentry onboarding I feel like a image processing workshop aimed at beginners should contain a short description of the image processing pipeline. Lacking any pictures myself, I will put a couple of diagrams in here, that I found using some google searches, so licensing may vary...

First We start with this overview diagram: overview of image capture

To summarize(kind of in reverse), here we see that a rasterized image can only be captured by an imaging device, if an object is properly lighted. To highlight the first couple of issues

• Lighting affects the visual perception of an object. light in a particular color can only be captured if the object receives this color and actually reflects it towards the device.
• All known devices today rasterize the image into something called pixels, which defines the lowest available resolution. The light intensity captured by this pixel therefore is the integral across the surface area of this pixel.

Moving forward to the imaging device itself, we can use a diagram like this: in-photo-camera capture To summarize, we see that from the outside the entire electromagnetic spectrum has to pass through the lens, optionally pass some filters in order to reach the sensor that converts photon counts to intensity levels in pictures.

This brings us to the issues that can occur in a device:

• assuming no filtering is present, the lens system affects the picture, obviously
• Intensity levels: there is a minimum sensitivity level and a maximum intensity in all devices that they can process. Hence a quantization procedure has to be applied, in order to transform intensities to values.
• color rasterization: In 99.9% of all consumer devices a color pattern is used(the image shows the so-called Bayer Pattern) to generate from a single capture of photon intensities a color image. This means that adjacent pixels only have information about particular color channels, and the other channels are obtained via interpolation.

Another diagram showing the entire pipeline

Pictures from here: [1] [2] And then we are at the point where a camera chooses a file format. File Format are chosen according to enhance transferability in a certain application. If your're looking at a image gathered by an expensive microscope with a program supplied by a vendor, chances are that the vendor has a file format for this particular application. If you're looking at camera photos with your friends a file format that is portable and adapted to the human visual system, (e.g. JPEG or JPEG2000) is chosen. Diagrams are often transferred using PNG.

Things we discussed today, but that I have not touched upon here:

• The generation of information, or the destruction of information using resizing operations.
• Color spaces
• psychological perception

Thanks for all your work on this episode!

[tobyhodges]

Thank you for the detailed issue @CaptainSifff. I agree that these are important points for e.g. a researcher to be aware of as they start to work with image data. I would like to try to find space for a discussion of this within the lesson, if we can.

My main concern is that this is a lot of (mostly theoretical) information for learners to grasp in addition to the new skills we want to teach them and the new concepts they need to understand to be able to apply those skills. For example, I do not think it would be a good idea to add these details of the image capture pipeline to the introductory episodes of the lesson, because it delays learners from getting some hands-on experience of working with the image data.

Is there a way that we can integrate parts of this discussion into the existing lesson content, distributed throughout the different episodes of the lesson, and perhaps tying everything together towards the end with an overview of image capture its most important technical considerations?

[bobturneruk]

I'm sure there could be refinement of the content to add some of this, very important, information, although I agree with @tobyhodges that we should probably be very careful about increasing the volume of content.

Another thing - I hope that the current content would enable someone to apply the techniques in the lesson to images obtained by means other than photography / microscopy with visible light. This would include things like astrophysics data, remote sensing, electron microscopy, scanning probe microscopy, optical coherence tomography - there's just loads of ways to get a digital image all of which have slightly different considerations in terms of how the imaging system translates the thing being imaged into pixels. Maybe we could handle this in abstract terms with a block diagram like Subject->Imaging System->Digital Image? Around the intro.? We could signpost resources for different types of imaging (including photography)?

We get very close to talking about quantisation in image basics when we talk about bit-depth - I think a sentence or two could get this concept in.

[uschille]

Adding @tobyhodges' comment in #330 here for reference:

Furthermore, if somebody still wants to take a run at https://github.com/datacarpentry/image-processing/issues/212 I wonder if this summary in Ethan's blogpost might provide a nice scaffold for that discussion?

We have to: 1) transfer data from the field; 2) identify ground control points; 3) combine 100s of individual drone images into one orthomosaic; 4) crop the ortho into pieces for the model; 5) run the model; 6) combine the crop-level predictions; 7) output the predictions; 8) archive the predictions and transfer them to a web server for visualization (which also involves a bunch of steps to display the imagery itself).