Issues with lepton.py -> struct unpack and understanding the metadata

[ben-kolber]

I was having trouble running the lepton.py module with a Lepton 2.5 with a PureThermal2 board via USB on a Linux Ubuntu 18.04. I came across some issues and wanted to both point them out and ask some questions.

1. In line 118 of lepton.py, you send to struct.unpack the image[-2 , :], where I believe you are trying to extract the last two lines and should add a semicolon such that image[-2 : , : ] is sent to struct unpack. res = struct.unpack("<2cII16x4h6xIh2xh8xhI4xhhhhhh64xI172x", image[-2,:]) # Error
2. When changing the value to the last two lines of the image, line 118 runs (it was failing before), but now the byte string unpacking doesn't really make sense in terms of the values extracted.

I was wondering whether the byte string is still relevant, and how exactly I should be reading in the camera metadata for pixel-level temperature calculation.

Thanks for the help!

[jveitchmichaelis]

Hi Ben,

In the datasheet for the Lepton it says there should be three telemetry rows (A-C). Flirpy requests the raw image from the camera (i.e. it requests Y16 video), so it should be 16-bit. Since the bits per pixel doubles, with the Lepton 3.5 the PT2 seems to merge row A/B. Therefore the line image[-2] is correct (for the v3.5), since it's the combined row A and B telemetry (320 bytes). i.e. you have

Line  0 (320 bytes)
...
Line 119
Telemetry A | Telemetry B <-
Telemetry C | Don't Care

I've only implemented Row A for the time being, but as you can see you need to unpack the whole line. That might explain why it works if you take both rows, but you should probably take image[-3:-1,:] instead (I'm actually surprised unpack lets you pass in a 2D array without flattening). Your camera is half the resolution, so you need to account for that i.e.

Line  0 (160 bytes)
...
Line 59
Telemetry A <-
Telemetry B
Telemetry C 

I think this is just how PT2 developers decided to present the telemetry (makes sense). When you read it out from the sensor it's just dumped at the start or end of the pixel stream.

If you have a radiometric Lepton, then it should just spit out radiometric images. You don't need to worry about conversion, it's all done in-camera. The conversion factor is 0.01 if TLinear mode is enabled. Double check - you should have 16-bit images coming from the camera, so if you multiply the pixel values by 0.01, does that look sensible?

Best, Josh

[jveitchmichaelis]

Still a bug though, because it means flirpy needs to take into account the frame size. As you see from the code, the telemetry extraction is a bit rough and ready :)

[ben-kolber]

Hey Josh, thanks for the responsiveness!

Lepton 2.5 is in fact radiometric and in some sense, the pixel values multiplied by 0.01 make (kinda) sense in Kelvin. I was using the first two lines of your code basically:

            cap = cv2.VideoCapture(device_id + cv2.CAP_V4L2)
            cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"Y16 "))
            cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

            img = cap.read()
            kelvin_matrix = img[:-3 , :] / 100

Now since the returned image is a (63,80), do I need to ignore the last three rows in order to get the kelvin temp matrix, i.e kelvin_matrix = img[:-3, :] As you mentioned to ignore the telemetry values and focus on the pixel level temperature.

Since the input is a 16-bit matrix, I divide the matrix by 100 and as you said, it should make sense. Do I need to take account of distance as well or the radiometric camera should offset that?

In essence, what I am trying to do is get the image as input from the Lepton with pixel-level temperatures. I would then mask the input image in certain sections and calculate the average temperature of the mask, based on the pixel values.

[jveitchmichaelis]

Yep that should work - generally assume Kelvin because it avoids negative numbers, so *0.01-273 should give you something sensible. Oops my bad though, there are only 3 rows so it's A, B and C. I updated my comment above... It would be good to know if you can extract the proper telemetry values by using e.g. np.flatten(image[-3:-1,:])?

Do I need to take account of distance as well or the radiometric camera should offset that?

Do you mean distance to the target? This is a tricky question. 99% of the time the uncertainties on lots of other things like the emissivity of the object, ambient temperature, etc is going to dominate over distance. The main thing to worry about is whether the thing you're imaging is in focus. The camera itself has no way of measuring distance, it will just have some typical value in the onboard calibration. If you really need to correct for it, you would need to change the image output to raw counts and do the calibration yourself.

[ben-kolber]

Running the np.flatten does return a matrix with values different from the Kelvin tmp's. I am quite new to thermal imaging so I don't honestly fully understand the meaning of the telemetry values! Should I just be looking at the order of the numbers and what they represent on the data sheet for the Lepton 2.5?

For the second part, I know exactly what I am measuring (pots and pans) for emissivity, and I have several other sensors to give me the humidity and even the distance to the object. I'll look into ways to take them into account as well with emissivity for silver / Teflon cookware.

I guess once I mask out the cooking area of the pan/pot, I can classify it to being silver / Teflon and apply the emissivity into the temperature calculation.

Thanks a lot for the help man! really saved me several hours of headaches!!!

[jveitchmichaelis]

More a sanity check for me so I can patch this in the future. Would you mind printing the values you get if you unpack the struct? You can look in the flirpy source to see which index means what. One thing you might be interested in is the sensor (FPA) temperature.

As for distance, the simplest way to check this is to measure it, I guess. Put a decent rtd sensor on the surface of the pan and see if you get the right temperature from the Lepton.

No problem, glad this helped!

[ben-kolber]

Yup! Currently in NYC time so will get to bed soon, but will continue to work this through and send you some updates. You can be sure of that!

Thanks again, truly!

[ben-kolber]

Hey There!

Hope you are doing well! With regard to your last point on taking emissivity into consideration in the temperature calculation, is there any way to manually input it into the Lepton's internal temperature calculation? I can't seem to find a way to take it into account (given that I know the ambient temperature) when getting values from the Lepton. I am trying to use it to calculate the temp of a silver pan.

Thanks for your time :)

[jveitchmichaelis]

Hmm I'm not sure there is. You might be able to take the raw count values, then do the temperature version manually but you'd need to extract the Planck coefficients to do that conversion. On some other cameras like the Tau this is possible - not sure about the Lepton though.

[jveitchmichaelis]

Will add a separate issue for the frame size stuff