92es / Thermal-Camera-Redux

Topdon TC001 (and clone) thermal camera app to read and display live and offline thermal data
38 stars 9 forks source link

Question about commandline options and additional features #7

Closed PMKrol closed 1 month ago

PMKrol commented 1 month ago

Hi!

I think this software is something I've been looking for my academic research. I'm using MaAnt Super Ir Cam and it seems to work way better than producers software.

I need to capture camera output for further OpenCV analyses (I'm capturing tool temperatures while drilling).

I hoped to have something like this (some bash script):

Lots of way to do this, so I'm focusing now on capturing video. I would like to do (for automation) few things:

            takeRecording = 1;
printf("\n%s-record [prefix] is coming soon ...\n%s", BLUE_STR(), RESET_STR() );
            if ( hasNext && validatePrefix( argv[ next ]) ) {
                recordingPrefix = argv[ next ];
                i++;
            }
        }

is this work in progress or stale?

I'll try to do those things by myself, but maybe those suggestions will be ok for You.

What I want to achieve: I have camera near to operating drill. I have in camera's view hot spot that is showing in some special moment. I'm capturing video, look every frame and check if there is hot hot spot, if there is - use this frame for analysis. I would like to extract: maximum temperature further temperatures on drill surface.

I prefer not to edit sources locally as this can be used by someone else.

Best regards!

edit2. best solution would be for me to save EVERY frame in some subfolder as PNG + txt file with some informations like for example max and min temp., scale, contrast and so on. This would be so awesome!

PMKrol commented 1 month ago

2hrs of messing around with the code and i have some success. I modified snapshoot() function so now it looks like this:

void snapshot( Mat *frame, const char *prefix, ProcessedThermalFrame *ptf) {

    //size_t strftime (char* ptr, size_t maxsize, const char* format, const struct tm* timeptr );
    time_t rawtime;
    struct tm * timeinfo;
    char now [128];
#define FN_SIZE 256
    char filename[FN_SIZE];
    char rawname[FN_SIZE];
    char txtname[FN_SIZE];

    char txt_info[FN_SIZE];

    time (&rawtime);
    timeinfo = localtime (&rawtime);

    strftime (now, sizeof(now), "%Y%m%d-%H%M%S", timeinfo);
    strftime (controls.snaptime, sizeof(controls.snaptime), "%H:%M:%S", timeinfo);

    if ( validatePrefix( prefix ) ) {
#define MAX_PREFIX (FN_SIZE - (sizeof(".png") + 1))
        // Make RPi compiler happy
        strncpy( filename, prefix, MAX_PREFIX );
        strncpy( rawname,  prefix, MAX_PREFIX );
        filename[ MAX_PREFIX ] = 0x00;
        rawname[  MAX_PREFIX ] = 0x00;
        strcat( filename, ".png");
        strcat( rawname,  ".raw");
    } else {
        // struct timeval tmnow;
        // char usec_buf[6]="00000";
        // //gettimeofday(&tmnow, NULL);
        // //sprintf(usec_buf,"%dZ",(int)tmnow.tv_usec);
        // sprintf(usec_buf, "%dZ", frame_cnt++);
  //
        // sprintf(filename, "%s/TC001%s.%s.png", startTime
        // , now, usec_buf);
        // sprintf(rawname,  "%s/TC001%s.%s.raw", startTime, now, usec_buf);

        char cnt_str[6]="00000";

        sprintf(cnt_str, "%05d", frame_cnt++);

        sprintf(txt_info,
                "Avg [C]: %3.2f\nMax [C]: %3.2f\nMin [C]: %3.2f\nContrast: %3.2f\nScale: %d\nMap: %s\nBlur: %d",
                ptf->avg.celsius, ptf->max.celsius, ptf->min.celsius, controls.alpha, MyScale, cmaps[controls.cmapCurrent]->name, controls.rad);

        sprintf(filename, "%s/F%s.png", startTime, cnt_str);
        sprintf(rawname,  "%s/F%s.tc0", startTime, cnt_str);
        sprintf(txtname,  "%s/F%s.txt", startTime, cnt_str);

    }

    //printf("%s", GREEN_STR() );
    //now(20231115-010012), filename(TC00120231115-010012.png), snaptime(01:00:12)
    //printf("\nnow(%s), filename(%s), snaptime(%s)\n\n", now,filename,controls.snaptime);
    //printf("%s", RESET_STR() );

    ofstream str (txtname, ios::out | ios::app);
    //str.open(txtname);
    str << txt_info;
    str.close();

    imwrite(filename, *frame);
    writeRawFrame( *threadData.rawFrame, rawname, 0, 0 );

}

It's not elegant modification but works for me. Now every frame is saved to folder named by start datetime with frame number. I should probably change it to time from start or something. Next to it raw file (with extension tc0) and txt file are saved. Txt looks like this:

Avg [C]: 23.66
Max [C]: 25.54
Min [C]: 23.10
Contrast: 1.00
Scale: 1
Map: None
Blur: 0

So now almost all my previous needs are not actual. But there are few things I still need or would be happy to see:

It would be cool if my always-snapshoot modification would be implemented as some swich, so I can put this github page in my academic paper (eventual). Also, my previous information about v4l options is incorrect (i forgot it was about some other device, not this camera).

Best regards!

p.s. I'm attaching my modified version of tc001 (there are few more changes). tc001.txt

92es commented 1 month ago

PMKrol,

I haven't touched the code for a few months.

A lot of what you mentioned in a prior post is already in the code.

You can set various configuration options at both compile and command line.

You can compile (see DEFAULT_FLAGS) to move the help menus to the left off the video and the temperature gradient widget off the screen to the right (based on your screen resolution, scaling factor and screen ratio). There are a couple images of this at the bottom of the 1st page.

Optional DEFALT_FLAGS in build_redux_rpi:

set temperatures on hud to Celcius. -DUSE_CELSIUS=1

move HUD and scale to be next to camera image, -DBORDER_LAYOUT=1

set temperatures scale, so I can define by commandline maximum temperature (i.e. 100C - white, and 20C - black).
-DDEFAULT_COLORMAP=N

add maximum and minimum text on hud, -DDEFAULT_FONT=N

-DROTATION=N -DDISPLAY_WIDTH=N -DDISPLAY_HEIGHT=N -DHUD_ALPHA=0.4 etc.

I do not send any commands to the camera because the vendor would not supply the proper control protocols and I did NOT want to risk bricking anyone's hardware.

You can also optimize the build for you hardware if you are running on a weak SBC or a stronger PC.

As for functionality, the offline mode for post processing and the scrollable horizontal and vertical thermal plotting rulers has been very helpful for my home projects.

Sincerely

GITHUB_BORDER_LAYOUT

In this image, the HUD menu partially overlays the image because of the scaling and display resolution restraints. It will adjust based on those constraints in real time.

Prototype Border 2

92es commented 1 month ago

PMKrol,

One of the TODO options was to add alarms to user defined temperature monitoring points with user specified actions. If a high or low alarm is triggered, it could take snapshots, run a shell script, etc.

Haven't implemented that yet.

92es commented 1 month ago

PMKrol,

One other thing, you can fork your own copy, make all the changes to you want and commit your changes to your fork that others will have access to. That is the beauty of open source !!!

92es commented 1 month ago

PMKrol,

Set the DISPLAY_WIDTH and DISPLAY_HEIGHT in the compile to your display resolution.

If you have a larger display, you can then use the scale and interpolation functionality to get better images (there are other filters as well). The HUD has a second help page and can be removed in its entirety.

You can also add more user temperature indicators to monitor different locations.

Your snapshot text dump code could be updated to dump ALL THE ACTIVE TEMP WIDGETS, not just the center of the screen.

I added the movable user defined temp widgets specifically for bench and fixed-mount usage.

Also try the movable thermal rulers. They plot 256 temps horizontally and 192 temps vertically in LANDSCAPE mode. Swap those numbers for PORTRAIT mode.

E.G. You could plot/graph the temps along the entire length of the drill bit.

Changing colormaps also helps depending on what color spectrum best depicts your use cases.

You can take a single snapshot, load the raw dump

in offline mode and then play with different colormaps and scaling options to see what works best for your requirements and then use those as your default startup/build options.

Hope this helps.

In user temp widget mode, you can add extra temp widgets.

GIT_HUB_USER_TEMPS

In thermal ruler mode, you could place a horizontal or vertical ruler over the length of the drill bit shaft and see how the bit heats up along its length.

GITHUB_PC_BORDER

PMKrol commented 1 month ago

Ok, ok. Slow downl please! ;)

I've forked Your code here: https://github.com/PMKrol/ThermalCamSnap, I modified my modifications, so now it is more elegant. To use continuous snap there is switch -contSnap.

I need some advise still:

I'll use those images for analyses in other software (Octave, some day), so PNG representation seems to be what I need - I'll be able to read temperatures wherever I need.

One more thing would be to mark „area of interest” and calculate average there, I'll try to figure this out some day...

Screenshot_20240605_124808

92es commented 1 month ago

What hardware are you running on ?

What is your display resolution ?

A text dump is a good idea. If I incorporate it in the main line, it will be incorporated in such a way to consider the rest of the data being harvested by the app as well as all of its presentation modes. Will also consider different types of controlled repetition. I have been looking for a lightweight JSON library to implement such things as well as a vehicle for loading and saving detailed configurations. A standardized format could be used for both features.

png's save without hud and scale (bordered layout), how to change that?

Just tried a snapshot with the HUD and Scale and it is working here. Video and graphics are composited into a final frame and then the final frame is dumped. If graphics are disabled, then the dump should not contain graphics rendered over the video.

Are you using the p key to take a snapshot ?

My coffee is getting cold. =(

TC00120240605-103706

is bone colormap start from 0 to 255?

All colormaps are 0-255. There are 2 types (stored in different formats) of colormaps, stock OEM and user defined.
OEM and user defined maps use separate manipulation and display APIs based on the way OpenCV implemented them. The camera's autoranging uses sub-ranges so it is a bit more involved than just selecting start/stop indexes. One of the reoccurring requests of customers of these cameras was to disable auto-ranging. I have implemented a couple of forms of disabling autoranging (fixed and dynamic growth). If you make changes to indexes, be aware other parts of the code and camera have dependencies that can be broken.

how to disable crosses?

Pressing the ? key will dump the key bindings to STDOUT. Transitioning to the HELP HUD screen using the h key will give a brief/shortened listing of the key bindings.

The h key toggles through what is displayed on the screen (HUD, Help, No HUD, No Graphics)
The o key cycles through ruler modes.
The right mouse button puts you back into user temp mode.  It also deletes added user temps.

Key Bindings:

a z: [In|De]crease Blur s x: +/- threshold from avg temp that contols min/max displays and ruler plot colors d c: Change interpolated window scale [camera native to fullscreen] f v: [In|De]crease Contrast g b: Cycle [for|back]wards through interpolation methods j m: Cycle [for|back]wards through Color[m]aps w : Cycle through single/dual[horizontal/vertical] [w]indow layouts 6 : Toggle between fullscreen and current scaled window size r : Toggle [r]ecording (.avi) 1 : Font 5 : Reset defaults p : Sna[p]shot (both .png and offline .raw) h : Cycle through overlayed screen data t : Toggle between Celsius and Fahrenheit y : Toggle Historgram filter (for gray scales) l : [Un]Lock camera's colormap auto ranging i k: Cycle [for|back]wards through locked auto ranging mapping methods 8 : Rotate camera 0, 90, 180, 270 degrees (Portrait and Landscape) 9 : Rotate display 0, 90, 180, 270 degrees (Portrait and Landscape) e : Toggle Freeze Frame on/off o : Displays and cycles through 5 temp ruler modes 3 : Ruler plot clip modes: none, outlier, below avg, above avg 4 : Ruler thickness - 1/5, 1/4, 1/3, 1/2, full height : Keypad Up/Down/Left/Right/Center(5) moves rulers : Left mouse adds user temps or moves rulers : Right mouse removes user temps and disables ruler mode / : Misc stdout help information q : Quit

92es commented 1 month ago

Update:

I assume you were using the -snapshot command line argument.

Grab a new copy of tc001.cpp. I moved the ( takeSnapshot ) section to after the display of the 1st frame to include the border layout.

Let me know if this works for you.

You will get a brief 1 frame display that I need to suppress in the future.

Hope this helps.

PMKrol commented 1 month ago

Grab a new copy of tc001.cpp. I moved the ( takeSnapshot ) section to after the display of the 1st frame to include the border layout.

Perfect! Thats it!

Pressing the ? key will dump the key bindings to STDOUT. Transitioning to the HELP HUD screen using the h key will give a brief/shortened listing of the key bindings.

Nope. I need to hide it by commandline or hardcode. I made it, bit dirty, but working, by modifying resetDefaults:

#if BORDER_LAYOUT
    controls.hud          = HUD_ONLY_VIDEO;
#else
    controls.hud          = HUD_HUD;
#endif

and modifying mainPrivate() in BORDER_LAYOUT section: } else /*if ( HUD_HUD == controls.hud )*/ { Previously enabling creating HUD image always except when HUD_HELP is selected.

So now by default in BORDER_LAYOUT HUD and SCALE shows up but without crosses.

My coffee is getting cold. =(

No good...

I need some advise still:

png's save without hud and scale (bordered layout), how to change that? **[done]**
is bone colormap start from 0 to 255?
how to disable crosses? **[done]**

About colormap... I still need some explanation. For example I have: F00100 with:

Avg [C]: 28.99
Max [C]: 31.41
Min [C]: 28.04
Contrast: 1.00
Scale: 1
Map: None
Blur: 0
Time: 1717669932893

I assumed then white (255) pixel will represent temperature of 31.41 C, black (0) - 28.04 C, so when I select pixel and want to know it's temperature, it will be: (max-min)/256*color + min. So, for example, pixel with color 127 will represent 29.71 C.

Is that correct or should I modify colormap and/or calculation method?

===== edit ===== how to extract part of image and calculate it's average temperature? I need only small direction on that, rest I'll probably be able to do by myself.

92es commented 1 month ago
I assumed then white (255) pixel will represent temperature of 31.41 C, black (0) - 28.04 C, so when I select pixel and want to know it's temperature, it will be: (max-min)/256*color + min. So, for example, pixel with color 127 will represent 29.71 C.

Is that correct or should I modify colormap and/or calculation method?

The raw temps are stored in Kelvin (See notes @ LeoDJ's conversion algo).

There are more Kelvin temps (greater than 256) then their are CLUT entries in a color map.

The PNG image does NOT accurately represent evenly distributed subranges of Kelvin temps to CLUT indexes in the color map (e.g. NOT 5:1, 10:1, 18:1, N:1, etc.). The camera AND the autoranging algorithm I implemented use "lossy image enhancing" filters to improve the visual appearance of the images (e.g. sharpen/smooth edges, etc.). There is NO way to map backwards from the PNG image pixels to the exact temps. The mapping is only 1 directional, not bi-directional. Think of lossy compression (MP3 vs FLAC).
This 1-way mapping also happens when you try to run in a Virtual Machine's guest OS, except it corrupts the original thermal data in the 2nd sub-frame making it impossible for the app to work properly.

There is also the scaling to take into consideration (as well as other filters Blur, Histogram, Interpolation, etc.). The PNG image is possibly scaled, the thermal data is not (if I remember correctly).

===== edit ===== how to extract part of image and calculate it's average temperature? I need only small direction on that, rest I'll probably be able to do by myself.

If you want exact pixel temps, you have to harvest them from the raw Kelvin video sub-frame (same way all the text and plotted onscreen temps are derived ( see processThermalFrame() , getRulerXPoints() and getRulerYPoints() ).

ProcessThermalFrame() harvests the min/max/avg values.
getRulerXPoints() grabs a row of Kelvin temps.
getRulerYPoints() grabs a column of Kelvin temps.

NOTE: These routines have been highly optimized to work on weaker hardware and can easily cause frame drops. Python is too slow to do what is done in this app without massive frame drops.

The raw thermal data is stored in a matrix in the 2nd sub-frame, 2-bytes per pixel (unsigned short) in row major order in landscape mode and is rotated according to current screen's rotation and Portrait/Landscape settings to keep the temp to pixel mapping.

PMKrol commented 1 month ago

The PNG image does NOT accurately represent evenly distributed subranges of Kelvin temps...

Got it! Although I think accuracy will be good enough (if scale is more or less linear), I'll focus on reading raw files with Octave for further analysis.

I see writeRawFrame():

        // Write header
    fwrite( &rows,    sizeof(unsigned short), 1, fp );
    fwrite( &cols,    sizeof(unsigned short), 1, fp );
    fwrite( &type,    sizeof(unsigned short), 1, fp );
    fwrite( &chan,    sizeof(unsigned short), 1, fp );

    // Write row/column data
    fwrite( &data[0], sizeof(unsigned short), (rows*cols), fp );

How to read this to get temp? Header is self explanatory. I understand this:

// Linear Parsing
// long kelvin = usKelvinPtr[0] + (usKelvinPtr[1] << 8); // LSByte + MSByte

But it's commented so it must be calculated somewhere else %)

NOTE: These routines have been highly optimized to work on weaker hardware and can easily cause frame drops. Python is too slow to do what is done in this app without massive frame drops.

Noted! There is FPS on HUD, is this FPS from camera or FPS not dropped? In my previous hardware setup I got up to 10 FPS. And it was quite qood. If I really get around 20 now, it is very good.

Currently I'm testing it on quite capable laptop, I was thinking to swich to something less powerfull, but I can go even more powerfull machine if needed. So I'm not very worried about optimaisations.

Many thanks for Your help and patience. This software will make my work LOT easier (even at this point).

92es commented 1 month ago

When I reverse engineered the 1st sub-frame to 2nd sub-frame image-2-temp matching to implement the auto-ranging freeze code, I made the same assumption as you. It didn't work. =)

I wrote some test code that kept track of every CLUT/Kelvin pairings, binned and sorted them. I discovered that the image enhancement routines could make a specific temp be represented all over the CLUT (no linearity, or consistency). That is why the rulers and text temps are much more representative of the thermal data.

I use unsigned short pointers and set their address to the start of the raw 2nd sub-frame and then use pointer arithmetic.

This sets the pointer to the beginning a specified row "y".

        // Start with Row Y offset
        unsigned short *usRowPtr = &((unsigned short *)(thermalFrame->datastart))[ y * TC_WIDTH ];

This sets the pointer to the beginning of a specified column "x".

        // Start with Column X offset
        unsigned short *usColPtr = &((unsigned short *)(thermalFrame->datastart))[ x ];

I think this is the conversion function you are looking for. Just pass in the contents of the short pointer. There is a thread on the EEVBlog, post #216 where LeoDJ posts his conversion algo. I announced this app there as well as its development progress.

float kelvin2Celsius(unsigned short kelvin) { // # LeoDJ's Kelvin conversion algorithm, post #216
        return ( ((float)kelvin / 64.0) - 273.15 );
}

The Topdon TC001 core and InfiRay P2 Pro core are rated at 25 FPS. I don't know if your camera uses the same 25FPS core.

If HUD is showing less than spec, the delta are dropped frames.

In offline mode, the code can run to hundreds of FPS (depending on hardware) because it is not read-blocked on the camera.

There is a STDIN thread that accepts commands from STDIN or a pipe. It processes commands up to 1 command per frame.

To benchmark and regression test, I run it via script in offline mode and slam it with thousands of commands running full FPS. It is a good way to catch race conditions as well as benchmarking.

On the single core/single threaded RaspPi's, there are compile options to pair down functionality (fewer temps in rulers, jump scrolling vs smooth scrolling) and resolution to run on those platforms. I am surprised it actually runs on those SBCs.

92es commented 1 month ago

Here is another function that may be of help, but it is computationally expensive (redundant incurred overhead for each lookup).

Pass in the X, Y coordinates and it returns the Celsius value.

float getCelsius(Mat *thermalFrame, int x, int y) {
        unsigned short *usStartPtr = &((unsigned short *)(thermalFrame->datastart))[0];

        ASSERT(( x < TC_WIDTH))
        ASSERT(( y < TC_HEIGHT))

        int linearI = (y * TC_WIDTH) + x;

        return kelvin2Celsius( usStartPtr[ linearI ] );
}
92es commented 1 month ago

One of the ruler optimizations was to do all of the ruler calculations in raw Kelvin instead of Celsius or Fahrenheit and then only convert when I had to display a text temp.

The rulers can plot up to 448 temps per frame, thus eliminating 448 conversions/frame was significant.

92es commented 1 month ago

You might be interested in the STDIN command input.

You could script periodic snapshots, change modes, scale, etc. and then shut down.

See test_driver.c which makes a script of regression commands.

PMKrol commented 1 month ago

When I reverse engineered the 1st sub-frame to 2nd sub-frame image-2-temp matching to implement the auto-ranging freeze code, I made the same assumption as you. It didn't work. =)

Once i tried to read camera image to something useful. I had this hunch that this green image is something I was looking for, but I couldnt work this out...

There is a thread on the EEVBlog, post #216 where LeoDJ posts his conversion algo.

=0

too easy...

The Topdon TC001 core and InfiRay P2 Pro core are rated at 25 FPS. I don't know if your camera uses the same 25FPS core.

Same here.

I made proof-of-concept script in Octave:

fid = fopen("F01487.tc0");

[val, count] = fread(fid, Inf, "int16"); 

rows = val(1);
cols = val(2);
type = val(3);
chan = val(4);

green = val(5+rows/2*cols:end);

newval = reshape(green, cols, rows/2)/64 - 273.15;

imin = min(min(newval));
imax = max(max(newval));

img_t = uint8 ((newval-imin)/(imax-imin) * 255);

imwrite(img_t, "tt.bmp");

And I'm happy to say it works! Screenshot_20240607_141420 F01487 tt F01487.txt F01487.tc0.txt

I get stable 24.8 FPS and CPU load is 25%.

Case closed! Many thanks.

92es commented 1 month ago

PMKrol,

You might want to change your Octave script to use uint16 instead of signed int. Kelvin temps are never negative.

Good luck in university !!!

PMKrol commented 1 month ago

True! I was at the beggining but I was getting weird values so I was trying diffrent things. Then I realised I was reading grayscale part of frame.

Btw. I can not stress out how thankful I am for Your help. Seriously... thanks!

92es commented 1 month ago

The frame buffer is actually 256 x 384 x 16-bits. 2 sub-frames of 256 x 192 x 16-bits.

LOL, some of the cloned brand's marketing teams lie about the frame rate of their cameras to boost sales.

They advertise 50FPS ( by counting each sub-frame as a full frame at 1FPS to falsely double the frame rate ).