melix / astro4j

Astronomy libraries for Java
Apache License 2.0
28 stars 5 forks source link

Helium D3 auto-misdetection #314

Closed vnp85 closed 2 months ago

vnp85 commented 2 months ago

First of all, thank you for your software.

Bug description I open a ser recording of a spectroheliogram around the helium D3 line. Quick mode misidentifies the spectral line, and the resulting Helium D3 extraction is just noise.

Perhaps make this into a feature request Make the number, next to the Helium D3 editable.

Reproduction Ver. 2.5.0 Use a recording that has no Sodium D lines in it, and the iron etc. in the vicinity are slightly obscure. Run quick mode. If the Helium D3 option shows up, chances are, it misses the line. In the uploaded image it is at -66 pixels from the detected spectral line, while JSol'Ex proposes -142.

Expected: the Helium D3 number to be around -66 in this case Observed: the Helium D3 extraction, if available at all, is -142 or so

I uploaded the average png, the helium line is around the middle of the image, two absorption lines below, one above it.

image

melix commented 2 months ago

The most likely explanation is that you have incorrectly set the "pixel size" in observation details. Can you double check?

vnp85 commented 2 months ago

This is my log, the pixel size looks correct.

` 13:41:41.234 [INFO] Output directory set to D:\2024-06-24\Sun-He-rp 13:41:41.236 [INFO] File 2024-06-24-0749_3-Sun-He-rp.ser 13:41:41.236 [INFO] Found Sharpcap metadata. Camera ZWO ASI678MM Binning 1 13:42:21.765 [INFO] Dimensions of raw image determined : 3696x5309 13:42:21.808 [INFO] Processing will require approximatively 673.67MB of disk space 13:42:21.978 [INFO] Detected spectral line Iron (Fe I) with binning 1 (for pixel size 2.00µm) 13:42:21.984 [INFO] Starting reconstruction... 13:42:21.984 [INFO] Distortion polynomial ax3 + bx2 + cx + d = 0

`

vnp85 commented 2 months ago

PS: it did work well with this average, containing a prominent sodium line image

melix commented 2 months ago

Would it be possible for you to share the SER file?

vnp85 commented 2 months ago

Yes, give me a sec.

melix commented 2 months ago

If the pb is that JSol'Ex confuses the Fe I line with Sodium D2, you should be able to select one or the other explicitly.

vnp85 commented 2 months ago

Letting the user edit the proposed number and method of extraction, could make it possible to inspect other lines as well.

Helium D3 (method: subtraction) [pixel distance] Some line (method: as is) [pixel distance]

Some line: like hydrogen epsilon, near CaH

So this is the ser with the mis-detection. One can actually see, in the flash spectrum, the magnificent proms visible today, so there's the Helium line, no doubt. (The hosting server is a raspberry, a bit slow on the bandwidth)

(~10 GB) http://narnia.go.ro/seagate2tb/2024-06-24/2024-06-24-0749_3-Sun-He-rp.ser

vnp85 commented 2 months ago

Here's the prom, frame extracted via AS4 and gamma applied

image

melix commented 2 months ago

Thanks, I'll take a look. 10GB SER file wow :) Regarding editing pixel distance, in this case it is easier to use scripts. It doesn't really make sense to add a parameter here since it is designed to work automatically. If it fails, then it's a bug that I should fix (or misconfiguration). You actually have one sample script which does Helium line extraction that you can tweak when you open the ImageMath mode (see also https://melix.github.io/astro4j/latest/en/jsolex.html#imagemath).

vnp85 commented 2 months ago

Thank you, I will take a look at the ImageMath helium alternative. The hydrogen epsilon isn't a big deal, I can just manually type a pixel shift value.

Off topic: do you know of a script that, given an image input (a snapshot from Sharpcap) can come back with wavelengths and lines seen (don't want to reinvent warm water)? Ideally with no parameters but reference data like snapshots around known spectral lines, like the Fraunhofers, either the Sodium or the Magnesium should be good starting points...

melix commented 2 months ago

Off topic: do you know of a script that, given an image input (a snapshot from Sharpcap) can come back with wavelengths and lines seen (don't want to reinvent warm water)? Ideally with no parameters but reference data like snapshots around known spectral lines, like the Fraunhofers, either the Sodium or the Magnesium should be good starting points...

Yes, you can use INTI Map for this. It's still experimental AFAIK and I've got varying results on it.

vnp85 commented 2 months ago

Off: Regarding the spectral line question, and see also the Helium extraction question -- I identified the line I was looking for as the He I at 447.15 nm, exactly as planned my session, but wasn't sure about the line. However, the result is ugly -- even though intellectually cool.

melix commented 2 months ago

Ah right, but JSol'Ex automatic processing is only available for the D3 line. You can do exactly what you want with the scripts though. In particular look at the find_shift function.

vnp85 commented 2 months ago

I'll look into it.

I hope you find the bug around the D3 line (I think it's going to be a pattern thing, though my Java knowledge is more than rusty). Visually, having seen this on my screen before switching to slit view, was mesmerizing.

image

melix commented 2 months ago

Is this done with a Sol'Ex or another SHG? Seems that the resolution if fairly high. From what I can tell, your crop window does not contain the Fe I line, which explains that JSol'Ex mismatched.

image

vnp85 commented 2 months ago

Very same Sol'Ex and 678MM with 2um pixels, different crop. I double checked.

I have a vertically full size still, a larger video that does contain a Sodium D line, and then several that don't contain the Sodium D line.

The one on the left, with the Sodium line works, and extracts Helium, the one on the right doesn't, albeit it is a crop of it.

image

The left one's logs

`12:29:20.685 [INFO] Output directory set to D:\2024-06-24\Sun-He 12:29:20.686 [INFO] File 2024-06-24-0741_6-Sun-He.ser 12:29:20.686 [INFO] Found Sharpcap metadata. Camera ZWO ASI678MM Binning 1 12:30:01.463 [INFO] Dimensions of raw image determined : 3696x2925 12:30:01.476 [INFO] Processing will require approximatively 371.16MB of disk space 12:30:01.816 [INFO] Detected spectral line Sodium (D2) with binning 1 (for pixel size 2.00µm) 12:30:01.845 [INFO] Starting reconstruction... 12:30:01.845 [INFO] Distortion polynomial ax3 + bx2 + cx + d = 0

The right one's

` 13:41:41.234 [INFO] Output directory set to D:\2024-06-24\Sun-He-rp 13:41:41.236 [INFO] File 2024-06-24-0749_3-Sun-He-rp.ser 13:41:41.236 [INFO] Found Sharpcap metadata. Camera ZWO ASI678MM Binning 1 13:42:21.765 [INFO] Dimensions of raw image determined : 3696x5309 13:42:21.808 [INFO] Processing will require approximatively 673.67MB of disk space 13:42:21.978 [INFO] Detected spectral line Iron (Fe I) with binning 1 (for pixel size 2.00µm) 13:42:21.984 [INFO] Starting reconstruction... 13:42:21.984 [INFO] Distortion polynomial ax3 + bx2 + cx + d = 0

vnp85 commented 2 months ago

My spectral resolution is high, I can easily see the rugged inside, the splitting of the Calcium lines, for example -- random frame of the Sol'Ex in Ca H

image

and right next to it, hydrogen epsilon

image

vnp85 commented 2 months ago

So, is that particular line's presence a must? Marked on the image

image

melix commented 2 months ago

Yes, as I said, auto-detection of the He D3 line will only work if either the Fe I or So D2 lines is visible in the crop window. If it's not, JSol'Ex may confuse other lines with the Fe1/D2 lines. This doesn't prevent you from using a different reference line, but in this case you have to go through scripting.

vnp85 commented 2 months ago

Thanks for clarifying.

The problem, perhaps for others as well, will be that the small pixels increase the vertical distance between the line candidates, and the data penalty, being additional area, is a square. I have 5 pixels (of 2 μm) where others have 4 (of 2.4 μm).

melix commented 2 months ago

The problem isn't much the distance between the lines, it's having a reference to compare with. However, you may want to use binning to reduce the size of your SER files. Using bin1 works if you have excellent observing conditions.

vnp85 commented 2 months ago

The computational penalty for having to include a pixel-distant spectral line is the readout speed / framerate in particular, not the video size per se. If there is curvature in the line, it makes things much worse.

Binning may or may not be a solution, as spectrally I feel like I have wonderful resolution.

vnp85 commented 2 months ago

(I just checked the difference between CaH on band and plus-minus 3. Binning is out of question :D)

stefanoaz commented 2 months ago

Hi Cèdric: I've been out of town a week. Now, when I use V2.5.1, the autodetection for He_d3 with a Sodium_d2 line is broken, it only produces the d2 line result. This autodetection function worked nicely in V2.4.9. Did a procedure change? Did the code change?

Steve

stefanoaz commented 2 months ago

Here's part of the (new) problem. Even though I have a binning of 1 selected in the observation details, and both the Sodium d2 line and the He d3 line are in the frame, the run log shows this:

"Detected spectral line H-alpha with binning 2 (for pixel size 2.00µm)"

There must have been a change in the code which has somehow hard-coded bin 2 when Autodetect is used for the wavelength.

vnp85 commented 2 months ago

I have also experienced problems with the 2.5.1 release and Sodium, (namely getting stuck with the sodium image processing), I think that issue is a bit different than what I exposed here.

stefanoaz commented 2 months ago

In the V2.4.9 version, similar frames produced this proper comment in the log:

11:11:01.420 [INFO] Detected spectral line Sodium (D2) with binning 1 (for pixel size 2.00µm)

I have verified that manual pixel offset does produce a helium image in my new SER files. Nothing has changed on my end since V2.4.9.

melix commented 2 months ago

There must have been a change in the code which has somehow hard-coded bin 2 when Autodetect is used for the wavelength.

There is no such thing. You have to be aware that spectral line detection is a sensitive algorithm, which works reasonably well, but can fail. Therefore, unless the same file worked in 2.4.9 and fails in 2.5.1, this cannot be considered a regression. I personally regularly experience failures in auto-detection, and experienced that even in 2.4.9. However, should the detection fail, you always have the ability to set explicitly the line that serves as reference in the cropping windows (e.g Sodium D2 or Iron Fe I).

The binning detection only triggers if there's no .txt metadata file alongside the SER file. Is this the case?

vnp85 commented 2 months ago

The binning detection only triggers if there's no .txt metadata file alongside the SER file. Is this the case?

Is JSol'Ex able to detect the cameraSettings.txt even when the names are a bit off? Here's an example from my archive, it happens when I record with the SHG: opening the record window some 30 seconds before the actual recording places the first frame of the ser file.

image

melix commented 2 months ago

Is JSol'Ex able to detect the cameraSettings.txt even when the names are a bit off?

No, the file names must match.

stefanoaz commented 2 months ago

Okay - I can verify that V2.5.1 does properly autodetect a SER file that was previously correctly autodetected by V2.4.9. So my new files from today might be a bit off for a sensitive algorithm - I was fighting with high cirrus clouds in the early part of the morning. The d2 and d3 lines were clearly there, but slight cirrus might have caused an issue.

I do not understand why, if I explicitly declared Bin 1 in the observation details, the program can ignore this in the autodetect when autodetect fails. The camera was set to bin 1. I have metadata in the metadata tab, but I'm not sure of the file you are referring to for metadata - I only see the run log in the window manager. Is there a location for metadata info?

melix commented 2 months ago

I do not understand why, if I explicitly declared Bin 1 in the observation details, the program can ignore this in the autodetect when autodetect fails. The camera was set to bin 1. I have metadata in the metadata tab, but I'm not sure of the file you are referring to for metadata - I only see the run log in the window manager. Is there a location for metadata info?

Right let me clarify. There's the acquisition software metadata file, which is the .txt file which lives alongside the SER file, and is written typically by SharpCap. If this file is found, then the camera name and binning are read from this file automatically, and assumed correct. Therefore it doesn't matter what you set in the observation details, it's this metadata file which tells us what to use.

The metadata tab is something specific to JSol'Ex, and shows the different operations which were executed on an image, the pixel shift, etc...

vnp85 commented 2 months ago

I doublechecked it, kind of a false alarm. It's when it's WinJupos filenames checked in the sharpcap options, AND also interference from my folder-watching renamer, as Sharpcap intends to rename the settingsTxt when it's done recording. Thus, unrealted (and in fact a bug in my ersatz-obsi software stack). As a side effect, in my case it could be that it is this that hinders the binning detection ineffective, although in the opening dialog of JSol'Ex, I manually set the binning and pixel size to the right values.

stefanoaz commented 2 months ago

Hi Cedric:

So - the answer to your acq metadata question is yes - the acq metadata .txt file is always present alongside the ser file in the same directory, for both the cases that fail autodetect and those which pass. My capture software is Firecapture. Here is the file text for a failing autodetect which mistakenly indicates bin2:

FireCapture v2.7.14 Settings

Camera=QHY5III678M Filter=L Profile=Sun Filename=Sun_073954.ser Date=062724 Date(UT)=062724 Start=073943.076 Mid=073954.757 End=074006.439 Start(UT)=143943.076 Mid(UT)=143954.757 End(UT)=144006.439 Duration=23.363s Date_format=MMddyy Time_format=HHmmss LT=UT -7h Frames captured=2988 FPS (avg.)=127 File type=SER Binning=1x1 Data=Mono ROI=3472x340 ROI(Offset)=200x484 Shutter=4.900ms Gain=38 (15%) Offset=0 Brightness=0 Contrast=5 HighSpeed=off USBTraffic=0 Histogramm(min)=0 Histogramm(max)=5 Histogramm=0% Noise(avg.deviation)=n/a Limit=10000 Frames

---------- ROI offset positions ---------- Frame 1: 200x484

melix commented 2 months ago

Thanks, but as I said, it's not because you have the metadata file that autodetection will succeed. IF there's a metadata file, then the binning that it contains is deemed accurate, but that's pretty much it.

stefanoaz commented 2 months ago

Did you test this with Firecapture? The Firecapture metadata is declaring bin 1 as the string "1x1". Not sure if that's what Shartpcap says for Bin 1. I thought jSolex was using the metadata. True?

It's declaring bin2 even though the metadata says bin 1x1.

melix commented 2 months ago

Yes I tested this with Firecapture. It's fairly easy to tell if the metadata file was found, the logs will contain an entry such as:

Found Firecapture metadata. Camera ZWO ASI178MM Binning 1

stefanoaz commented 2 months ago

Here's a logfile from a successful autodetect session. It does not contain the entry you mention, even though the acq txt file is there, beside the ser file on the same level in the directory.

14:23:35.275 [INFO] Output directory set to /Volumes/T7/Firecapture/Sun/061824 14:23:35.275 [INFO] File Sun_065905copy.ser 14:24:39.048 [INFO] Dimensions of raw image determined : 3500x2973 14:24:39.050 [INFO] Processing will require approximatively 535.87MB of disk space 14:24:39.061 [INFO] Starting reconstruction... 14:24:39.061 [INFO] Distortion polynomial ax3 + bx2 + cx + d = 0

melix commented 2 months ago

what's the file name of your .txt file? It should be Sun_065905copy.txt

stefanoaz commented 2 months ago

It is. And in no other of my autodetect or any other sessions, is a mention of the metadata file found in the logfile. Here's the directory structure...

Screenshot 2024-06-27 at 15 14 41
stefanoaz commented 2 months ago

Text from a different day, different successful autodetect, metadata file present but no mention of it in the log:

10:06:53.991 [INFO] Output directory set to /Volumes/T7/Firecapture/Sun/061724 10:06:53.991 [INFO] File Sun_071422.ser 10:07:54.932 [INFO] Dimensions of raw image determined : 3424x2934 10:07:54.934 [INFO] Processing will require approximatively 517.35MB of disk space 10:07:55.042 [INFO] Detected spectral line Sodium (D2) with binning 1 (for pixel size 2.00µm) 10:07:55.042 [INFO] Starting reconstruction... 10:07:55.042 [INFO] Distortion polynomial ax3 + bx2 + cx + d = 0

melix commented 2 months ago

I think I have found the bug with parsing Firecapture metadata. Can you try the development version?

stefanoaz commented 2 months ago

Cèdric: I'll try it tonight and report back. Thanks for checking it out.

stefanoaz commented 2 months ago

Cédric:

Results were excellent with the parsing fix in V2.5.2. Two things happened.

  1. The log file showed: "Found Firecapture metadata. Camera QHY5III678M Binning 1"
  2. The SER file that formerly had a malfunction in the autodetect, now properly autodetects and produces clear He_d3 images. It identifies the main line as Sodium D2 instead of H-alpha, and it detected the correct offset of -249.9 pixels.

Here's the log:

Output directory set to /Volumes/T7/Firecapture/Sun/062724 File Sun_073848.ser Found Firecapture metadata. Camera QHY5III678M Binning 1 Date 2024-06-27T14:38:37.448 SER file contains 2904 frames Color mode : MONO (2 bytes per pixel, depth = 16 bits) Width: 3472, height: 340 Solar parameters Carrington rotation = 2285, B0 = 2.51°, L0 = 30.39°, P = -4.02° Computing average image and detecting limb Dimensions of raw image determined : 3472x2903 Processing will require approximatively 519.06MB of disk space Detected spectral line Sodium (D2) with binning 1 (for pixel size 2.00µm) Starting reconstruction... Distortion polynomial ax3 + bx2 + cx + d = 0


Here's the info in the jSolEx Metadata tab:

Image

Pixel Shift = -249.90 pixels Transformation history

Ellipse parameters C(x,y) = ax² + bxy + cy² + dx + ey + f = 0

Carrington rotation = 2285, B0 = 2.51°, L0 = 30.39°, P = -4.02° PixelShiftRange = PixelShiftRange[minPixelShift=-280.0, maxPixelShift=20.0, step=30.0] Reference coordinates:OFFSET_X[490.32778407505384] OFFSET_Y[-91.00803874030976] MARKER[0.0]

stefanoaz commented 2 months ago

With this encouraging result - I will try some Bin2 tomorrow on the Helium and the Ca-H lines, if the clouds cooperate. Observing in Phoenix Arizona in the summer must be done before 0900 local time, or it gets too hot outside.

Merci Cèdric!

stefanoaz commented 2 months ago

Followup:

As of this morning with V2.5.2dev, Bin2 mode works great on He_d3 and Ca-H, where I do not need the full camera resolution and large file size. jSolEx picks up the metadata info and autopopulates the camera fields in the Observation Details form - I don't remember it doing this before. Significant bug fix - thanks.