Errored Coronagraph Imagery

[Jesterface23]

Hello, I have a list compiled of images that may need to be reprocessed, Imagery List.xlsx

STEREO A and B C2 imagery has whiteout images. SOHO C3 imagery has a mix of of all sorts of issues compared to different C3 imagery sources.

SOHO C3 imagery is missing for all of 2009/12/31.

STEREO A throws an error for the timestamps between 2022/12/25 06:39Z and 07:24Z org.helioviewer.jhv.io.LoadLayer.request - {"frames":5,"message":null,"uri":"jpip://helioviewer.ias.u-psud.fr:8080//movies/STEREO_A_SECCHI_COR2_white-light_F2011-12-25T06.31.30Z_T2011-12-25T07.31.30ZL.jpx"} org.helioviewer.jhv.io.LoadLayer$Callback.onFailure: Duplicated time stamps: jpip://helioviewer.ias.u-psud.fr:8080//movies/STEREO_A_SECCHI_COR2_white-light_F2011-12-25T06.31.30Z_T2011-12-25T07.31.30ZL.jpx

Also to note, for STEREO A from epoch 1331597328009 to 1331682853011, 1524036253006 to 1524059653012, 1524068653006 to 1524074053011, and 1652783023007 to 1653177223016 all show distances of 0 and do not render in JHelioviewer. STEREO B has the same from 1331597369222 to 1331682894224.

[dgarciabriseno]

Hi, sorry for taking so long to look into this. I confirmed that STEREO after time 1331597328009 (I used 2012-03-13T08:22:00 - 2012-03-14T08:22:00) don't render in JHelioviewer, they do render in helioviewer.org, most likely something in the metadata.

I did get the same error on 2011/12/25 06:39Z, I can look into that.

@wafels is out this week, but next week I'll have him take a look at the missing images.

@Jesterface23 For the spreadsheet you attached, are the dates for any specific source? (STEREO A/B/SOHO?). As a side note, we don't have a lot of bandwidth for repairing images, it's a pretty involved process so it will take a while to get around to.

[dgarciabriseno]

That patch fixes the Duplicated time stamps: but unfortunately jpx files are cached and I can't easily pick and choose which ones to clear, so it's possible you hit a cached jpx that has this error. If this happens, just change the start time or end time by 1 second and it'll make a new jpx. Cache is based on start/end times.

[dgarciabriseno]

@bogdanni Can you look into why some images won't render in JHelioviewer? Query STEREO-A COR2 on dates 2012-03-12T04:00:00 - 2012-03-14T00:00:00 The image boundary where the rendering breaks is here: Rendered Correctly - https://helioviewer.org/jp2/COR2-A/2012/03/12/white-light/2012_03_12__23_39_00_009__STEREO-A_SECCHI_COR2_white-light.jp2 Not Rendered - https://helioviewer.org/jp2/COR2-A/2012/03/13/white-light/2012_03_13__00_08_15_008__STEREO-A_SECCHI_COR2_white-light.jp2

[bogdanni]

The second image renders in my copy, you must have an old version of JHV, please always try with the latest available. The root issue is that the distance to Sun (DSUN_OBS) is 0 in the file metadata. Latest versions of JHV place the observer at Earth position if DSUN_OBS < 1 Rsun.

[bogdanni]

The best would be regenerate those JP2 files if there are updated FITS files with correct metadata.

[bogdanni]

I've committed a change to log those bad DSUN_OBS values, will be in version 4.4 or later. The log can be consulted via menu Help>Show Log...

[dgarciabriseno]

Yep, I am using an older version of JHelioviewer, I need to update. Thanks for clearing that up.

[Jesterface23]

@dgarciabriseno They were a mix of ROB, IAS, or GSFC, depending on which had the larger file size over the span of 5 days.

Most of SOHO C3's imagery issues would be comparing the below to the Helioviewer imagery from the start to the end of the day, https://iswa.ccmc.gsfc.nasa.gov/IswaSystemWebApp/index.jsp?i_1=15&l_1=40&t_1=270&w_1=500&h_1=500&s_1=2013-10-10%2022:06:00.0_1_75_3

Given the process is going to take a while, one thing I've wanted to do is learn and figure out how to make the image files.

[dgarciabriseno]

Thanks, I see the dark spots that shouldn't be there. I agree these seem like they need to be reprocessed. @wafels has more info on how they're processed. I haven't worked directly with processing images, I just know they're processed with IDL which requires a license to run.

[wafels]

The Helioviewer JP2 creation process is based on the processing pipelines used by the instrument teams. It is not identical to those processes. This means that images on Helioviewer can be different from those from the instrument teams. If there is a breakdown in getting data from the instrument teams (for whatever reason) then it is possible that the instrument teams have images and Helioviewer does not. We measure how many images we have as a function of time using plots like these:

https://api.helioviewer.org/statistics/bokeh/coverages/SOHO_coverage.html

As you can see there are many days and weeks where we don't have image data. This could be due to failures in the Helioviewer processing pipeline, or there was some issue with the instrument that meant that no data was taken. On some days an instrument may not have been operating for known reasons (calibrations, for example) and so there are fewer images than in nominal operations, or even none.

We can use the coverage statistics to create a more general purpose gap-filler, but this has not been done yet. This task - which would be project by itself - has been flagged as an issue here: https://github.com/Helioviewer-Project/api/issues/50 .