Splitting Doas/Ifit and Pyplis config settings

[ubdbra001]

Wanted to start an issue to get your opinion on some my thoughts on this @twVolc.

I'm inclined to split these parts of the config, but I want to make sure I haven't overlooked anything.

So, my understanding is that running pyplis is dependent on the info produced by doas/ifit (I'll refer to both of these as doas from now), so for the end user there are three scenarios:

• Just run doas
• Run doas and then run pyplis
• Load doas and then run pyplis

My first question is: is this correct? My second question is: do you think you need the doas config settings for an analysis to be reproducible?

[twVolc]

I guess there's a 4th scenario - simultaneously running doas and pyplis, which is what will be happening when real-time processing is occurring, i.e. all data will be coming in simultaneously, it will be processed side-by-side and this will allow real-time calibration of the SO2 images with the doas data. Other than that, yes I'm pretty sure you're correct on the above.

My second question is: do you think you need the doas config settings for an analysis to be reproducible?

I guess this depends on your starting point of the analysis. But fundamentally yes you need doas config settings to get from raw data all the way through to processed output. If you're just loading the doas data that has already been processed then no you wouldn't need any doas config settings, they won't affect the pyplis processing in any way (I'm pretty sure).

[ubdbra001]

it will be processed side-by-side and this will allow real-time calibration of the SO2 images with the doas data

That's a good point, in this case would the doas data be processed before the SO2 images (I guess yes?)? and would doas have to be re-run for different intervals (e.g. different days)?

I guess this depends on your starting point of the analysis. But fundamentally yes you need doas config settings to get from raw data all the way through to processed output. If you're just loading the doas data that has already been processed then no you wouldn't need any doas config settings, they won't affect the pyplis processing in any way (I'm pretty sure).

On this I'm trying to be as flexible as possible, and so I'm thinking of a processing run as from the raw data to final output, with all the intermediate steps (doas result, cailbration stats, etc) being saved along the way.

[twVolc]

Sorry to be slow to this, I've just realised I didn't repond. In an ideal world the DOAS data would be processed before the SO2 camera images - or at least that is of course required for us to eventually calibrate the SO2 camera imagery. But in reality we can't be certain what data will arrive first. This is probably something that you'll encounter and may need to work on when doing the real-time processing: how do we deal with data arriving at different times, there will have to be some kind of backwards check to see what we already have and what images can therefore be calibrated and what ones need to wait for more DOAS data before they can be calibrated. As the DOAS spectra files are so much smaller than the images, I would imagine in practice we almost always will receive the spectra before the images for any given time, but I don't know if this can be guaranteed.

The DOAS results will need to be reset for different days, as the calibration is only valid for that specific day and over a specific time interval within that day - that interval is set by the user as the time before DOAS data points are removed; typically this would be somewhere between 30-60 minutes.