search

NuSTAR / nustar-clock-utils

Utilities for a precise timing calibration of NuSTAR data
3 stars 3 forks source link

Safe handling of incomplete data #28

Open bwgref opened 3 years ago

bwgref commented 3 years ago

This is an edge case, but occasionally when we run things the temperature database includes times where we haven't downloaded the actual data from the spacecraft yet and/or there are gaps in some of the instrument engineering files (i.e. the temperature measurements).

In nustar_clean_clock this resulted in an ambiguous crash (something about an underlying fitting routine nothing able to smooth the data because there wasn't any data to smooth). And in the clockfile generation this can lead to (apparently) poor modeling of the spline (because it's using missing data).

The easiest solution is: wait until the data are downloaded from the spacecraft and do it again. But it might be good to build in a sanity check when the various databases are loaded in to make sure that there are not substantial gaps in either the engineering files or in the clock/frequency files that we get from Bryce.

I have a set of files here that I can use to test a few things here (downloaded from the server for testing on my local machine). I'll see if there is an obvious place to put a sanity check that will raise an error like "Error: Databases don't match" or something to make it a little more obvious when this is the case. But I probably won't get back to this until January at this point.

bwgref commented 3 years ago

Marked as "enhancement" because it's a bug in the data and not in the software ;)