ATM1B qfit files without a header use hard-coded ITRF values

trey-stafford commented 2 months ago

The header usually contains (we think, see below) the ITRF for each granule, which is how we prefer to set the source ITRF. The files without a header have to fall back on a hard-coded list of ITRFs for various date ranges.

Relevant PR: https://github.com/nsidc/iceflow/pull/31

qfit file headers

For qfit files that do have headers, we extract the ITRF from a string in the header that we're not really sure much about. There is documentation about the qfit file format here: https://nsidc.org/sites/nsidc.org/files/files/ReadMe_qfit.txt, but it doesn't say much about what the header contains aside from:

each header record is filled with a string of ascii characters containing information on file processing history.

An example qfit file header is given below:

``` Wed May 19 16:09:29 2010 /Volumes/isbrae1/processing/Antarctica2009/qfit/4cT3_applanix/20091109 qfitftest_photo_flg_bias cqfitfttmp.cmd Input: 20091109_203148.atm4cT3.rangeExample.vld Output: 20091109_203148.atm4cT3.rangeExample.qi 0 0 0 / input,output device(1=tape,0=disk,-1=none), photo_pos file (0=yes,-1=none) 20091109_203148.atm4cT3.rangeExample.vld ./091109_aa_l12_cfm_itrf05_18may10_palm_roth_amu2 20091109_203148.atm4cT3.rangeExample.qi 0 9999999999 15 2 / st,sp,timebias(added to AOL data),0=scaled int output,1=binary output 0.0 0.0 0.0 / INS pitch, roll, heading bias from INS mount to airframe 22.1243 -0.4050 -0.0865 25.930 257.4421 179.683 44.5118 1 1 / nadang,pitmb,rolmb,rng bias,scan,hdg mb,beta,pitscale,rolscale -1.994 0.140 3.927 / x,y,z distance of AOLtm trk pnt from GPS antenna(x forwd,y along starboard wing,z down) 695137. 3130011.4 2000. 26. / ref point - DMS DMS M, local angle 1 2 1 /scanner orientation (0=reverse;1=normal); passive output (1=y,2=n); passv delay (usecs) 0 / compute wave mapping vectors (off-nadir angle & bearing) (0=no) ==== ATM4cT3 installation in NASA 817 DC-8 at Dryden/Palmdale in preparation for 2009 Chile/Antarctic. T3 INS transferred from Applanix 510 data provided by Rose Dominguez/NASA Ames antenna offsets from 090812 measurements on DC-8 biases modified 20100517 from altdify following estimated cfm 20100418 based on multiple Punta Arenas ramp passes setup using INS-to-airframe bias for cqfitftest_photo_flg_bias, however bias was not independently determined - range bias from ramp estimates (not ground test) was used to incorporate effects of INS-to-airframe rotation and lever arm ---------- cambot photo output file ENabled setup sm 100518 rerun jfs 100519 ******************************************* ******************************************* ```

The ITRF for this granule is determined from this string:

./091109_aa_l12_cfm_itrf05_18may10_palm_roth_amu2

which appears to be an input filename or program. At some point we thought this was enough to indicate the ITRF for the data, but there is no clear documentation (we take the _itrf05_ bit to mean this granule uses ITRF2005).

Hard-coded ITRF date ranges

The hard-coded ITRF ranges have questionable provenance. They come from valkyrie, and there's no documentation around how they were found.

The documentation for the source datasets isn't very helpful:

Data are given in geographic latitude and longitude coordinates. Data coordinates are referenced to the WGS84 ellipsoid. Reference frame is prescribed by the International Terrestrial Reference Frame (ITRF) convention in use at the time of the surveys.

It is not clear what the "convention" was during data collection.

I tracked down some old notes from the project about this, and they suggest that we figured out the ranges by looking at the headers, but nothing about how we determine the correct ITRF for files without headers. There appear to be at least a couple of dates worth of data that do not conform to expectations. 2022-11-22 and 2022-12-14 use ITRF97 instead of ITRF2000, which is used for all other dates between 2001-12-18 to 2007-05-11. So...do we know for sure that the files without headers actually use the ITRF indicated in the hard-coded list? Not sure.

Action

We should dig into this further, and perhaps reach out to PIs associated with these datasets for clarification. Questions we need to answer are:

Does the ITRF string we extract from the header actually represent the ITRF the source data are referenced to?
What is the ITRF for those qfit files that lack a header?

trey-stafford commented 2 months ago

@eigenbeam might have some additional notes or memories on this?

mstudinger commented 4 weeks ago

@trey-stafford please feel free to reach out to me for any help with your questions. I am happy to help and probably your best option at this point. I’ve worked with @eigenbeam on the question of reference datums for ATM many years ago when he developed valkyrie.

But to get your question: Not really important, but I think you are confusing QFIT and ICESSN data products/formats – and keep in mind these formats/data products were designed before 1993 and were not intended to be distributed by a DAAC… All QFIT files have headers, whether they are in the legacy flat binary format or the OIB era HDF5. As you say they contain information about the GPS trajectory that was used to reference the lidar data and that has the ITRF epoch in its file name. Back then ITRF epochs had two digit years, which was obviously changed to 4 digits after Y2K. I think you mean legacy ICESSN data without header (https://nsidc.org/sites/default/files/blatm2-v001-userguide.pdf). These files come with a summary sidecar file that includes the QFIT header with the trajectory (and ITRF) information you show in your post. The same summary file also lists the aircraft trajectory and ITRF epoch further down. Look for lines like that: Aircraft trajectory from File 070511_aa_l12_jgs_itrf00_19jun07_6138

As said, please feel free to reach out to me with questions. Glad to see that you are working on this.