search

SuperDARN / rst

Radar Software Toolkit (RST)
https://superdarn.github.io/rst/
GNU General Public License v3.0
22 stars 18 forks source link

Capturing data processing metadata in dmap files #479

Closed ecbland closed 1 year ago

ecbland commented 2 years ago

Discussion

At the DAWG meeting on 1 March 2022, we briefly discussed improving the way that data processing options are recorded in dmap files.

Category

Examples

  1. origin.command records the command used to generate the output file. However, this overwrites the command used to generate the input file:
make_fit -fitacf-version 3.0 20180323.0801.00.lyr.rawacf > 20180323.0801.00.lyr.fitacf
string  "origin.command" = "make_fit -fitacf-version 3.0 20180323.0801.00.lyr.rawacf"

trim_fit -st 09:00 -ex 00:02 20180323.0801.00.lyr.fitacf3 > lyr.trim.fitacf3
string  "origin.command" = "trim_fit -st 09:00 -ex 00:02 20180323.0801.00.lyr.fitacf3"
  1. If another fitting algorithm was used (lmfit, fitex), this information is not captured at all (except origin.command)
make_lmfit 20180323.0801.00.lyr.rawacf > lyr.lmfit
int "fitacf.revision.major" = 0
int "fitacf.revision.minor" = 0
  1. fitacf.revision.major and fitacf.revision.minor are not included in grid/map files, so it is impossible for a user to determine which version of FitACF was used to produce these files.

Note: I'm not planning to work on this right now; just opening the issue so that we capture the action item from the meeting. If someone is keen to take the lead with this, please "assign" yourself to this issue. We can discuss this at our next telecon meeting, and we can seek input from the PIs when we are ready.

egthomas commented 2 years ago
  1. fitacf.revision.major and fitacf.revision.minor are not included in grid/map files, so it is impossible for a user to determine which version of FitACF was used to produce these files.

We should also keep in mind that a grid file can contain information from at least 1 to any number of different radars and channels, each of which may have been processed using a different fitting algorithm. So a new field could be added to the GridSVec structure (and output grid file) which retains whatever new fitting algorithm string (or field) is also added to the fitacf-format files.

egthomas commented 2 years ago

Just repeating my suggestion from #494 that tdiff values also be recorded for each record in fitacf-format files.

egthomas commented 1 year ago

@ecbland I think we've accomplished the goal of this pull request with regards to fitacf-format files, right?

A few months ago I had a look at adding more metadata fields to grid-format files (e.g., algorithm, tdiff, etc) but it is complicated by the way that grid files are produced by make_grid via the gtablewrite library.

ecbland commented 1 year ago

@egthomas Yes, we've achieved the goal for the fitacf files. I'll close this now, and if we want to revisit the grid-format files another time then can open a new issue.