search

DOI-USGS / ISIS3

Integrated Software for Imagers and Spectrometers v3. ISIS3 is a digital image processing software package to manipulate imagery collected by current and past NASA and International planetary missions.
https://isis.astrogeology.usgs.gov
Other
195 stars 166 forks source link

mrf2isis - Import PDS formatted MiniRF level1 or level2 image cube into Isis format #2280

Closed ascbot closed 5 years ago

ascbot commented 5 years ago

Author Name: Tammy Becker (Tammy Becker)

Original Assignee: John Bonn

Mentor: Trent Hare

The user test needs to specifically address importing the level2 (map) into ISIS

Note:

DoD:

ascbot commented 5 years ago

Original Redmine Comment Author Name: John Shinaman (John Shinaman) Original Date: 2017-04-18T20:33:13Z

App tests for level2 data DO NOT check the labels appropriately to check pixel offset.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Stuart Sides (@scsides) Original Date: 2017-05-01T22:59:58Z

User test plans are done. These have been converted to bugs, so they will be resolved.

ascbot commented 5 years ago

Original Redmine Comment Author Name: John Bonn (John Bonn) Original Date: 2017-05-12T22:35:43Z

PDS to ISIS mapping is defined in: /usgs/cpkgs/isis3/data/base/translations/pdsProjectionLineSampToXY.def We are looking to confirm the xOff and yOff values within this file.

From the file:


1.  This file is used by the ProcessImportPds class to define the offset and 
1.  multiplication factors to convert between  sample/line and x/y.
1.  
1.  The Keyword defines which keyword is searched for in the pds file.  The
1.  Pattern is then used to match the value in the defined keyword.  If there
1.  is a match,  the given values are used.
1.  The defaults used in ProcessImportPds are:
1.      xMult  =  -1.0
1.      yMult  =   1.0
1.      xOff   =  -0.5
1.      yOff   =  -0.5
#
1. Equations used are
#
1.     upperLeftX = xmult * (sampleOffset + xOff) * PixelResolution
1.     upperLeftY = ymult * (lineOffset + yOff) * PixelResolution
#
1.  Anytime a new group is added to this file, all 6 keywords need to be
1.  defined (Keyword,Pattern,xMult,yMult,xOff,yOff).
#

To find conversion for the data file you are looking at there will be an entry in the pdsProjectionLineSampToXY.def file similar to:

CHANDRAYAAN MINIRF MONOSTATIC RADAR

Group = Selection Keyword = "DATA_SET_ID" Pattern = "CHAN1-L-MRFFR-5-CDR-MAP-V1.0"

The "Keyword" should match an entry in the source data file and the "Pattern" is an associated regex-style expression to match on the Keyword's value.

In this example the source data file (FSL_CDR_LV2_00200_1Q.LBL) looked like this:

DATA_SET_ID = "CHAN1-L-MRFFR-5-CDR-MAP-V1.0"

Within the associated Group in the pdsProjectionLineSampToXY.def file there are

xOff    = -0.5
yOff    = -0.5

entries that may need to be changed.

In talking to Trent we need to do following steps:

1) assume overrides in pdsProjectionLineSampToXY.def are correct. 2) remove "STANDARD PDS" entry from pdsProjectionLineSampToXY.def - we assume default entry is not used. Document default values in pdsProjectionLineSampToXY.def 3) correct the default values for xOff and yOff in void ProcessImportPds::GetProjectionOffsetMults() 4) document pattern matching in pdsProjectionLineSampToXY.def (substring or exact string) 5) make test case for each PDS ingestion program 6) notify user if overrides are used. We need to specify how to do notification (entry in print file, new label, force user to use command line option to enable override)

Approach for fixing for each program (This is to ensure each program is using the new PDS defaults. The concern is some programs may have the defaults hard-coded.) : 1) Input programs should already have tests that use catlab to verify input. Verify a catlab test exists and check that it does not exclude the UpperLeftX or UpperLeftY in the input/*.DIFF files for the test. Mrf2isis has an example test for programs that do not use catlab. 2) change to new PDS defaults 3) confirm test cases fail with new PDS defaults (data set with overridden PDS values should still pass) 4) check failing test cases UpperLeftCornerX and UpperLeftCornerY are each offset by PixelResolution 5) update test case to pass.

mrf2isis is reading ProcessImportPds::GetProjectionOffsetMults. Note that there's an entry in the pdsProjectionLineSampToXY.def file to override defaults so mrf2isis is currently using correct values and test for the correct values. This means the mrf2isis will not need any change.

Note: Currently mrf2isis level2 tests do not compare the output cube labels to the truth cube labels. In reviewing the UpperLeft[X,Y] values in the truth file I noticed they did not match the output values. We know the output is generated using the correct entries in the pdsProjectionLineSampToXY.def file so I'm assuming the output values are correct and will update the truth values to match. A test will also be added to compare the output and truth labels.


1. CHANDRAYAAN MINIRF MONOSTATIC RADAR
Group = Selection
Keyword = "DATA_SET_ID"
Pattern = "CHAN1-L-MRFFR-5-CDR-MAP-V1.0"
xMult = -1.0
yMult = 1.0
xOff = 0.5
yOff = 0.5
End_Group
1. LRO MINIRF MONOSTATIC RADAR
Group = Selection
Keyword = "DATA_SET_ID"
Pattern = "LRO-L-MRFLRO-5-CDR-MAP-V1.0"
xMult = -1.0
yMult = 1.0
xOff = 0.5
yOff = 0.5
End_Group

Note that the other mission specific PDS import programs were tested as part of this ticket and have been updated, specifically tickets #4513, #4516, #4517, #4518. Once a deployment plan has been made the truth files for the hirdr2isis #4513, and crism2isis #4518 can be regenerated (after the code update ad .def update) and checked in - they have been fully tested for this change and the test programs that will test the change (specifically that check UpperLeft[X,Y]) are already present and tested. The ProcessImportPds::GetProjectionOffsetMults() and .def file changes will need to be checked in/updated at the same time.

During testing the mical program also came up as a candidate for PDS conversion (there was a Line and Sample change) but it was determined to not be releated to the PDS data but was instead a change in how tile sizes were calculated. The actual test ignored changes to these values (via the .DIFF file).

Deployment

The pdsProjectionLineSampToXY.def is in the data directory and in use by processors so any changes to it need to be coordinated with the processors. Also the other programs in #2244 will probably need to be deployed at the same time. The plan for deployment is as follows:

1) Deploy a new copy of pdsProjectionLineSampToXY.def as pdsProjectionLineSampToXY_V2.def file with the default PDS3 entry removed. The updated programs will reference the new pdsProjectionLineSampToXY_V2.def file and the ProcessImportPds::GetProjectionOffsetMults method will be updated with the correct defaults. The currently deployed programs will continue using the existing values in pdsProjectionLineSampToXY.def so they will not be impacted.

2) When the new code is deployed it will reference the updated pdsProjectionLineSampToXY_V2.def and have the correct updated hardcoded default values. The old pdsProjectionLineSampToXY.def will be deprecated but will be left in place fo the old versions of the programs. Note that test data will be updated when the new code is deployed. If old versions of the programs are tested against the new test data they may fail the tests.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Trent Hare (Trent Hare) Original Date: 2017-05-16T18:10:36Z

I agree with the roll-out as John has described. But given that pdsProjectionLineSampToXY.def is used from the ISIS3DATA area, once this is changed, might it impact users who update (rsync) "DATA" but not ISIS? Should this file be versioned? e.g. pdsProjectionLineSampToXY_v2.def or time tagged?

ascbot commented 5 years ago

Original Redmine Comment Author Name: Stuart Sides (@scsides) Original Date: 2017-07-14T17:29:38Z

Closed after testing by Trent