ascbot commented 5 years ago

Author Name: Tammy Becker (Tammy Becker)

Original Assignee: Stuart Sides

The purpose of this ticket is to centralize information regarding the ISIS Pixel Projection Offset issue when importing and exporting maps (e.g., DEMs, Mosaic Maps, etc).

Tickets for modifying specific ISIS applications regarding this offset issue should be "related" to this main ticket for context and tracking.

The crux of the matter is how ISIS references the coordinate system of an imported digital map, particularly the "upper left pixel". The coordinate system reference is applied upon import (pds2isis) and export (isis2pds) and depends on the definition of the source product (i.e., The pixel offset definition differences between ISIS and PDS).

Additional coordinate system details are the latitude system, radius (especially triaxial), and longitude direction.

"Watchers" of this post are initially based on previous and current involvement/interest.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2015-04-27T18:15:23Z

Brent Archinal's email (April 26, 2015) following a meeting with TU Berlin [E. Tasdelen and K. Willner].

"I would suggest the steps to be considered/taken in the long run are:

Documentation (investigation and addition to ISIS documentation) 1a) How does ISIS handle each data set on input? 1b) How does ISIS handle each data set on output?

(Jeff Anderson's e-mail of 2011 January 27 says the input calculations are described in $ISIS3DATA/base/translations/pdsProjectionLineSampToXY.def, but there may be other locations as well. I assume that file is used by PDS2ISIS style programs, but what about input of DEMs or other files?)

Investigation/decisions 2a) Are the datasets being handled correctly on input? 2b) Is ISIS "doing the right thing" internally, i.e. to make products. 2c) Should datasets be handled the same way on output as on input (probably)?
Software changes, based on step 2 (if any). 3a) Change input steps as necessary. 3b) Other ISIS changes as necessary so it does the right thing internally. 3c) Change output steps to match input steps (presumably). 3d) Update documentation (being careful to save the old documentation on what ISIS previously did, somehow)
Optional: Create a white paper or some other publication(s) describing the issue and how it's handled for both PDS 3 and 4, and for GDAL. 4a) Distribute appropriately, e.g. ISIS documentation; publication; distribution to PDS personnel, GDAL author(s) "

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2015-04-27T18:16:34Z

Trent Hare's email [April 27, 2015]: " Just FYI - submit update to GDAL to fix PDS offset ingest bug. https://trac.osgeo.org/gdal/ticket/5941

Please check. I can update if you find anything incorrect.

To correct in ISIS the file $ISIS3DATA/base/translations/pdsProjectionLineSampToXY.def

These lines should be changed to

STANDARD PDS

Should always be last

Group = Selection Keyword = "PDS_VERSION_ID" Pattern = "PDS3" xMult = -1.0 yMult = 1.0 xOff = 0.5 yOff = 0.5 End_Group And I think this line for MOLA should be added

Group = Selection Keyword = "DATA_SET_ID" Pattern = "MGS-M-MOLA-5-MEGDR-L3-V1.0" xMult = -1.0 yMult = 1.0 xOff = -0.5 yOff = -0.5 End_Group "

ascbot commented 5 years ago

Original Redmine Comment Author Name: Trent Hare (Trent Hare) Original Date: 2015-04-28T16:46:20Z

If the defaults are truly deemed incorrect (all signs to point to that from multiple avenues), then we will need to also fix the source code defaults.

from: http://isis.astrogeology.usgs.gov/Object/Programmer/_process_import_pds_8cpp_source.html

01424 void ProcessImportPds::GetProjectionOffsetMults(double &xoff, double &yoff, 01425 double &xmult, double &ymult) { 01426 01427 xmult = -1.0; 01428 ymult = 1.0; 01429 xoff = -0.5; 01430 yoff = -0.5;

patched to 01429 xoff = 0.5; 01430 yoff = 0.5;

ascbot commented 5 years ago

Original Redmine Comment Author Name: Trent Hare (Trent Hare) Original Date: 2015-04-28T23:14:39Z

If we are really going to do this on import (if proven correct) then exporting will need to also be updated. http://isis.astrogeology.usgs.gov/Object/Programmer/_process_export_pds_8cpp_source.html

from: 00776 lineOffset += 0.5; // Add half a line to get to the center of (1,1) 00781 sampleOffset += 0.5; // Add half a sample to get to the center of (1,1)

to

00776 lineOffset -= 0.5; // Subtract half a line to get to the center of (1,1) 00781 sampleOffset -= 0.5; // Subtract half a sample to get to the center of (1,1)

A great test image is LOLA DEM since it is so small, simple and global 1440x720 PDS image: http://pds-geosciences.wustl.edu/lro/lro-l-lola-3-rdr-v1/lrolol_1xxx/data/lola_gdr/cylindrical/img/ldem_4.img detached label: http://pds-geosciences.wustl.edu/lro/lro-l-lola-3-rdr-v1/lrolol_1xxx/data/lola_gdr/cylindrical/img/ldem_4.lbl parent link: http://pds-geosciences.wustl.edu/missions/lro/lola.htm

Currently it does appear ISIS is exporting to PDS files one pixel off.

-Trent

ascbot commented 5 years ago

Original Redmine Comment Author Name: Christopher Isbell (Christopher Isbell) Original Date: 2015-12-04T20:42:59Z

I'll start by showing some basic testing results, toward confirming the "one pixel off" issue as known prior and discussed in this thread. Then, will follow with additional detail. (I plan to "submit" at stages throughout this session, so watchers will likely see multiple updates).

First, a few (mostly known) points (corrections welcome):

Assumptions/conclusions:

isis2pds creates output PDS3 files containing incorrect line sample_projection_offset values (off by one for each axis), intended to be offsets relative to +center+ (per PDS3 and ISIS2 definitions) +of pixel 1,1+.
* And, considering ingest of such files (per definition file):
```
 #  The defaults used in ProcessImportPds are:
 #      xMult  =  -1.0
 #      yMult  =   1.0
 #      xOff   =  -0.5
 #      yOff   =  -0.5
 ...
```

#** Therefore, a PDS3 labeled file with incorrect offset values will be adjusted (correctly, for UpperLeftCornerX/Y values) at ingest via _pds2isis_ (good).
&nbsp;
1. +*However*+, if a mission/project produces PDS3 files containing +correct+ offsets, _pds2isis_ (without a relevant definition file entry) will introduce an "error" (to UpperLeftCornerX/Y values) at ingest.
&nbsp;
#* Of course, +*if*+ we know this, we can define appropriate scalar/translation values (per subject) via the definition file.
&nbsp;
1. Now, a "simple" 1ppd case via ISIS3 (360 samples x 180 lines) ...
&nbsp;
#* _isis2pds_ output (global equi/simp/sinu):

OBJECT = IMAGE LINES = 180 LINE_SAMPLES = 360 ... CENTER_LONGITUDE = 180.0 ... MAP_RESOLUTION = 1.0 <PIX/DEG> ... MAXIMUM_LATITUDE = 90.0 MINIMUM_LATITUDE = -90.0 EASTERNMOST_LONGITUDE = 360.0 WESTERNMOST_LONGITUDE = 0.0 LINE_PROJECTION_OFFSET = 90.499999999998 SAMPLE_PROJECTION_OFFSET = 180.5

#* And, according to PDS3 and ISIS2 definitions:
```(from equator/180 longitude origin)
line offset should be   [ (90x1) - 0.5] =  89.5
sample offset should be [(180x1) - 0.5] = 179.5

** So, yes, isis2pds offset values are incorrect (by one pixel, significant at low res, less significant at higher res).

And, a slightly less simple case ...

* isis2pds output (equirectangular clat=0, = simple cylindrical)


OBJECT = IMAGE
LINES                    = 7666
LINE_SAMPLES             = 15331
...
OBJECT = IMAGE_MAP_PROJECTION
...
MAP_PROJECTION_TYPE      = "EQUIRECTANGULAR"
CENTER_LATITUDE          = 0.0 <deg>
CENTER_LONGITUDE         = 180.0 <deg>
...
MAP_RESOLUTION           = 42.586033748662 <pix/deg>
MAP_SCALE                = 1.0 <km/pixel>
MAXIMUM_LATITUDE         = 90.0 <deg>
MINIMUM_LATITUDE         = -90.0 <deg>
EASTERNMOST_LONGITUDE    = 360.0 <deg>
WESTERNMOST_LONGITUDE    = 0.0 <deg>
LINE_PROJECTION_OFFSET   = 3833.5 <pixel>
SAMPLE_PROJECTION_OFFSET = 7666.5 <pixel>

#* Per definitions:
```(from equator/180 longitude origin)
line offset should be   [nint( 90 x 42.586033748662) - 0.5] = 3832.5 
sample offset should be [nint(180 x 42.586033748662) - 0.5] = 7664.5 ?
.. or ? ..
sample offset should be [round up(180 x 42.586033748662) - 0.5] = 7665.5
(re "extra" sample (or line) above, and "final details" below)

** Again, isis2pds offset values are off by one pixel.

Along with, a Polar Stereographic case ...


OBJECT = IMAGE
LINES                    = 3078
LINE_SAMPLES             = 3078
...
OBJECT = IMAGE_MAP_PROJECTION
...
MAP_PROJECTION_TYPE      = "POLAR STEREOGRAPHIC"
...
CENTER_LATITUDE          = 90.0 <deg>
CENTER_LONGITUDE         = 0.0 <deg>
...
MAP_RESOLUTION           = 42.586033748662 <pix/deg>
MAP_SCALE                = 1.0 <km/pixel>
MAXIMUM_LATITUDE         = 90.0 <deg>
MINIMUM_LATITUDE         = 55.0 <deg>
EASTERNMOST_LONGITUDE    = 360.0 <deg>
WESTERNMOST_LONGITUDE    = 0.0 <deg>
LINE_PROJECTION_OFFSET   = 1539.5 <deg>
SAMPLE_PROJECTION_OFFSET = 1539.5 <deg>

(from pole origin) line offset should be [ (3078/2) - 0.5] = 1538.5 sample offset should be [ (3078/2) - 0.5] = 1538.5



&nbsp;
+*Summary (current ISIS3 function)*+. Again, most of this is already known by others. Per other project work and cartographic interest, and as requested, I've provided related testing/confirmation and additional input and background here. I hope this is helpful.
* _isis2pds_ creates output PDS3 files containing incorrect projection_offset values.
* An _isis2pds_ generated PDS3 labeled file (thus, reflecting incorrect offset values) will be corrected at ISIS3 ingest via _pds2isis_.
* An ISIS3 ingest via _pds2isis_ will adjust projection_offset values according to relevant entries (if they exist) within the ProcessImportPds definition file.
* +*Any other*+ file ingested via _pds2isis_ will have projection offset values adjusted by default definition values (see item 1. above).
** That is, a _pds2isis_ input file containing "correct" projection offset values will have offset values adjusted (by default) at ingest (resulting in incorrect ISIS3 UpperLeftCornerX/Y values).
&nbsp;

+*Some final details:*+
* This issue/problem becomes more complex when ...
** Due to "round-off" and/or other cartographic calculations, a resulting image array contains an extra line and/or sample (e.g. 361x181 vs 360x180, and re item 4. above).
** ..., thus uncertainty on the precise location of projection origin within a specific pixel, etc.
** And other complexities I'm sure.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Christopher Isbell (Christopher Isbell) Original Date: 2016-01-27T22:28:28Z

Although "Updated by" date did not change, there has been lengthy recent addition to "C Isbell" item (#10) above. Thanks, Chris

ascbot commented 5 years ago

Original Redmine Comment Author Name: Christopher Isbell (Christopher Isbell) Original Date: 2016-01-28T16:03:07Z

Comments/Questions/Conclusions: Based on ticket history here, and discussions with others, it appears this is the summary of what should be done regarding ISIS export and import. Comments/correction welcome. And, I assume such decisions/actions would come following final discussions among management/developers/stakeholders (most "Watchers" here):

For export, make changes to code/config files such that isis2pds produces correct offset values (re THare item#9 above).
For import, make changes to code/config files such that, +by default+, pds2isis applies +no correction+ (i.e. scalar=1.0, translation=0.0) to input file projection offset values ... ... +except+ where ProcessImportPds content defines otherwise per +specific+ products/datasets. This means there should be no default values (as there is now) within ProcessImportPds. * Therefore, ProcessImportPds (and other code/config files?) would need to be updated to contain entries for +ALL*+ known products/datasets previously produced with "incorrect" offset values. (True?)

Finally, a current project example: We plan to produce MESSENGER MDIS PDS3 archive products containing correct projection offset values. Currently, a pds2isis import of those products would result in incorrect ISIS3 UpperLeftCornerX/Y values. ** With changes discussed/proposed here, these MDIS products (or any other "correct" products) would be ingested within ISIS without spurious offsets applied.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Christopher Isbell (Christopher Isbell) Original Date: 2016-02-16T20:06:06Z

Important Update!

Re per #12 content above:

For import, make changes to code/config files such that, +by default+, pds2isis applies +no correction+ (i.e. scalar=1.0, translation=0.0) to input file projection offset values ...

Note: This statement is +NOT+ universally accurate regarding scalar & translation values. See KBecker/CIsbell per recent MESSENGER-MIDS implementation/testing.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Trent Hare (Trent Hare) Original Date: 2016-02-23T01:17:35Z

didn't mean to changed from Bug to Recommendation

ascbot commented 5 years ago

Original Redmine Comment Author Name: Trent Hare (Trent Hare) Original Date: 2016-02-23T01:19:52Z

I agree with the steps that Chris proposed above in #12.

For further background and clarification into the subject, here is a LPSC abstract also: http://www.hou.usra.edu/meetings/lpsc2016/pdf/1812.pdf

I was worried about ISIS2 cubes, which are also converted to ISIS3 using pds2isis, but an update in the translation table was applied in 2008 which appears to have it correctly mapped.

-Trent

ascbot commented 5 years ago

Original Redmine Comment Author Name: Trent Hare (Trent Hare) Original Date: 2016-02-23T16:50:48Z

Just for documenting, Kris added local support for the forthcoming Messenger DEM PDS archive and HRSC PDS archive. This is not a fix just a work-around for these two data sets.

So within. $ISIS3DATA/base/translations/pdsProjectionLineSampToXY.def

these two sections were added to support proper registration:

  # MESSENGER DEM Products archived in PDS on March, 2016. These products
  # have been corrected for the PDS environment such that offsets are correct.
  # Therefore, on ingest into ISIS, additional adjustments are required by
  # pds2isis. The values in the group below will accomodate this condition.
  # Products derived by the USGS on 2015-12-02. Contact: Kris J Becker.
  Group = Selection
    Keyword = "DATA_SET_ID"
    Pattern = "MESS-H-MDIS-5-DEM-ELEVATION-V1.0"
    xMult   = -1.0
    yMult   = 1.0
    xOff    = 0.5
    yOff    = 0.5
  End_Group

  # HRSC products archived in PDS
  Group = Selection
    Keyword = "INSTRUMENT_NAME"
    Pattern = "HIGH RESOLUTION STEREO CAMERA"
    xMult   = -1.0
    yMult   = 1.0
    xOff    = 0.5
    yOff    = 0.5
  End_Group

ascbot commented 5 years ago

Original Redmine Comment Author Name: Jason Laura (Jason Laura) Original Date: 2016-11-10T18:31:25Z

DoD for the import version:

Change 2 lines of code and run the tests
Identify what broke
Research the 'break' for each to see if you have actually fixed it - this is documentation
Clean up user test tickets if it did not break anything

ascbot commented 5 years ago

Original Redmine Comment Author Name: John Bonn (John Bonn) Original Date: 2017-05-22T20:28:08Z

See #4507 for notes on deployment.

Programs that export PDS: mrf2pds hidtmgen mdisddr lopdsgen mdis2pds hirdrgen isis2pds hideal2pds

A test plan for output programs could involve reading in ISIS cubes and writing as PDS using both ISIS and GDAL and comparing the output. This has not been done but would be a very good test to run.

Truth data to checkin from m02244 branch: (Note this list was generated using code that had a different unrelated bug committed but did not have the corresponding truth data committed - the truth data and code were out of sync. The implication is this list may have more changes than necessary) /work/users/jbonn/m02244/isis/src/mro/apps/crism2isis/tsts/mrdr /work/users/jbonn/m02244/isis/src/lro/apps/mrf2isis/tsts/level2 /work/users/jbonn/m02244/isis/src/mro/apps/hirdr2isis/tsts/default /work/users/jbonn/m02244/isis/src/base/apps/isis2pds/tsts/unitConversion /work/users/jbonn/m02244/isis/src/base/apps/isis2pds/tsts/offsetTest /work/users/jbonn/m02244/isis/src/lro/apps/mrf2pds/tsts/CH_Lev2 /work/users/jbonn/m02244/isis/src/lro/apps/mrf2pds/tsts/CH_Lev3 /work/users/jbonn/m02244/isis/src/mro/apps/hidtmgen/tsts/color /work/users/jbonn/m02244/isis/src/mro/apps/hidtmgen/tsts/dtmOnly /work/users/jbonn/m02244/isis/src/mro/apps/hidtmgen/tsts/equi /work/users/jbonn/m02244/isis/src/mro/apps/hidtmgen/tsts/nonDefaultNames /work/users/jbonn/m02244/isis/src/mro/apps/hidtmgen/tsts/orthoOnly /work/users/jbonn/m02244/isis/src/mro/apps/hidtmgen/tsts/outputTypes /work/users/jbonn/m02244/isis/src/mro/apps/hidtmgen/tsts/polar /work/users/jbonn/m02244/isis/src/mro/apps/hirdrgen/tsts/color /work/users/jbonn/m02244/isis/src/mro/apps/hirdrgen/tsts/manual /work/users/jbonn/m02244/isis/src/mro/apps/hirdrgen/tsts/red /work/users/jbonn/m02244/isis/src/mro/apps/hirdrgen/tsts/red2 /work/users/jbonn/m02244/isis/src/odyssey/apps/thmbasemap1/tsts/dayonly /work/users/jbonn/m02244/isis/src/odyssey/apps/thmbasemap1/tsts/nightonly /work/users/jbonn/m02244/isis/src/mro/apps/hicolormos/tsts/case01 /work/users/jbonn/m02244/isis/src/mro/apps/himos/tsts/case01 /work/users/jbonn/m02244/isis/src/mro/apps/himos/tsts/case02pole /work/users/jbonn/m02244/isis/src/messenger/apps/mdisedrinfo/tsts/default /work/users/jbonn/m02244/isis/src/messenger/apps/mdisedrinfo/tsts/kernelchk /work/users/jbonn/m02244/isis/src/messenger/apps/mdisedrinfo/tsts/multi /work/users/jbonn/m02244/isis/src/local/apps/camdev/tsts/all /work/users/jbonn/m02244/isis/src/control/apps/autoseed/tsts/seeddef /work/users/jbonn/m02244/isis/src/base/apps/caminfo/tsts/boundary /work/users/jbonn/m02244/isis/src/base/apps/caminfo/tsts/pole /work/users/jbonn/m02244/isis/src/base/apps/caminfo/tsts/uselabel /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/default /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/flat /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/no_sample_line /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/pointlist_append /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/pointlist_error /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/pointlist_flat /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/pointlist_ground /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/pointlist_no_append /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/set_sample_or_line /work/users/jbonn/m02244/isis/src/base/apps/campt/tsts/setSL /work/users/jbonn/m02244/isis/src/base/apps/fx/tsts/bandDependent /work/users/jbonn/m02244/isis/src/base/apps/fx/tsts/camera /work/users/jbonn/m02244/isis/src/base/apps/phocube/tsts/allbands /work/users/jbonn/m02244/isis/src/base/apps/phocube/tsts/default /work/users/jbonn/m02244/isis/src/base/apps/photrim/tsts/base /work/users/jbonn/m02244/isis/src/base/apps/photrim/tsts/emission /work/users/jbonn/m02244/isis/src/kaguya/apps/mimap2isis/tsts/default

ascbot commented 5 years ago

Original Redmine Comment Author Name: John Bonn (John Bonn) Original Date: 2017-05-25T20:59:35Z

Summary of Changes.. The original tickets had lists of programs that would potentially would be impacted by PDS changes. (The list was generated by searching for "IMAGE_MAP_PROJECTION"). A couple of the programs (gllnims2isis, hrsc2isis) were determined not to be impacted since they did not do map projections.

The remaining programs were tested to confirm they actually tested the validity of the offset values. Tests were added to three programs to check PDS input/ouput validity: pds2isis, isis2pds, and mrf2isis. After this step it is assumed all programs impacted by the PDS change had test cases for the change.

By reverting the changes and running the tests we can generate the list of impacted programs:

mrf2pds crism2isis hidtmgen hirdr2isis hirdrgen isis2pds pds2isis kaguyatc2isis mimap2isis

Trent has the concern that some mission-specific export programs may have their own PDS output routine that could have by-passed this process.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Trent Hare (Trent Hare) Original Date: 2017-05-26T01:03:34Z

Many mission PDS export applications are not intended for level2 (map projected data). Most ISIS3 generated PDS files are only created for level 1 data only (e.g. LROC, MESSENGER).

The 4 ISIS export applications I see for level 2 data (map projected) are isis2pds, mrf2pds, hidtmgen, and hirdrgen. All have been tested and require pixel offset fixes.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Trent Hare (Trent Hare) Original Date: 2017-05-26T22:04:48Z

Getting really close on this ticket! I have attached (linked) an edited version of the new $ISIS3DATA/base/translations/pdsProjectionLineSampToXY_V2.def file for the purposes of:

remove PDS products which now match the PDS defaults as defined in ProcessImportPDS.cpp (and as described in the top of this *V2.def file
Add in MOLA products into the *V2.def file to correct them. They were coming in correctly but with this code update we need to correct them. The MOLA labels do have an original 1 pixel offset issue.

why: I would like this *V2.def file to be only the PDS products which are found to need overrides (have bad label offsets). This also may help with this ticket to report when a PDS offset override is needed: https://isis.astrogeology.usgs.gov/fixit/issues/4887

I can't attached files so it is here: /usgs/shareall/thare/isis3/pdsProjectionLineSampToXY_V2.def

ascbot commented 5 years ago

Original Redmine Comment Author Name: Stuart Sides (@scsides) Original Date: 2017-07-14T17:21:16Z

Closed after testing by Trent

DOI-USGS / ISIS3

Pixel (Line/Sample) Projection Offset Issue #1995

STANDARD PDS

Should always be last

* And, considering ingest of such files (per definition file):

** So, yes, isis2pds offset values are incorrect (by one pixel, significant at low res, less significant at higher res).

* isis2pds output (equirectangular clat=0, = simple cylindrical)

** Again, isis2pds offset values are off by one pixel.

A test plan for output programs could involve reading in ISIS cubes and writing as PDS using both ISIS and GDAL and comparing the output. This has not been done but would be a very good test to run.